Method and system for predicting musculoskeletal diseases based on deep learning

Abstract

This invention provides a deep learning-based method and system for predicting musculoskeletal diseases that reduces initial installation and maintenance costs compared to sensor-based systems. [Solution] The method includes the steps of: receiving video footage of a worker to be filmed and the specific environment in which the worker is located; using a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video footage; extracting key points corresponding to multiple joints included in the worker's body based on the first bounding box; calculating the angles between the multiple joints using the extracted key points and calculating a risk score for the calculated angles between the multiple joints; identifying the degree of danger to the worker based on the calculated risk score; and generating information on countermeasures corresponding to the identified degree of danger.

Description

Technical Field

[0001] The present invention relates to a method and system for predicting musculoskeletal diseases based on deep learning.

[0002] This research was conducted with research funding (RS-2023-00251002) from the Japan Science and Technology Agency (KAIA), supported by the Ministry of Land, Infrastructure, Transport and Tourism.

Background Art

[0003] Musculoskeletal diseases refer to pain or injuries that occur in the musculoskeletal system, such as muscles, nerves, tendons, ligaments, bones, and surrounding tissues. Musculoskeletal diseases occur in various parts of the body, such as the neck, waist, arms, and legs.

[0004] Musculoskeletal diseases are caused by excessive use of specific body parts due to incorrect working postures or repetitive work. In particular, in the construction industry, workers are burdened greatly on the musculoskeletal system because they lift heavy objects and perform repetitive work or maintain uncomfortable postures for a long time.

[0005] In relation to this, according to a report by the World Health Organization (WHO), it has been revealed that the economic losses due to musculoskeletal diseases are very large. That is, such musculoskeletal diseases are chronic pains that affect not only daily life but also economic activities.

[0006] Recently, as working posture evaluation methods, various working posture evaluation methods such as self-reporting, direct measurement, and observational evaluation methods have been provided. However, the self-reporting method may be subjective and have low reliability, and direct measurement may interfere with work due to the wearing of sensors.

[0007] Therefore, in order to solve such problems, a method of analyzing the posture of workers and automatically evaluating the risk level by utilizing deep learning technology is necessary.

Summary of the Invention

Problems to be Solved by the Invention

[0008] The present invention provides a deep learning-based method and system for predicting musculoskeletal disorders that can predict potential musculoskeletal disorders that may arise from various incorrect postures.

[0009] More specifically, the present invention provides a deep learning-based method and system for predicting musculoskeletal disorders that can analyze a worker's posture and movements in real time and provide immediate feedback.

[0010] Furthermore, the present invention provides a deep learning-based method and system for predicting musculoskeletal disorders that can predict dangerous postures early and respond flexibly to the predicted dangerous postures. [Means for solving the problem]

[0011] To solve the above problems, the deep learning-based musculoskeletal disease prediction method according to the present invention may include the steps of: receiving video of a worker to be filmed and a specific environment in which the worker is located; using a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video; extracting keypoints corresponding to a plurality of joints included in the worker's body based on the first bounding box; calculating the angles between the plurality of joints using the extracted keypoints and calculating a risk score for the calculated angles between the plurality of joints; identifying the degree of risk to the worker based on the calculated risk score; and generating information on countermeasures corresponding to the identified degree of risk.

[0012] Furthermore, the video includes at least one of the worker's posture, movements, movement patterns, the worker's body, and the specific environment, and the specific environment may include at least one of the workspace where the worker is working, the location of tools, equipment, and workpieces located in the workspace.

[0013] Furthermore, in the step of identifying the bounding boxes, the received video may be processed as input to the deep learning model, the deep learning model may analyze the worker and the specific environment from the video, and use the analysis results to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment.

[0014] Furthermore, in the step of extracting the key points, the joint positions corresponding to the plurality of joints may be adjusted according to the physical structure of the worker based on the first bounding box, and the key points may be extracted from the joint positions adjusted according to the physical structure.

[0015] Furthermore, the calculation of the joint angle may involve generating a vector corresponding to the joint position based on the extracted keypoint, and then using the generated vector to calculate the joint angle between the joint positions.

[0016] Furthermore, the calculation of the risk score may involve predicting the worker's risk level using the calculated joint angles, and then calculating the risk score for the joint angles based on the predicted results.

[0017] Furthermore, in the step of identifying the degree of risk, the risk score for the calculated joint angle is processed as input to the deep learning model, and the deep learning model identifies the degree of risk based on the risk score from among a plurality of degrees of risk based on a predetermined criterion, and the plurality of degrees of risk may include at least one of a first degree of risk identified based on the risk score not meeting the predetermined criterion, a second degree of risk identified based on the risk score meeting the predetermined criterion, and a third degree of risk identified based on the risk score exceeding the predetermined criterion.

[0018] Furthermore, each of the multiple risk levels has matching information on different countermeasures, and the information on different countermeasures includes at least one of the following: first countermeasure information matched to the first risk level, second countermeasure information matched to the second risk level, and third countermeasure information matched to the third risk level. In the step of generating the countermeasures information, if the identified risk level corresponds to at least one of the second and third risk levels which are different from the first risk level, the second or third countermeasure information for predicting the musculoskeletal disease may be generated.

[0019] On the other hand, the musculoskeletal disease prediction system based on deep learning according to the present invention includes a data collection unit that receives video of a worker to be filmed and a specific environment in which the worker is located, and a control unit that uses a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video. The control unit can extract keypoints corresponding to a plurality of joints in the worker's body based on the first bounding box, calculate the angles between the plurality of joints using the extracted keypoints, calculate a risk score for the calculated angles between the plurality of joints, identify the degree of risk to the worker based on the calculated risk score, and generate information on countermeasures corresponding to the identified degree of risk.

[0020] On the other hand, the program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, and the program may include instructions to execute the following steps: receiving video of a worker to be filmed and a specific environment in which the worker is located; using a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video; extracting keypoints corresponding to a plurality of joints included in the worker's body based on the first bounding box; calculating the angles between the plurality of joints using the extracted keypoints and calculating a risk score for the calculated angles between the plurality of joints; identifying the degree of danger to the worker based on the calculated risk score; and generating countermeasures information according to the identified degree of danger. [Effects of the Invention]

[0021] As described above, according to the deep learning-based musculoskeletal disease prediction method and system according to the present invention, images related to workers and the working environment can be collected using only a camera device. Thereby, in the present invention, compared with the conventional sensor-based system, the initial installation and maintenance costs can be reduced.

[0022] Also, according to the deep learning-based musculoskeletal disease prediction method and system according to the present invention, the joint position can be detected using a deep learning model, and the working posture can be analyzed by calculating the angle between joints. Thereby, in the present invention, without using additional equipment, the joint position can be detected from only the image, and even when the worker changes the posture or performs a complex movement, the joint position can be tracked in real time.

[0023] Furthermore, according to the deep learning-based musculoskeletal disease prediction method and system according to the present invention, the posture and movement of the worker can be analyzed in real time, and immediate feedback can be provided. Thereby, the user can recognize the possibility of developing musculoskeletal diseases at an early stage and take preventive measures.

[0024] Furthermore, according to the deep learning-based musculoskeletal disease prediction method and system according to the present invention, dangerous postures can be predicted early, and this can be supported so that the worker can recognize it in real time. Thereby, the worker can immediately correct the dangerous posture, maintain a safe posture, and prevent the occurrence of musculoskeletal diseases.

Brief Description of Drawings

[0025] [Figure 1] It is a conceptual diagram for explaining a deep learning-based musculoskeletal disease prediction system according to the present invention. [Figure 2] It is a flowchart for explaining a deep learning-based musculoskeletal disease prediction method. [Figure 3]This is a conceptual diagram illustrating the deep learning-based method for predicting musculoskeletal diseases according to the present invention. [Figure 4] This is a conceptual diagram illustrating the deep learning-based method for predicting musculoskeletal diseases according to the present invention. [Figure 5] This is a conceptual diagram illustrating the deep learning-based method for predicting musculoskeletal diseases according to the present invention. [Figure 6] This is a conceptual diagram illustrating the deep learning-based method for predicting musculoskeletal diseases according to the present invention. [Figure 7] This is a conceptual diagram illustrating one embodiment of the user interface screen according to the present invention. [Figure 8] This is a conceptual diagram illustrating one embodiment of the user interface screen according to the present invention. [Modes for carrying out the invention]

[0026] The embodiments disclosed herein will be described in detail below with reference to the accompanying drawings, but regardless of the reference numerals used in the drawings, identical or similar components will be given the same reference numerals, and redundant descriptions thereof will be omitted. The suffixes “module” and “part” used for components in the following description are added or used interchangeably for the sake of ease of writing the specification and do not have any distinguishing meaning or role in themselves. Furthermore, when describing the embodiments disclosed herein, if it is determined that a detailed description of the relevant prior art would obscure the gist of the embodiments disclosed herein, such detailed description will be omitted. In addition, the accompanying drawings are intended solely to facilitate the understanding of the embodiments disclosed herein, and the technical ideas disclosed herein should not be limited by the accompanying drawings and should be understood to include all modifications, equivalents and substitutions that fall within the concept and technical scope of the present invention.

[0027] Terms including ordinal numbers such as "1st," "2nd," etc., may be used to describe various components, but the components are not limited to those defined by these terms. These terms are used solely to distinguish one component from another.

[0028] When it is stated that one component is “connected” or “linked” to another component, it should be understood that it may be directly connected or linked to the other component, but there may also be another component between them. On the other hand, when it is stated that one component is “directly connected” or “directly linked” to another component, it should be understood that there is no other component between them.

[0029] A singular expression includes plural forms unless otherwise clearly indicated in the context.

[0030] In this application, terms such as “includes” or “having” are intended to specify the presence of features, figures, steps, actions, components, parts, or combinations thereof as described in the specification, and should be understood not to preemptively exclude the possibility of the presence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0031] This invention relates to a method and system for predicting musculoskeletal disorders based on deep learning. The musculoskeletal disorder prediction system according to the present invention may be a system that predicts potential musculoskeletal disorders that may arise from various incorrect postures of the user (or worker). Alternatively, the musculoskeletal disorder prediction system according to the present invention may be a system that analyzes the user's posture and movements in real time and provides immediate feedback.

[0032] In one embodiment, the musculoskeletal disease prediction system according to the present invention can improve the working posture and specific environment of the subject being photographed, and can be used in high-risk work environments such as industrial sites and construction sites.

[0033] Here, musculoskeletal disorders (MSDs) can refer to conditions involving pain and injury to the tissues that make up the musculoskeletal system, such as muscles, joints, nerves, ligaments, tendons, and bones. Such musculoskeletal disorders mainly occur when excessive strain is placed on specific areas due to incorrect posture, repetitive movements, excessive force use, or an inappropriate work environment.

[0034] Major joints can refer to joint points that are important when analyzing body movement and posture. For example, major joints may include individual joints such as the head, neck, thoracic spine (center of the upper body, thorax), waist (or pelvis), left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0035] The musculoskeletal disease prediction system according to the present invention includes a deep learning model, and the deep learning model according to the present invention may be an object detection (or detection, recognition, etc.) model based on YOLOv8.

[0036] In one embodiment, the structure of the YOLOv8 model may be configured to include three main components: a backbone network, a neck network, and a head network.

[0037] Backbone networks primarily perform feature extraction, transforming input images into multi-scale, multi-level feature maps through a series of convolutional layers, pooling operations, and other transformations.

[0038] YOLOv8 can apply a cross-stage subset network (CSPNet) as the core module of its backbone network. CSPNet can improve feature representation and reduce computational complexity by partially connecting feature maps between stages. The feature map may be separated into two parts: one for direct feature extraction, and the other for cross-stage connection, after which both parts may be merged.

[0039] The neck network sits between the backbone and the sensing head, integrating and enhancing the features generated by the backbone network and complementarily combining feature maps of various resolutions. A key technique in VOLOv8's neck design is C2F, which improves semantic feature representation by combining features at various levels with contextual information.

[0040] Another technique is Path Aggregation (PANet), which can combine features at various resolutions and semantic levels through top-down path aggregation.

[0041] The head design acts as the output layer of the model and can be responsible for specific object detection tasks such as object class, location, and confidence score. YOLOv8 head designs can typically include multi-detection heads designed to process feature maps of various scales, such heads can leverage convolutional and fully connected layers to process input feature maps and generate the final detection results.

[0042] Furthermore, YOLOv8 head design can employ either an anchor-free or anchor-based approach. The anchor-free approach reduces computational complexity and improves detection speed by directly predicting the object's center point, width, and height, while the anchor-based approach can detect objects of various shapes and sizes by adjusting a predefined anchor box.

[0043] In other words, the musculoskeletal disease prediction system according to the present invention may include a deep learning model based on the YOLOv8 model in order to predict various musculoskeletal diseases that may occur in industrial or construction sites.

[0044] The present invention will be discussed in more detail below with reference to the attached drawings. Figure 1 is a conceptual diagram illustrating the musculoskeletal disease prediction system according to the present invention. Figure 2 is a flowchart illustrating the musculoskeletal disease prediction method based on deep learning, and Figures 3, 4, 5, and 6 are conceptual diagrams illustrating the musculoskeletal disease prediction method based on deep learning according to the present invention. Furthermore, Figures 7 and 8 are conceptual diagrams illustrating one embodiment of the user interface screen according to the present invention.

[0045] On the other hand, as shown in Figure 1, the musculoskeletal disease prediction system 100 according to the present invention may include at least one of the following: a data acquisition unit 110, a data processing unit 120, a communication unit 130, a storage unit 140, a control unit 150, and a deep learning model 160.

[0046] The data acquisition unit 110 may be configured to collect various data about the worker being photographed and the specific environment in which the worker is located. For example, the data acquisition unit 110 can collect video (or image) data related to musculoskeletal diseases at a construction (or industrial) site from various sources (e.g., AI-Hub, control system, database, web crawling, API, server linked to the musculoskeletal disease prediction system 100, external server, etc.). The data acquisition unit 110 may also include at least one vision-based data acquisition device.

[0047] For example, a vision-based data acquisition device is a device that captures visual information such as images and videos to collect data. This device mainly includes a camera, image sensor, and video processing software, and can be used to collect and analyze visual data. Examples of vision-based data acquisition devices include digital cameras, smartphone cameras, CCTV cameras, vision systems for autonomous vehicles, industrial vision systems, and smart glasses (e.g., Google Glass). TM This may include at least one of the following: a drone, scanning equipment (e.g., a 3D scanner), and a vision-based robotic system.

[0048] However, in the present invention, the types of components comprising the data acquisition unit 110 are not particularly limited, and it is sufficient as long as they realize the functions defined by each device.

[0049] The data processing unit 120 may be configured to process the collected video data, convert it into image frames, identify (or select, extract, etc.) high-quality data, and then expand the data by image augmentation. In this case, the data processing unit 120 may annotate the converted image frames, focusing on objects of interest (e.g., workers, heavy equipment, goods, transport equipment, etc.).

[0050] The communication unit 130 may then be connected wirelessly or via a wired network to a control system (or control server, or control room), equipment (or device), external server, and one or more networks, and may be configured to receive or transmit overall data and information necessary for the operation of the musculoskeletal disease prediction system 100.

[0051] Here, the control system is configured to control workers located at a specific site (e.g., a construction site or an industrial site) and can manage workers using various information provided by the musculoskeletal disease prediction system 100 (e.g., information on countermeasures for dangerous postures of workers).

[0052] Furthermore, the communication unit 130 can receive video footage collected from the data collection unit 110. For example, the communication unit 130 can receive information from the data collection unit 110 such as the movements of construction site workers, location information, tool information, dangerous postures, and the surrounding environment.

[0053] Furthermore, the communication unit 130 can support various communication methods depending on the communication standard of the device being communicated with.

[0054] For example, the communication unit 130 supports WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), and Bluetooth. TMIt may be configured to communicate with a communication target using at least one of the following technologies: (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), or Wireless USB (Wireless Universal Serial Bus).

[0055] On the other hand, the memory unit 140 may also be named "database (DB)" or "memory," and may be configured to store various information related to the present invention. In the present invention, the memory unit 140 may be provided in the musculoskeletal disease prediction system 100 itself. Furthermore, at least a part of the memory unit 140 may be configured as a cloud server (or cloud storage). In other words, the memory unit 140 only needs to be a space in which information (or data) necessary for the operation of the musculoskeletal disease prediction system 100 according to the present invention is stored, and it can be understood that there are no restrictions on physical space.

[0056] The memory unit 140 may store various information about the subject being photographed. For example, the various information about the subject being photographed may include: i) physical information of the subject (joint positions, angles between joints, body center position, movement patterns, etc.), ii) work posture information (type of work posture, identification of inappropriate postures, frequency of posture changes, etc.), iii) activity information during work (movements during work, work speed and rhythm, etc.), iiii) specific environmental information (surrounding environmental conditions, usage status of work tools, arrangement of the workspace, etc.), additional data for risk assessment (work time data, workload, risk factors of the work environment, etc.), and v) individual worker data (worker's physical characteristics, work experience and skill level, etc.).

[0057] Specifically, the memory unit 140 contains information about the subject being photographed, the subject's physical characteristics (e.g., height, weight), the type of movement of the subject, records of posture changes, the arrangement of the specific environment, information on tools and equipment being used, conditions of the specific environment (e.g., spatial size, equipment position), images of the subject and characteristic environment captured by the vision-based data acquisition device, images preprocessed for training the deep learning model 160, augmented images (rotation, enlargement, color adjustment, etc.), bounding boxes (bounding box information for the subject and the specific environment), keypoints (position information of the subject's major joints), coordinate values ​​(x, y) indicating the position of each joint of the subject, angles between joints, and posture evaluations. Information regarding at least one of the following may be matched and stored: calculated values ​​for the subject, the risk score of the subject calculated by the deep learning model 160, risk identification results (low risk, medium risk, high risk), posture suitability evaluation results of the subject, frequency and duration of posture changes, movement patterns of the subject, frequency of repeated movements, history of posture changes performed by the subject, history of changes in the specific environment and changes in the arrangement, weights and parameters of the trained model, training dataset and validation dataset, calculation method for joint angles, criteria for calculating the risk score, system operation settings, posture analysis of the subject, changes in the risk score, records of corrective measures, state analysis of the specific environment, suggestions for improvements, risk score change graph, and posture analysis statistics.

[0058] On the other hand, the control unit 150 can play a role in controlling the overall operation of the musculoskeletal disease prediction system 100 related to the present invention. The control unit 150 can process signals, data, information, etc. that are input or output through the above-mentioned components, or can perform a series of data processing to provide or process appropriate information and functions to the user.

[0059] On the other hand, the present invention provides a method and system for predicting musculoskeletal diseases based on deep learning. More specifically, the present invention provides a deep learning model 160 that can accurately detect the position and movement of workers, work equipment, work tools, and objects such as workpieces at construction (or industrial) sites, thereby minimizing the risk of accidents. Below, we will examine in more detail the method for predicting musculoskeletal diseases based on the deep learning model.

[0060] On the other hand, the present invention can perform the process of receiving images relating to the worker being photographed and the specific environment in which the worker is located (see S310, Figure 2).

[0061] First, the control unit 150 can receive video by distinguishing between the worker being filmed and the specific environment in which the worker is located.

[0062] Here, the video footage of the workers may include various actions and movements performed by the workers within the workspace at the construction site. For example, it may include actions such as a worker lifting an object above their head, a worker bending or straightening their body, or a worker pushing or pulling an object.

[0063] As shown in Figure 3, construction sites may involve various actions that can cause musculoskeletal disorders, such as workers carrying materials, using ladders, and handling heavy tools. Furthermore, images related to specific environments may include not only equipment such as carts, lumber, saws, hammers, paint, drills, ladders, and pickaxes, but also the workpieces and structures themselves.

[0064] In the present invention, the distinction between an operator and the specific environment in which the operator is located can be understood as a process performed to enable the deep learning model 160 to accurately detect objects in various situations.

[0065] Furthermore, the control unit 150 can collect video (or image) data related to safety accidents at construction sites from various sources (e.g., AI-Hub, control system 1000, database, web crawling, API, server linked to musculoskeletal disease prediction system 100, external server, etc.). For example, the control unit 150 can receive visual data including various classes such as workers, transport equipment, storage equipment, and work equipment at construction sites.

[0066] Here, the data on musculoskeletal disorders at construction sites may include at least one of the previously filmed images of the worker and the specific environment in which the worker is located.

[0067] In this case, the received video footage of the worker being filmed, and the video footage of the specific environment in which the worker is located, may be footage taken from various angles to reflect various aspects of the construction site, and the control unit 150 can collect (or receive) video footage separated by object.

[0068] On the other hand, the control unit 150 can process the received video as input to the deep learning model 160.

[0069] The control unit 150 can apply various preprocessing techniques and extensions to the received video in order to optimize the performance of the deep learning model 160. In particular, in this invention, data integrity can be ensured during the data preprocessing process by removing duplicate data, removing outliers, and verifying and correcting bounding boxes 450 and key points 550.

[0070] First, the control unit 150 can perform preprocessing on images related to the construction site and images of working postures.

[0071] In one embodiment, the control unit 150 can improve data quality by identifying and removing images from among images relating to workers and images relating to a specific environment that contain the same object multiple times through the removal of duplicate data in the preprocessing process.

[0072] In this case, the control unit 150 can remove noise from the video using a Gaussian blur filter and non-local means denoising technology in the preprocessing process, thereby improving the accuracy of object detection.

[0073] The Gaussian blur filter is used to remove high-frequency components from an image and reduce localized noise. In particular, in complex environments such as construction sites, background noise can interfere with object detection. Using Gaussian blur to solve this problem smooths out the noise, allowing for clearer detection of object boundaries.

[0074] Furthermore, non-local means denoising is an effective technique for denoising the entire image, reducing unwanted background noise while preserving the features of important objects. This allows for the effective removal of noise caused by various lighting conditions and shadows.

[0075] Furthermore, the control unit 150 can normalize the collected data so that the feature values ​​have the same scale during the training of the deep learning model 160, thereby preventing bias towards specific values ​​during the training process and allowing algorithms such as gradient descent to operate more effectively.

[0076] The control unit 150 can normalize the pixel values ​​contained in images of the worker being photographed and images of the specific environment to a specific range (for example, a range of 0 to 1) in order to improve the performance of the deep learning model 160. Normalization ensures the stability of learning and the consistency of numerical calculations, which can improve the learning speed and performance of the deep learning model 160.

[0077] On the other hand, in the present invention, a pre-trained deep learning model can be used to identify a first bounding box corresponding to the worker and a second bounding box corresponding to a specific environment from the video (see S320, Figure 2).

[0078] A bounding box 450 is a rectangular area used to define the position of a specific object in an image or video. It is widely used in deep learning-based object detection or segmentation, representing the area where an object exists in terms of coordinates and size.

[0079] The control unit 150 can define bounding boxes (or bounding boxes 450) for objects in video or images and utilize these as training data for the deep learning model 160. This allows the deep learning model 160 to learn features and patterns that distinguish between objects (workers) and the background (specific environment).

[0080] For example, as shown in Figure 4, the control unit 150 can process a first bounding box 450 corresponding to the worker and a second bounding box 451 corresponding to the work tool included in the preprocessed image. In this case, the present invention uses a pre-set annotation tool (for example, ROBOFLOW). TM Using this method, bounding boxes 450 and 451 can be applied to the images of the worker and the work tools, respectively.

[0081] In this invention, for the sake of explanation, the first bounding box 450 corresponding to the worker and the second bounding box 451 corresponding to the work tool are not distinguished separately, but are described as a bounding box 450.

[0082] For example, pre-configured annotation tools offer a user-friendly interface and automated annotation capabilities, enabling fast and accurate annotation even for large datasets. Real-time feedback during the annotation process allows for immediate verification of dataset quality, and automated annotation utilizing pre-trained models significantly improves work speed.

[0083] Furthermore, the control unit 150 can verify the quality of the annotated data using a pre-configured annotation tool. For example, during the data quality verification process, the control unit 150 can verify the images related to the worker and the specific environment, and if there are images to be corrected where the bounding box 450 has been incorrectly specified, it can correct the images to be corrected using a pre-configured annotation tool.

[0084] The deep learning model 160 receives video data relating to the worker and a specific environment as input, recognizes a specific area of ​​the image (the worker and the work tools), and uses a pre-configured annotation tool to identify the bounding box 450 surrounding that area.

[0085] As described above, the annotation tool used in the present invention provides support for various formats, allowing data to be easily used in various learning frameworks such as the YOLOv8 model. In particular, the present invention allows for labeling of multiple distinct objects (e.g., people, carts, wood, ladders, drills, hammers, etc.) and converting this into a format that can be immediately applied to training the deep learning model 160, enabling efficient training without a separate conversion process.

[0086] Furthermore, the annotation tool offers ease of use, automation capabilities, support for various formats, and integration with data extension features, enabling efficient annotation of large datasets, real-time data quality review, and rapid error correction. This ensures that annotation work is performed consistently and accurately, ultimately contributing to improved performance of the deep learning model 160.

[0087] On the other hand, the present invention allows for the extraction of key points corresponding to multiple joints within the worker's body based on a first bounding box (see S330, Figure 2).

[0088] The first bounding box 450 may be a bounding box 450 corresponding to the worker.

[0089] The control unit 150 can identify and define the locations of joints included in the worker's body based on the first bounding box 450. In this case, the control unit 150 can use the deep learning model 160 to determine the approximate location of the joints and understand the worker's body structure to identify which parts are joints. In this process, the control unit 150 can explore regions of interest (ROIs) within the bounding box 450 to identify the worker's body structure and calculate the overall positional range of major body parts (head, shoulders, arms, legs, etc.). It can also detect points within the bounding box 450 that could be joint locations and identify them as candidate regions.

[0090] Furthermore, the control unit 150 can extract keypoints 550 at each of the different joint positions identified within the bounding box 450. In this case, the present invention allows for the keypoints 550 to be manipulated using a pre-configured annotation tool (e.g., the ROBOFLOW tool) during the keypoint extraction process.

[0091] For example, as shown in Figure 5, a human skeletal structure may be defined consisting of 16 key points 550, including the head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, right hand, left hip, right hip, left knee, right knee, left foot, and right foot.

[0092] The control unit 150 can use a pre-configured annotation tool to calculate precise positions based on joint candidates identified within the bounding box 450, and extract keypoints 550 by adjusting the positions of different joints. In this process, if a particular keypoint 550 is not visible, it can be displayed with the bounding box 450 hidden. If a keypoint 550 does not exist (for example, not within an object or outside the image), it can be deleted.

[0093] The control unit 150 can adjust the positions corresponding to multiple joints included in the worker's body based on the identified first bounding box 450, and extract the keypoints of the adjusted joint positions in the form of (x, y) or (x, y, z) in a 2D or 3D coordinate system.

[0094] The control unit 150 can adjust the scale, rotation, distortion, etc., through the image alignment process so that the position of each joint of the body is accurately extracted.

[0095] For example, in the case of a 2D image, if the worker's posture is tilted or distortion occurs due to the camera angle, the control unit 150 can adjust the joint positions to the precise locations within the bounding box 450. Furthermore, even if the size and position of each joint are not constant within the image, the control unit 150 can adjust them to the same size and proportions at the precise locations within the bounding box 450. In addition, if the worker rotates in a specific direction, the control unit 150 can apply a rotational transformation based on the relative positions of the joints.

[0096] For example, in the case of a 3D image, the control unit 150 can use the deep learning model 160 to convert the relative positions between each body joint into precise coordinates and adjust them. The deep learning model 160 can learn the range of motion of the joints and the relationships between them, and perform precise position adjustments. Furthermore, the control unit 150 can adjust image distortion or differences in distance between joints to fit the deep learning model 160 using a multi-point linear correction method or geometric transformation.

[0097] A 2D image is a planar image composed of two dimensions (horizontal and vertical). All pixels are defined as fixed coordinates and do not include depth (z-axis) information. The coordinate system has the form (x, y), where each pixel can indicate its horizontal (x) and vertical (y) position.

[0098] A 3D image is composed of three dimensions (width, height, and depth) and includes spatial depth (z-axis) information of the object. This allows the object's position to be defined in real space. The coordinate system has the form (x, y, z), and in addition to width (x) and height (y), it also includes depth (z) information. In addition to pixels, it may also include information such as the spatial arrangement of each point, distance, and texture.

[0099] For example, as shown in Figure 6, key points 550 can be extracted so that the joints located on the worker's body are represented in point form. For example, key points 550 can be extracted by analyzing 16 major joints. Since the key points 550 represent the positions of the worker's joints in point form, movements can be expressed concisely and quantitatively, and this can be used for evaluating work posture, analyzing injury risk, and improving work efficiency.

[0100] After the key points 550 are extracted, the control unit 150 uses an annotation tool (for example, Roboflow). TMBy preprocessing and expanding the data using ), the volume and diversity of the dataset can be increased. For example, in the data preprocessing step, the control unit 150 can scale all images in the dataset to 640 x 640 pixels and apply contrast stretching to automatically adjust the contrast of the images. In this case, the images become sharper and more detailed, enabling image analysis and processing.

[0101] The annotation tool is integrated with data augmentation capabilities, allowing the (deep learning) model to learn more effectively in various environments by augmenting the dataset in various situations after the annotation process. This integrated functionality allows the annotation and augmentation processes to be handled on a single platform, significantly improving work efficiency.

[0102] Furthermore, the annotation tool can continuously improve the accuracy and reliability of the dataset by providing an environment in which annotation errors can be easily corrected. In other words, the present invention makes it possible to effectively construct accurate and diverse datasets for the development of a deep learning model 160 that predicts musculoskeletal diseases in construction sites by utilizing the annotation function and data quality control tool of the annotation tool.

[0103] On the other hand, in the present invention, images can be augmented after data preprocessing and image annotation processes. In the image augmentation process, a series of transformations are applied to the original image to expand the size, diversity, and type of the dataset, thereby effectively augmenting the data without requiring the collection of new data. Here, the data used for augmentation may be at least one of the images before bounding boxes 450 and keypoints 550 are assigned, and / or the images after bounding boxes 450 and keypoints 550 have been assigned. In the present invention, the data to be augmented is not limited to just one of these.

[0104] The control unit 150 can construct a training dataset by augmenting the images (or annotated images) of the worker being photographed and the specific environment in which the worker is located, using a pre-configured augmentation technique. Here, the pre-configured augmentation technique may be a technique selected to transform the data in order to more appropriately reflect the various actions of the worker and environmental conditions.

[0105] On the other hand, the present invention allows for the calculation of angles between multiple joints using extracted key points, and the calculation of a risk score for the calculated angles between multiple joints (see S340, Figure 2).

[0106] The control unit 150 can manage the data flow and calculate angles based on the coordinates of the key points 550. For example, the extracted key points 550 may be represented in coordinate form (x, y). The control unit 150 can calculate vectors between the key points 550 and mathematically define the movement of the joints.

[0107] The control unit 150 can simplify joint calculations by projecting the positions of the extracted keypoints 550 onto a 2D plane (image coordinates). This allows for the calculation of joint angles from a 2D image without 3D data. For example, inverse trigonometric functions may be used for the calculation.

[0108] For example, the control unit 150 can define a vector using the coordinates of two adjacent joints connected around the position of each key point 550. For example, as shown in Figure 7, (a) shows the angle when the arm is bent at the key points 550 located at the shoulder 711, elbow 712, and wrist 713, and (b) shows the angle when the arm is extended at the key points 550 located at the shoulder 711, elbow 712, and wrist 713. The shoulder 711, elbow 712, and wrist 713 are just one example of adjacent joints, and various joints may be selected.

[0109] The control unit 150 can define coordinates such as (x1, y1) for the shoulder 711, (x2, y2) for the elbow 712, and (x3, y3) for the wrist 713. In this process, the vector between each key point 550 may be defined as the difference between two points. For example, it may be defined as the first vector v1 (elbow 712 - shoulder 711) and the second vector v2 (elbow 712 - wrist 713). In this case, the control unit 150 can calculate the magnitudes of the first vector v1 = (x2 - x1, y2 - y1) and the second vector v2 = (x3 - x2, y3 - y2) using the following [Mathematical Formula 1].

[0110]

number

[0111] Furthermore, the control unit 150 controls the joint angle (A i The magnitude of ) can be calculated using the following [Mathematical Formula 2].

[0112]

number

[0113] On the other hand, the musculoskeletal disease prediction system 100 according to the present invention may include a joint angle calculation algorithm. The joint angle calculation algorithm may be an algorithm that outputs an angle when a key point 550 is input. Specifically, this process is performed by a control unit 150, which can store and manage the extracted key point coordinates. Three indices of key points 550 necessary for calculating the angle are defined. For example, they can be specified as a first index (x1, y1), a second index (x2, y2), and a third index (x3, y3). In this case, the control unit 150 can use a preset function to obtain a first vector ((x1, y1), (x2, y2)) and a second vector ((x1, y1), (x3, y3)) as vectors for the first index (x1, y1), the second index (x2, y2), and the third index (x3, y3). Then, using a preset function, the angle between the first vector and the second vector can be output. The joint angle calculation algorithm according to the present invention can be expressed as follows.

number

[0114] Furthermore, the control unit 150 can convert the calculated angles between multiple joints into a format that the deep learning model 160 can analyze and process, and input it into the model.

[0115] The control unit 150 can vectorize the calculated joint angles to fit the input format of the deep learning model 160 and normalize them to a certain range (e.g., 0, 1). For example, if the maximum angle is 180 degrees, 90 degrees may be converted to 0.5 and 45 degrees to 0.25. In this process, if there are missing joint angles (e.g., keypoints are not accurately extracted), missing value processing can be performed. In this case, the control unit 150 can replace the missing values ​​with the average value of the corresponding joint angles and replace the missing values ​​with 0. The control unit 150 can standardize the size of the input data to match the deep learning model 160. For example, if the input size of the model is N=20, the number of joint angle data can be adjusted to 20. If there is a shortage, the missing data can be added to meet the data size requirement, and if there is an excess, unnecessary data can be removed to adjust to an appropriate size.

[0116] Furthermore, the control unit 150 can analyze the input joint angle data using the deep learning model 160 and calculate a risk score 800 related to the working posture. The risk score 800 may be used as an indicator to evaluate the stability, efficiency, and likelihood of injury of the working posture.

[0117] Furthermore, the deep learning model 160 may have a structure that can calculate work posture and a risk score of 800. The deep learning model 160 can transmit the input data to various layers and analyze work posture patterns and risk elements.

[0118] Specifically, the deep learning model 160 can process the input data to extract features related to the work posture, perform calculations to evaluate the risk of the posture based on the feature data, and generate regression or classification results that calculate a risk score of 800.

[0119] For example, the deep learning model 160 may be designed to calculate a risk score of 800 on a value between 0 and 7. The deep learning model 160 may also be designed to analyze the risk elements of work posture based on the input data and ultimately calculate a risk score of 800 by utilizing the extracted features and joint angles.

[0120] On the other hand, the present invention allows for the process of identifying the degree of risk to workers based on the calculated risk score (see S350, Figure 2).

[0121] The deep learning model 160, upon inputting the acquired risk score 800, can identify the degree of risk to the worker and evaluate the worker's condition. By identifying the risk, the stability of the work posture and the likelihood of injury can be quantitatively evaluated.

[0122] The deep learning model 160 can identify the input risk score 800 as first-degree risk, second-degree risk, and third-degree risk based on pre-defined criteria.

[0123] For example, a pre-set standard can be specified using the range corresponding to the second level of risk as the standard.

[0124] The deep learning model 160 can determine that the working posture contains risk factors and that there is a possibility of injury. For example, if the risk score is between 3 and 4, it can be identified as a second-level risk (medium risk).

[0125] The deep learning model 160 can determine that the working posture is stable and the risk of injury is minimal. For example, if the risk score is between 0 and 3, it can be identified as Level 1 risk (low risk).

[0126] The deep learning model 160 can determine that the working posture is extremely dangerous and requires immediate improvement. For example, if the risk score is greater than 4 but less than or equal to 7, it can be identified as the third level of risk (high risk).

[0127] Furthermore, the control unit 150 can flexibly select and calculate joint angles according to various work environments, thereby identifying the degree of risk. In this case, the control unit 150 can identify high-risk postures by analyzing the angle changes of different joints and provide information on countermeasures such as risk warnings and recommendations for improvement.

[0128] On the other hand, the present invention can perform a process to generate information on countermeasures corresponding to the identified level of risk (see S360, Figure 2).

[0129] The control unit 150 can generate information on countermeasures to ensure worker safety and reduce the possibility of injury based on the identified first risk level (low risk), second risk level (medium risk), and third risk level (high risk). By providing specific countermeasures information according to the identified risk level, the control unit 150 can improve the environment and enhance worker efficiency and safety.

[0130] The information on countermeasures generated for each of the risk levels identified in this invention may differ.

[0131] Each identified risk level may have matching information for different response measures. Here, the matching information for different response measures may include first response measures matched to the first risk level, second response measures matched to the second risk level, and third response measures matched to the third risk level.

[0132] For example, information on different response measures may include at least one of the following: i) monitoring and warning of work posture, provision of basic training, adjustment of work time, adjustment of the environment, and maintenance of work records; ii) guidance on real-time correction, recommendation of equipment use, improvement of the work environment, additional safety training, recommendation of work interruption, and restriction of repetitive work; iii) immediate warning and work interruption, provision of work posture correction programs, notification of potential injury, equipment support, confirmation of workers' health status, and periodic monitoring; iv) adjustment of temperature, humidity, and lighting, wearing of safety equipment, confirmation of equipment status, and marking of hazardous areas; v) real-time warnings based on deep learning, provision of AR / VR simulations, and data-driven improvements; vi) optimization of work processes, strengthening of teamwork, updating of worker training programs and work regulations; vii) support for emergency medical care, accident reporting systems, work stoppage, and evaluation. However, the information included in information on different response measures is not necessarily limited to these, and various other measures may be included in addition to those mentioned.

[0133] In this case, the measures taken based on the first response measures information, the second response measures information, and the third response measures information may be different from each other.

[0134] For example, as shown in Figure 8, the control unit 150 can generate appropriate warning messages 810 and notifications based on the risk score 800. (a) indicates that the worker is lifting an object in a posture that puts strain on the waist. In this case, a notification may be provided along with a warning message 810 that reads, "Please assume the correct posture with your waist straight and your knees bent." Here, the notification can be provided as a warning sound along with a warning display to inform the worker of the dangerous situation.

[0135] The control unit 150 can visually guide the worker to the corrected posture 850 proposed by the deep learning model 160. The control unit 150 can also generate an explanatory message 820 that emphasizes the importance of the correct working posture along with the corrected posture. For example, (b) shows that the corrected posture is conveyed as an image. In this case, an explanatory message 820 such as "This posture can prevent back injuries" may be provided along with the corrected posture image 850. The control unit 150 can make the corrected posture easy for the worker to understand by generating a 3D simulation image or animation to help the worker understand it intuitively, and by using image processing technology to compare the corrected posture with the current posture.

[0136] The control unit 150 can provide learning feedback to the deep learning model 160 based on the data stored in the memory unit 140.

[0137] Furthermore, the control unit 150 can continuously track the worker's working conditions and provide immediate warnings when new hazardous situations arise. Examples of hazardous situations include incorrect posture and situations where physical strain occurs due to the work object or tools.

[0138] Furthermore, the control unit 150 can generate individually optimized corrective information based on the worker's work pattern and data previously stored in the memory unit 140. For example, it may provide more detailed corrective posture guidance to novice workers and concise warnings and corrective messages to experienced workers.

[0139] Furthermore, the control unit 150 can suggest adjustments to the work environment. For example, it can suggest adjustments to the height of the workbench, the arrangement of equipment, and the use of auxiliary equipment.

[0140] As described above, the deep learning-based musculoskeletal disease prediction method and system according to the present invention can collect video footage of the worker and work environment being filmed and provide the results of analyzing this footage. Since data collection is possible using only a camera device, initial installation and maintenance costs can be reduced compared to sensor-based systems.

[0141] Furthermore, the deep learning-based musculoskeletal disease prediction method and system according to the present invention can analyze working posture by detecting joint positions using the deep learning model 160 and calculating the angles between joints. This allows for the detection of joint positions solely from video footage without the need for sensors or additional equipment, and enables real-time tracking of joint positions even when the worker changes posture or performs complex movements.

[0142] Furthermore, the deep learning-based musculoskeletal disease prediction method and system according to the present invention uses joint angles as input and provides warnings and improvement guidelines accordingly. This makes it possible to identify the possibility of musculoskeletal disease occurring early and take preventive measures.

[0143] Furthermore, the deep learning-based musculoskeletal disease prediction method and system according to the present invention predicts dangerous postures early and supports workers in recognizing them in real time. As a result, workers can immediately correct dangerous postures and maintain safe postures, thereby preventing the occurrence of musculoskeletal diseases.

[0144] On the other hand, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a medium (or recording medium) that is readable by such a computer.

[0145] Furthermore, the present invention described above can be implemented as computer-readable code or instruction words on a medium on which a program is recorded. In other words, the present invention can be provided in the form of a program.

[0146] On the other hand, computer-readable media include all kinds of recording devices that store data readable by a computer system. Examples of computer-readable media include HDDs (Hard Disk Drives), SSDs (Solid State Disks), SSDs (Silicon Disk Drives), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0147] Furthermore, the computer-readable medium may include storage and may be a server or cloud storage accessible by electronic devices via communication. In this case, the computer can download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0148] Furthermore, in this invention, the computer is an electronic device equipped with a processor, i.e., a CPU (Central Processing Unit), and its type is not particularly limited.

[0149] On the other hand, the above detailed description should not be interpreted as limiting in any way, but should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of the equivalents of the present invention are included within the scope of the present invention.

Claims

1. The steps include receiving video footage of the worker to be filmed and the specific environment in which the worker is located, The steps include using a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video, Based on the first bounding box, the steps include extracting keypoints corresponding to multiple joints within the worker's body, The steps include: calculating the angles between the multiple joints using the extracted key points, and calculating a risk score for the calculated angles between the multiple joints; A step of identifying the degree of risk to the worker based on the calculated risk score, A method for predicting musculoskeletal diseases based on deep learning, characterized by comprising the step of generating information on countermeasures corresponding to the identified risk level.

2. The aforementioned video is This includes at least one of the worker's posture, movements, movement patterns, the worker's body, and the specific environment. The aforementioned specific environment is The deep learning-based method for predicting musculoskeletal disorders according to claim 1, characterized in that it includes at least one of the following: the workspace where the worker is working, and the location of tools, equipment, and workpieces located in the workspace.

3. In the step of identifying the bounding box, The received video is processed as input to the deep learning model. The deep learning model analyzes the worker and the specific environment from the video, A method for predicting musculoskeletal disorders based on deep learning, according to claim 2, characterized in that a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment are identified using the analysis results.

4. In the step of extracting the aforementioned key points, Based on the first bounding box, the joint positions corresponding to the plurality of joints are adjusted according to the physical structure of the worker. A method for predicting musculoskeletal disorders based on deep learning according to claim 3, characterized in that key points are extracted from the joint positions adjusted according to the aforementioned body structure.

5. The calculation of the aforementioned joint angle is as follows: Based on the extracted keypoints, a vector corresponding to the joint position is generated. A method for predicting musculoskeletal diseases based on deep learning, according to claim 4, characterized in that the joint angle between the joint positions is calculated using the generated vector.

6. The calculation of the aforementioned risk score is as follows: Using the calculated joint angles, the worker's risk level is predicted. A method for predicting musculoskeletal diseases based on deep learning, according to claim 5, characterized in that the risk score for the joint angle is calculated based on the predicted results.

7. In the step of identifying the degree of risk, The risk score for the calculated joint angle is processed as input to the deep learning model. In the deep learning model described above, based on pre-set criteria, the risk level based on the risk score is identified from among multiple risk levels. The aforementioned multiple levels of risk are, A first risk level identified based on the fact that the risk score does not meet the predetermined criteria, A second risk level is identified based on the fact that the risk score meets the predetermined criteria, A method for predicting musculoskeletal diseases based on deep learning according to claim 6, characterized in that it includes at least one of the following: a third risk level identified based on the risk score exceeding the predetermined criteria.

8. Each of the aforementioned multiple risk levels has matching information on different countermeasures, The aforementioned information on different countermeasures is, This includes at least one of the following: first response information matched to the first risk level, second response information matched to the second risk level, and third response information matched to the third risk level. In the step of generating the aforementioned countermeasure information, If the identified risk level corresponds to at least one of the second risk level and the third risk level, which are different from the first risk level, A method for predicting musculoskeletal diseases based on deep learning, as described in claim 7, characterized by generating the second or third countermeasure information for predicting the musculoskeletal disease.

9. A deep learning-based musculoskeletal disease prediction system is A data collection unit that receives video footage of the worker being filmed and the specific environment in which the worker is located, The system includes a control unit that uses a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video, The control unit, Based on the first bounding box, keypoints corresponding to multiple joints in the worker's body are extracted. Using the extracted keypoints, the angles between the multiple joints are calculated, and a risk score is calculated for the calculated angles between the multiple joints. Based on the calculated risk score, the degree of risk to the worker is identified. A deep learning-based musculoskeletal disease prediction system characterized by generating information on countermeasures corresponding to the identified risk level.

10. A program executed by one or more processes in an electronic device and stored on a computer-readable recording medium, The aforementioned program, The steps include receiving video footage of the worker to be filmed and the specific environment in which the worker is located, The steps include using a pre-trained deep learning model to identify a first bounding box corresponding to the worker and a second bounding box corresponding to the specific environment from the video, Based on the first bounding box, the steps include extracting keypoints corresponding to multiple joints within the worker's body, The steps include: calculating the angles between the multiple joints using the extracted key points, and calculating a risk score for the calculated angles between the multiple joints; A step of identifying the degree of risk to the worker based on the calculated risk score, A program stored on a computer-readable recording medium, characterized by including a command word to perform the step of generating information on countermeasures corresponding to the identified level of risk.

Images