Work technique recognition / analysis system and method

Abstract

To provide a work technique recognition / analysis system and method which include a hand technique motion analysis model for evaluating accuracy of motions of at least one hand part of a worker in a work area during work.SOLUTION: A work technique recognition / analysis system 1 includes an image acquisition module, a recognition module, and a motion analysis module. The image acquisition module is provided in a work area and is used to capture a series of images of motions of at least one hand part of a worker in the work area during work. The recognition module is connected to the image acquisition module, includes a two-dimensional joint recognition model, a joint three-dimensional coordinate recognition model, and a hand part skeleton joint model direction comprehensive optimization model, and is used to recognize a joint position of the worker's hand part based on the series of images for generating hand part joint position coordinates. The motion analysis module receives the hand part joint position coordinates, and performs analysis and comparison based on the hand part joint position coordinates of the worker and a standard work technique parameter set for obtaining analysis and comparison results.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a work skill recognition and analysis system and method, and particularly to a work skill recognition and analysis system and method related to machine learning.

Background Art

[0002] Cell therapy is a method that utilizes biology, cytology, and molecular biology to repair, replace, or improve the functions of damaged or lost tissues and organs. In recent years, with the enactment of the "Act on the Control of the Implementation or Use of Specified Medical Technologies, Tests, Verifications, and Medical Devices (abbreviated as the Special Control Act)", the development of cell therapy has been accelerating. Cell therapy mainly involves administering artificially processed "human cells" into the patient's body to grow desired cells (e.g., knee cartilage) in the patient's body or attack diseased cells (e.g., cancer cells) in the patient's body. Specifically, in current common cell therapy, mainly cells in the patient's body are collected, activated and proliferated in a sterile and dust-free working space, and a cell preparation is manufactured and administered into the patient's body to achieve the effects of treatment and repair.

[0003] In the manufacturing process of cell preparations used for cell therapy, it is necessary to maintain the working environment at a certain level of cleanliness. Both the workbench and equipment must be in a sterile state so that the cell preparation is not contaminated and the efficacy of cell therapy is not affected. In addition, the operator's work skills, work procedures, and work habits are closely related to the quality of the cell preparation. For example, due to their work habits, operators frequently move their hands above the cell culture dish during the work process, causing pathogenic bacteria on the hands to fall into the cell culture dish. Or, due to inappropriate work by the operator, the movement of the hand when using a pipette to aspirate the cell culture fluid becomes too fast, resulting in the rapid inflow of the cell culture fluid into the pipette and the mixing of dust and foreign matter in the pipette into the cell culture dish. All of the above factors may cause contamination in the manufacturing process of cell preparations. Also, in the process of subculturing cells, if the cell collection process is incorrect (for example, if cells are accidentally aspirated when discarding old cell culture fluid, or if the volume of aspirated cells is inaccurate), the growth density and confluence of the cultured cells will vary from batch to batch, affecting the quality of cell culture and causing batch-to-batch errors.

[0004] In addition, during the work process, operators may not notice abnormalities due to the continuity of their work movements or their own work habits. Also, since pathogenic bacteria and dust in the working environment are invisible to the naked eye, abnormalities cannot be detected promptly, and often, it is not until the final collection after several days that the product contamination in the manufacturing process is noticed for the first time. Therefore, in summary, in the work process related to cell therapy, the accuracy of the operator's work skills, work procedures, and work habits all directly affect the quality of the product. However, currently, there is no hand recognition technology in the market to confirm the accuracy of work related to cell therapy.

[0005] In addition, in the prior art, there is a hand recognition system that analyzes and evaluates the accuracy of an operation based on the hand joint points of the operation movements within a single image. However, in the actual work process, the hands within a single image may overlap, or blind spots of the recognition system may occur, resulting in distortion in the distance recognition between hand joint points and inaccurate analysis results in many cases. Further, in the conventional hand recognition system, since the distance recognition of the hand joint points within a single image is distorted, even more distortion occurs when used for the recognition and detection of consecutive images, and there is a risk of further reduction in accuracy.

[0006] Therefore, in order to solve the above-described conventional problems, it is necessary to research and develop a new work technique recognition and analysis system and method.

Summary of the Invention

Problems to be Solved by the Invention

[0007] In view of the above, the present invention provides a work technique recognition and analysis system and method in order to solve the above-described conventional problems.

Means for Solving the Problems

[0008] The present invention provides a work skill recognition and analysis system that can be used to evaluate the accuracy of the movements of at least one hand in the work area of a worker during work. The work skill recognition and analysis system includes an image acquisition module, a recognition module, and a motion analysis module. The image acquisition module is provided in the work area. The image acquisition module is used to capture continuous images of the movements of at least one hand in the work area of the worker during work. The recognition module is connected to the image acquisition module. The recognition module includes a two-dimensional joint recognition model, a joint three-dimensional coordinate recognition model, and a comprehensive optimization model for the direction of the hand skeleton joint model. The two-dimensional joint recognition model is used to recognize the joint positions of the worker's hand based on the continuous images. The joint three-dimensional coordinate recognition model is connected to the two-dimensional joint recognition model and is used to generate hand joint position coordinates corresponding to the joint positions of the worker's hand from multiple continuous images at different angles of the movements of at least one hand in the work area of the worker captured by a plurality of image acquisition modules. The hand joint position coordinates are three-dimensional coordinates. The comprehensive optimization model for the direction of the hand skeleton joint model is connected to the joint three-dimensional coordinate recognition model and is used to improve the accuracy of the hand joint position coordinates by performing a comprehensive optimization process on the hand joint position coordinates based on the constraints of the worker's phalange length and the constraints of the phalange joints. The two-dimensional joint recognition model, the joint three-dimensional coordinate recognition model, and the comprehensive optimization model for the direction of the hand skeleton joint model are constructed by training a work skill dataset using a first machine learning method. Further, the motion analysis module is connected to the recognition module. The motion analysis module receives the hand joint position coordinates, performs analysis and comparison based on the hand joint position coordinates of the worker and a standard work skill parameter set, and obtains an analysis and comparison result, thereby including a skill motion analysis model used to evaluate the accuracy of the movements of at least one hand in the work area of the worker during work. The skill motion analysis model is constructed by training a standard work skill dataset using a second machine learning method.

[0009] For three-dimensional coordinates, the intersection point of the emission lines from each image acquisition module is calculated by triangulation, or the midpoint of the shortest distance of the emission lines from each image acquisition module is calculated as an approximation of the intersection point by the midpoint method.

[0010] The motion analysis module further includes a motion type recognition model used to recognize and classify various types of hand motions by a third machine learning method based on the hand joint position coordinates in the continuous images captured by the image acquisition module.

[0011] The motion analysis module further includes a motion / sequence accuracy recognition model used to determine whether the motion and work procedure of at least one hand of the operator are correct by analyzing and comparing the hand joint position coordinates and the standard work procedure parameter set.

[0012] The work procedure recognition / analysis system further includes a warning module. The warning module is connected to the motion analysis module and is used to prompt the operator to pay attention by issuing a notification message based on the analysis / comparison result obtained by analyzing and comparing the hand joint position coordinates in the work area of the operator during work and the standard work procedure parameter set.

[0013] The notification message further includes an audio notification and a lamp notification.

[0014] When the analysis / comparison result obtained by analyzing and comparing the hand joint position coordinates in the work area of the operator during work and the standard work procedure parameter set is inconsistent, the warning module issues an audio notification. Note that the audio notification is an error warning.

[0015] The image acquisition module includes a camera, and the camera is provided in the work area.

[0016] The work area includes an experimental workbench, a sterile workbench, and a cell workbench.

[0017] The first machine learning method, the second machine learning method, and the third machine learning method further include at least one of artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), decision trees, support vector machines (SVMs), random forests, k-nearest neighbors (KNN), k-means clustering, principal component analysis (PCA), linear regression, logistic regression, gradient boosting machines, deep belief networks (DBNs), recursive neural networks (RecNNs), reinforcement learning, autoencoders, Gaussian processes, and complex neural networks.

[0018] The present invention further provides a work skill recognition and analysis method used to evaluate the accuracy of at least one hand movement in the work area of an operator during work. The work skill recognition and analysis method includes the following steps.

[0019] The image acquisition module captures continuous images of at least one hand movement in the work area of an operator during work.

[0020] The two-dimensional joint recognition model of the recognition module recognizes the joint positions of the operator's hand based on the continuous images.

[0021] The joint three-dimensional coordinate recognition model of the recognition module generates hand joint position coordinates corresponding to the joint positions of the operator's hand based on a plurality of consecutive images from different angles of at least one hand movement in the operator's working area captured by a plurality of image acquisition modules. The hand joint position coordinates are three-dimensional coordinates.

[0022] The comprehensive optimization model of the hand skeleton joint model direction of the recognition module improves the accuracy of the hand joint position coordinates by performing comprehensive optimization processing on the hand joint position coordinates based on the constraints of the operator's phalanx length and phalanx joints. The two-dimensional joint recognition model, the joint three-dimensional coordinate recognition model, and the comprehensive optimization model of the hand skeleton joint model direction are constructed by training the working skill dataset using the first machine learning method.

[0023] The skill movement analysis model of the movement analysis module receives the hand joint position coordinates, performs analysis and comparison based on the operator's hand joint position coordinates and the standard working skill parameter set, and obtains the analysis and comparison results, thereby evaluating the accuracy of at least one hand movement in the operator's working area during work. The skill movement analysis model is constructed by training the standard working skill dataset using the second machine learning method.

[0024] For three-dimensional coordinates, the intersection point of the projection lines from each image acquisition module is calculated by the triangulation method, or the midpoint of the shortest distance of the projection lines from each image acquisition module is calculated as an approximation of the intersection point by the midpoint method.

[0025] The working skill recognition and analysis method further includes the step of the movement type recognition model of the movement analysis module recognizing and classifying various types of hand movements by the third machine learning method based on the hand joint position coordinates in the consecutive images captured by the image acquisition module.

[0026] The operation technique recognition and analysis method further includes a step in which the operation and sequence accuracy recognition model of the motion analysis module analyzes and compares the hand joint position coordinates and the standard operation technique parameter set to determine whether the operation and work procedure of at least one hand of the operator are correct.

[0027] The operation technique recognition and analysis method further includes a step in which the attention arousal module issues a notification message based on the analysis and comparison results obtained by analyzing and comparing the hand joint position coordinates and the standard operation technique parameter set in the work area of the operator during work, thereby prompting the operator's attention.

[0028] The notification message further includes an audio notification and a lamp notification.

[0029] When the analysis and comparison results obtained by analyzing and comparing the hand joint position coordinates and the standard operation technique parameter set in the work area of the operator during work are inconsistent, the attention arousal module issues an audio notification. Note that the audio notification is an error attention warning.

[0030] The image acquisition module includes a camera, and the camera is provided in the work area.

[0031] The work area includes an experimental workbench, a sterile workbench, and a cell workbench.

[0032] The first machine learning method, the second machine learning method, and the third machine learning method further include at least one of artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), decision trees, support vector machines (SVMs), random forests, k-nearest neighbors (KNN), k-means clustering, principal component analysis (PCA), linear regression, logistic regression, gradient boosting machines, deep belief networks (DBNs), recursive neural networks (RecNNs), reinforcement learning, autoencoders, Gaussian processes, and complex neural networks.

Advantages of the Invention

[0033] In summary, the present invention combines artificial intelligence and a hand recognition system and uses them to analyze the accuracy of the movements of an operator's hand, thereby providing a work skill recognition and analysis system and method for improving product quality in the manufacturing process and the consistency of products in the manufacturing process. By performing comprehensive optimization processing based on the finger bone lengths and joint constraints of the operator using a comprehensive optimization model for the hand bone joint model direction, the recognition accuracy of the joint positions of the hand is improved, and the distortion in the recognition process is reduced. In addition, based on the analysis and comparison results of the hand joint position coordinates and standard work skill parameter sets of the operator during the cell culture period, a notification message is issued by the attention module to alert the operator whether the current movement is correct. Thereby, the operator is promptly alerted to correct the movement of the hand, improving the accuracy of the movement of the hand.

Brief Description of the Drawings

[0034]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0035] Subsequently, in order to make the advantages, spirit, and features of the present invention more easily and clearly understandable, specific examples will be used and described in detail with reference to the drawings. It should be noted that these specific examples are merely representative specific examples of the present invention, and the specific methods, devices, conditions, materials, etc. illustrated do not limit the present invention or the corresponding specific examples. Moreover, each component in the drawings is used only to represent their relative positions and is not described based on actual ratios. Also, the step numbers of the present invention are only for distinguishing different steps and do not represent the order of the steps. The above is explained in advance.

[0036] Refer to FIG. 1. FIG. 1 shows a functional block diagram of a work skill recognition and analysis system 1 in a specific embodiment of the present invention. The work skill recognition and analysis system 1 in this specific embodiment can be used to evaluate the accuracy of at least one hand movement in the work area of a worker during work. The work skill recognition and analysis system 1 in this specific embodiment includes an image acquisition module 11, a recognition module 12, and a motion analysis module 13. The image acquisition module 11 is provided in the work area. The image acquisition module 11 is used to capture continuous images of at least one hand movement in the work area of a worker during work. The recognition module 12 is connected to the image acquisition module 11. The recognition module 12 includes a two-dimensional joint recognition model, a joint three-dimensional coordinate recognition model 122, and a hand skeleton joint model direction comprehensive optimization model 123. The two-dimensional joint recognition model 121 is used to recognize the joint positions of the worker's hand based on the continuous images. The joint three-dimensional coordinate recognition model 122 is connected to the two-dimensional joint recognition model 121 and is used to generate hand joint position coordinates corresponding to the joint positions of the worker's hand from multiple continuous images at different angles of at least one hand movement in the work area of the worker captured by a plurality of image acquisition modules. The hand joint position coordinates are three-dimensional coordinates. The hand skeleton joint model direction comprehensive optimization model 123 is connected to the joint three-dimensional coordinate recognition model 122 and is used to improve the accuracy of the hand joint position coordinates and reduce the distortion in the recognition process by performing a comprehensive optimization process on the hand joint position coordinates based on the constraints of the worker's finger bone length and finger joint constraints. The two-dimensional joint recognition model 121, the joint three-dimensional coordinate recognition model 122, and the hand skeleton joint model direction comprehensive optimization model 123 are constructed by training a work skill dataset using a first machine learning method. Further, the motion analysis module 13 is connected to the recognition module 12. The motion analysis module 13 includes a skill motion analysis model 131 that receives the hand joint position coordinates, performs analysis and comparison based on the worker's hand joint position coordinates and a standard work skill parameter set, and obtains an analysis and comparison result to evaluate the accuracy of at least one hand movement in the work area of a worker during work.The manual operation analysis model 131 is constructed by training a standard operation manual dataset using a second machine learning method.

[0037] In this specific embodiment, for three-dimensional coordinates, the intersection point of the emission lines from each image acquisition module is calculated by triangulation, or the midpoint of the shortest distance of the emission lines from each image acquisition module is calculated as an approximation of the intersection point by the midpoint method. In addition, the batch with the highest final quality may be selected as the standard operation manual parameter set, and the standard operation process may be set based on the manual parameters of the hand movements corresponding to that batch. However, in practice, it is not limited to this, and the user may select and set the standard operation manual parameter set based on their own needs or work requirements. Also, in this specific embodiment, the operation manual dataset may include the manual parameters of accurate hand movements during the process of various workers performing various operations. In addition, in practice, although the sizes, widths, and distances between joints of the palms of the majority of workers do not match, the overall hand structures are similar. Therefore, by implementing a machine learning method based on the operation manual dataset, the optimal operation manual parameters (i.e., the standard operation manual parameter set) of each worker can be obtained.

[0038] In practice, the image acquisition module 11 may be a camera. Moreover, the camera may be provided in the working area. The working area may include an experimental workbench, a sterile workbench, a cell workbench, or other spaces and stages where the work of the manufacturing process can be carried out. The joint three-dimensional coordinate recognition model 122 acquires images of the operator's hand from different angles by means of a plurality of cameras. The joint three-dimensional coordinate recognition model 122 first reconstructs the three-dimensional coordinates of the joints by calculating the intersection points between the emission lines from each camera by means of triangulation. Also, when the emission lines do not intersect exactly, the midpoint method is used to calculate the midpoint of the shortest distance as an approximation of the intersection point, enabling accurate and stable three-dimensional positioning. In addition, the hand skeleton joint model direction comprehensive optimization model 123 performs comprehensive optimization processing based on the operator's phalange length and joint constraints, thereby improving the recognition accuracy of the joint position and reducing the distortion in the recognition process. Regarding the optimization process, first, two-dimensional joint points are acquired from images at a plurality of angles and converted into a three-dimensional model by means of triangulation. Next, comprehensive correction is performed based on the phalange length and joint constraints, and it is associated with the original image by means of model pose control and projection functions. Through these processes, finally, an optimized three-dimensional hand skeleton model is generated. According to the process of this module, not only is the recognition accuracy of the joint position improved, but also the distortion in the recognition process is reduced, thus ensuring the stability and accuracy of the model.

[0039] Refer to FIGS. 1 and 3 together. FIG. 3 shows a flowchart of the steps of the work skill recognition and analysis method in a specific embodiment of the present invention. The steps of the work skill recognition and analysis method in FIG. 3 can be achieved by the work skill recognition and analysis system 1 in FIG. 1. As shown in FIG. 3, in this specific embodiment, the work skill recognition and analysis method includes the following steps.

[0040] Step S1: The image acquisition module 11 captures continuous images of at least one hand movement of the operator in the working area during work.

[0041] Step S2: The two-dimensional joint recognition model 121 of the recognition module 12 recognizes the hand joint position coordinates of the operator based on the continuous images.

[0042] Step S3: The joint three-dimensional coordinate recognition model 122 of the recognition module 12 generates hand joint position coordinates corresponding to the joint positions of the operator's hand from a plurality of continuous images at different angles of at least one hand movement in the working area of the operator captured by the plurality of image acquisition modules 11. The hand joint position coordinates are three-dimensional coordinates.

[0043] Step S4: The hand skeleton joint model direction comprehensive optimization model 123 of the recognition module 12 improves the accuracy of the hand joint position coordinates by performing comprehensive optimization processing on the hand joint position coordinates based on the constraints of the operator's finger bone length and finger joint constraints.

[0044] Step S5: The manual operation analysis model 131 of the motion analysis module 13 receives the hand joint position coordinates, performs analysis and comparison based on the hand joint position coordinates of the operator and the standard operation manual parameter set, and obtains the analysis and comparison results, thereby evaluating the accuracy of at least one hand movement in the working area of the operator during operation.

[0045] The two-dimensional joint recognition model 121, the joint three-dimensional coordinate recognition model 122, and the hand skeleton joint model direction comprehensive optimization model 123 are constructed by training a manual operation data set using the first machine learning method. Also, the manual operation analysis model 131 is constructed by training a standard operation manual data set using the second machine learning method.

[0046] Refer to FIG. 2. FIG. 2 shows a functional block diagram of the work skill recognition and analysis system 2 in another specific embodiment of the present invention. This specific embodiment is different from the above-described specific embodiment in the following points. That is, the work skill recognition and analysis system 2 in this specific embodiment further includes an attention arousal module 14. The attention arousal module 14 is connected to the motion analysis module 13 and is used to prompt the operator's attention by issuing a notification message based on the analysis and comparison results obtained by analyzing and comparing the hand joint position coordinates in the operator's work area during work with the standard work skill parameter set. In this specific embodiment, the notification message may further include an audio notification and a lamp notification. When the analysis and comparison results obtained by analyzing and comparing the hand joint position coordinates in the operator's work area during work with the standard work skill parameter set are inconsistent, the attention arousal module 14 issues an audio notification. Note that the audio notification is an error attention warning. On the other hand, when the analysis and comparison results are consistent, the attention arousal module 14 does not issue an audio notification. In addition, when the analysis and comparison results are consistent, the lamp notification may present green, and when the analysis and comparison results are inconsistent, the lamp notification may present red. However, in practice, it is not limited to this, and the user may set the notification message according to their own needs or the requirements of the work process.

[0047] Refer to FIGS. 2 and 4 together. FIG. 4 shows a flowchart of the steps of the work skill recognition and analysis method in another specific embodiment of the present invention. The steps of the work skill recognition and analysis method in FIG. 4 can be achieved by the work skill recognition and analysis system 2 in FIG. 2. As shown in FIG. 4, in this specific embodiment, the work skill recognition and analysis method further includes step S6 that is executed after step S5.

[0048] Step S6: The attention arousal module 14 prompts the operator's attention by issuing a notification message based on the analysis and comparison results obtained by analyzing and comparing the hand joint position coordinates in the operator's work area during work with the standard work skill parameter set.

[0049] Note that, as a point to be noted, since the other modules, models, and corresponding functions of the work procedure recognition and analysis system 2 in this specific embodiment are almost the same as the corresponding modules in the above-described specific embodiment, they will not be described in detail again here.

[0050] In another specific embodiment, the operation analysis module 13 may further include other models. Here, refer to FIG. 5. FIG. 5 shows a functional block diagram of the operation analysis module 13 in FIG. 1. As shown in FIG. 5, the operation analysis module 13 may further include an operation type recognition model 132 and an operation / sequence accuracy recognition model 133. These both improve the accuracy of hand movement recognition and the judgment of work accuracy by AI technology. The operation type recognition model 132 exclusively analyzes the continuous images captured by the image acquisition module. The operation type recognition model 132 can recognize and classify various types of hand movements based on a trained hand movement dataset by machine learning methods, and can quickly and accurately identify any of the movements such as grasping, rotating, pinching, etc., and promptly provide an operation classification result. The operation / sequence accuracy recognition model 133 focuses on the accuracy of the operation process. The operation / sequence accuracy recognition model 133 compares and analyzes based on the hand joint position coordinates of the operator with a preset standard work procedure parameter set. Through such comparison, it is possible to detect whether the operator has completed a predetermined operation in the correct order and determine the accuracy of the work. Also, from the analysis / comparison results, the operation / sequence accuracy recognition model 133 can evaluate whether the hand movements and processes of the operator during work meet the standards, and ensure the accuracy and consistency of the work. In addition, the work procedure recognition / analysis method in this specific embodiment may further include the following steps by combining with the operation type recognition model 132 and the operation / sequence accuracy recognition model 133. That is, the operation type recognition model 132 recognizes and classifies various types of hand movements by the third machine learning method based on the hand joint position coordinates in the continuous images captured by the image acquisition module 11. Also, the operation / sequence accuracy recognition model 133 analyzes and compares the hand joint position coordinates with the standard work procedure parameter set to determine whether the hand movements and work procedures of the operator are correct.

[0051] The first machine learning method, the second machine learning method, and the third machine learning method mentioned in the operation technique recognition and analysis system in this specific embodiment further include artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), decision trees, support vector machines (SVMs), random forests, k-nearest neighbors (KNN), k-means clustering, principal component analysis (PCA), linear regression, logistic regression, gradient boosting machines, deep belief networks (DBNs), recursive neural networks (RecNN), reinforcement learning, autoencoders, Gaussian processes, complex neural networks, or any other machine learning algorithm or neural network algorithm, and are obtained by performing machine learning on a dataset. The selection of the machine learning or neural network algorithm is performed according to the needs of the user.

[0052] In summary, the present invention combines artificial intelligence and a hand recognition system and uses it to analyze the accuracy of the operator's hand movements, thereby providing a work skill recognition and analysis system and method for improving product quality and product consistency in the manufacturing process. By performing comprehensive optimization processing based on the finger bone lengths and joint constraints of the operator using the hand bone joint model direction comprehensive optimization model, the recognition accuracy of the joint positions of the hand is improved and the distortion in the recognition process is reduced. In addition, based on the analysis and comparison results of the hand joint position coordinates and the standard work skill parameter set of the operator during the cell culture period, a notification message is issued by the attention module to alert the operator whether the current movement is correct. Thereby, the operator is promptly alerted to correct the hand movement, improving the accuracy of the hand movement.

[0053] The detailed description of the above preferred specific embodiments is intended to more clearly describe the features and spirit of the present invention, and is not intended to limit the scope of the present invention by the preferred specific embodiments disclosed above. Rather, the above description is intended to cover configurations having various modifications and equivalences within the scope of the claims sought by the present invention. Therefore, the scope of the claims sought by the present invention should be interpreted as broadly as possible based on the above description to cover all possible modifications and configurations having equivalences.

Description of Reference Numerals

[0054] 1, 2 Work skill recognition and analysis system 11 Image acquisition module 12 Recognition module 121 2D joint recognition model 122 Joint 3D coordinate recognition model 123 Hand bone joint model direction comprehensive optimization model 13 Motion analysis module 131 Skill motion analysis model 132 Motion type recognition model 133 Motion / sequence accuracy recognition model 14 Attention module Steps S1 to S6

Claims

1. A work technique recognition and analysis system for evaluating the accuracy of at least one hand movement in a work area of ​​a worker during work, comprising: an image capture module disposed in the work area and adapted to capture successive images of the movement of the at least one hand of the worker in the work area during the work; a recognition module coupled to the image acquisition module, a two-dimensional joint recognition model used to recognize joint positions of the hand of the operator based on the successive images; a joint three-dimensional coordinate recognition model that is connected to the two-dimensional joint recognition model and is used to generate hand joint position coordinates corresponding to joint positions of the hand of the worker based on a plurality of consecutive images from different angles of the motion of the at least one hand in the working area of ​​the worker during the work captured by the plurality of image acquisition modules, the hand joint position coordinates being three-dimensional coordinates; a hand skeleton joint model orientation comprehensive optimization model that is connected to the joint three-dimensional coordinate recognition model and is used to improve the accuracy of the hand joint position coordinate by performing a comprehensive optimization process on the hand joint position coordinate based on the phalange length constraint and the phalange joint constraint of the worker; The two-dimensional joint recognition model, the three-dimensional joint coordinate recognition model, and the hand skeleton joint model orientation comprehensive optimization model are constructed by training a work technique dataset using a first machine learning method; a motion analysis module connected to the recognition module, the motion analysis module including a manual motion analysis model used to receive the hand joint position coordinates, perform analysis and comparison based on the hand joint position coordinates of the worker and a standard work technique parameter set, and obtain an analysis and comparison result, thereby evaluating the accuracy of the motion of the at least one hand of the worker in the working area during the work, the manual motion analysis model being constructed by training a standard work technique dataset by a second machine learning method; A work technique recognition and analysis system including:

2. 2. The work technique recognition and analysis system according to claim 1, wherein the three-dimensional coordinates are calculated by calculating an intersection point of emission lines from each of the image acquisition modules by a triangulation method, or by calculating a midpoint of the shortest distance of the emission lines from each of the image acquisition modules as an approximation of the intersection point by a midpoint method.

3. The motion analysis module further comprises:

2. The work technique recognition and analysis system according to claim 1, further comprising: a motion type recognition model used to recognize and classify various types of hand motions by a third machine learning method based on the hand joint position coordinates in the continuous images captured by the image acquisition module.

4. The motion analysis module further comprises:

2. The work technique recognition and analysis system according to claim 1, further comprising a movement / sequence accuracy recognition model used to determine whether the at least one hand movement and work procedure of the worker are correct by analyzing and comparing the hand joint position coordinates and the standard work technique parameter set.

5. Furthermore, 2. The work technique recognition and analysis system according to claim 1, further comprising an attention calling module connected to the motion analysis module and used to call the attention of the worker by issuing a notification message based on the analysis and comparison results obtained by analyzing and comparing the hand joint position coordinates in the working area of ​​the worker during the work with the standard work technique parameter set.

6. The work technique recognition and analysis system according to claim 5 , wherein the notification message further includes a voice notification and a lamp notification.

7. 7. The work technique recognition and analysis system according to claim 6, wherein when the analysis and comparison result obtained by analyzing and comparing the hand joint position coordinates in the working area of ​​the worker during the work with the standard work technique parameter set is inconsistent, the attention calling module issues the voice notification, and the voice notification is an error attention warning.

8. The work technique recognition and analysis system according to claim 1 , wherein the image acquisition module includes a camera, the camera being provided in the work area.

9. The work technique recognition and analysis system according to claim 1 , wherein the work area includes a laboratory work bench, a sterile work bench, and a cell work bench.

10. The first machine learning method, the second machine learning method, and the third machine learning method may further include artificial neural networks (ANN), convolutional neural networks (CNNs), recurrent neural networks (RNNs), decision trees (Decision Trees), support vector machines (SVM), random forests (Random Forests), k-Nearest Neighbors (KNN), k-Means Clustering, principal component analysis (Principal Component Analysis (PCA)), and the like.

4. The work technique recognition and analysis system according to claim 3, further comprising at least one of the following: Pairwise Convolutional Neural Networks (PCA), Linear Regression, Logistic Regression, Gradient Boosting Machines, Deep Belief Networks (DBN), Recursive Neural Networks (RecNN), Reinforcement Learning, Autoencoders, Gaussian Processes, and Complex Neural Networks.

11. A work technique recognition and analysis method used to evaluate the accuracy of a movement of at least one hand of a worker in a work area during work, comprising: an image capture module capturing successive images of the movement of the at least one hand in the work area of ​​the worker during the task; A two-dimensional joint recognition model of a recognition module recognizes joint positions of the hand of the operator based on the successive images; A step in which a joint three-dimensional coordinate recognition model of the recognition module generates hand joint position coordinates corresponding to joint positions of the hand of the worker based on a plurality of consecutive images from different angles of the movement of the at least one hand in the working area of ​​the worker during the work captured by a plurality of the image acquisition modules, the hand joint position coordinates being three-dimensional coordinates; a hand skeleton joint model orientation comprehensive optimization model of the recognition module performs a comprehensive optimization process on the hand joint position coordinates based on the phalange length constraint and the phalange joint constraint of the worker, thereby improving the accuracy of the hand joint position coordinates; and the two-dimensional joint recognition model, the three-dimensional joint coordinate recognition model, and the hand skeleton joint model orientation comprehensive optimization model are constructed by training a work technique dataset using a first machine learning method; a manual motion analysis model of a motion analysis module receiving the hand joint position coordinates, analyzing and comparing based on the hand joint position coordinates of the worker and a standard task technique parameter set, and obtaining an analysis and comparison result, thereby evaluating accuracy of the at least one hand motion of the worker in the working area during the work, and the manual motion analysis model is constructed by training a standard task technique dataset by a second machine learning method.

12. The method for recognizing and analyzing a work technique according to claim 11, wherein the three-dimensional coordinates are calculated by calculating an intersection point of the emission lines from each of the image acquisition modules by a triangulation method, or by calculating a midpoint of the shortest distance of the emission lines from each of the image acquisition modules as an approximation of the intersection point by a midpoint method.

13. Furthermore, 12. The method of claim 11, wherein the motion type recognition model of the motion analysis module includes a step of recognizing and classifying various types of hand motions by a third machine learning method based on the hand joint position coordinates in the continuous images captured by the image acquisition module.

14. Furthermore, 12. The work technique recognition and analysis method according to claim 11, further comprising a step of: determining whether the at least one hand movement and work procedure of the worker are correct by analyzing and comparing the hand joint position coordinates and the standard work technique parameter set in the movement and sequence accuracy recognition model of the movement analysis module.

15. Furthermore, 12. The work technique recognition and analysis method according to claim 11, further comprising a step of: an attention drawing module drawing attention to the worker by issuing a notification message based on the analysis and comparison result obtained by analyzing and comparing the hand joint position coordinates in the working area of ​​the worker during the work with the standard work technique parameter set.

16. The work technique recognition and analysis method according to claim 15 , wherein the notification message further includes a voice notification and a lamp notification.

17. 17. The work technique recognition and analysis method according to claim 16, wherein, when the analysis and comparison result obtained by analyzing and comparing the hand joint position coordinates in the working area of ​​the worker during the work with the standard work technique parameter set is inconsistent, the attention calling module issues the voice notification, and the voice notification is an error attention warning.

18. The method of claim 11 , wherein the image acquisition module includes a camera, and the camera is provided in the work area.

19. The method for recognizing and analyzing a work technique according to claim 11 , wherein the work area includes a laboratory work bench, a sterile work bench, and a cell work bench.

20. The first machine learning method and the second machine learning method further include artificial neural networks (ANN), convolutional neural networks (CNNs), recurrent neural networks (RNNs), decision trees (Decision Trees), support vector machines (SVM), random forests (Random Forests), k-Nearest Neighbors (KNN), k-Means Clustering, principal component analysis (Principal Component Analysis (PCA)), and the like. The work technique recognition and analysis method according to claim 13, further comprising at least one of the following: PCA, Linear Regression, Logistic Regression, Gradient Boosting Machines, Deep Belief Networks (DBN), Recursive Neural Networks (RecNN), Reinforcement Learning, Autoencoders, Gaussian Processes, and Complex Neural Networks.

Images