Method and system for recognizing personnel work state based on twin data

Abstract

The application discloses a personnel operation state recognition method and system based on twin data, recognizes the posture of production line operators, obtains twin data of the postures of workshop operators, acquires human posture three-dimensional space feature sequences and time sequences of positions of each human key point of personnel operation processes through the twin data, and constructs operation posture space-time feature data sets; the twin data in unknown operation processes are matched with known features in the operation posture space-time feature data sets, the process to which the operation process belongs is determined according to the value output by a similarity function, and the working hours of the operation process are solved by introducing a time window function. The application can provide a basis for working hour analysis of production line operators and can provide basic data for personnel operation state driving of a digital twin workshop.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing, and particularly relates to a personnel operation state recognition method and system based on twin data. BACKGROUND

[0002] In recent years, China's agricultural machinery industry has developed rapidly. Although tractor production in large enterprises currently adopts assembly production lines, the production automation degree in the assembly workshop is relatively low compared with the automobile industry. In the workshop production process, personnel are involved in all assembly stations, and the proficiency of personnel and other factors affect the operation efficiency. Even the assembly efficiency is low, which makes the operation station the slowest link in the production process, i.e., causes the production line bottleneck station problem. In order to understand the operation hours of personnel at each station and solve the bottleneck station problem of production line personnel, the workshop managers currently mainly analyze the timing at key stations of the production line or analyze the operation hours of personnel in the operation video through on-site camera shooting, find out the bottleneck station, and optimize the production mode to improve the balance rate of the production line. However, these operation hour analysis methods will consume a large amount of time cost and labor cost. SUMMARY

[0003] In view of the deficiencies in the prior art, the application provides a personnel operation state recognition method and system based on twin data, which can provide operation hour calculation of operation procedures for personnel operation analysis at key stations of the production line.

[0004] The application achieves the above technical object through the following technical means.

[0005] The personnel operation state recognition method based on twin data comprises the following steps:

[0006] Recognizing the posture of the production line personnel to obtain twin data of the posture of the workshop personnel;

[0007] Obtaining the human posture three-dimensional space feature sequence and the time sequence of each human key point position of the personnel operation process through the twin data, and constructing an operation posture space-time feature data set;

[0008] Matching the twin data in the unknown operation process with the known features in the operation posture space-time feature data set, determining the operation procedure to which the operation process belongs according to the value output by the similarity function, and solving the operation hour of the operation procedure by introducing a time window function;

[0009] The recognizing the posture of the production line personnel comprises recognizing the personnel and recognizing the three-dimensional posture of the personnel.

[0010] The twin data of the workshop operator is obtained by establishing a mapping relationship between the virtual human body model and the standard human body model, mapping the three-dimensional key point data to the standard human body model, and then mapping the three-dimensional key points to the virtual human body model, so as to obtain the twin data of the posture of the workshop operator.

[0011] Further technical solutions, the process of identifying the operator is:

[0012] Collecting the video of the operation process of the personnel by using the video acquisition system;

[0013] The operation video is decomposed into pictures to form a video picture library for the workstation operator;

[0014] According to different working procedures of the work operation process, the pictures are labeled to form a label file;

[0015] The data set formed by the label file and the pictures is input into a deep learning model for model training;

[0016] According to the trained deep learning model, the operator in the video is identified and the complex environment is filtered out.

[0017] Further technical solutions, the process of identifying the three-dimensional posture of the operator is:

[0018] When the input is labeled posture data, the two-dimensional posture data that has been labeled is input into the Videopose3d algorithm to obtain the root joint trajectory and the position of all joints relative to the root joint; the error value between the joint position and the labeled result is calculated, which is used to train the loss function;

[0019] When the input is unlabeled posture data, after obtaining the root joint trajectory and the position of all joints relative to the root joint, the weighted error value is calculated with the bone length in the labeled data training process, and the loss function of the semi-supervised process is trained; in the unlabeled data training, the three-dimensional joint position predicted is obtained;

[0020] After the above training, the recognition model of the spatial three-dimensional key points of the operator posture is obtained, and the joint position output by the model is used as the human key point.

[0021] Further technical solutions, the process of mapping the three-dimensional key point data to the standard human body model is:

[0022] According to the connection order between the human skeleton joints, the human structure is divided into 6 bone chains;

[0023] Each bone chain is a set of key points: head b1={8, 9, 10}, right arm b2={8, 14, 15, 16}, left arm b3={8, 11, 12, 13}, spine b4={8, 7, 0}, right leg b5={0, 1, 2, 3} and left leg b6={0, 4, 5, 6};

[0024] The direction of the human body key point direction vector is determined by the above key point set, and the human body key point direction unit vector is determined by the bone chain set

[0025]

[0026] In the formula, is a vector composed of connected key points, is the length of the vector in the pixel space, I and J represent two connected key points;

[0027] According to the standard size of each joint of an adult human body in the national standard, the vector of each joint in the human body bone chain in the actual physical space is obtained as follows:

[0028]

[0029] wherein, L I_J is the length of each joint of the standard human body.

[0030] In a further technical solution, the three-dimensional space feature of the human body posture is an angle feature of the upper and lower limbs of the human body, the upper limb angle feature is an angle formed by a bone chain between wrist, elbow and shoulder joint key points of the human body, and the lower limb angle feature is an angle formed by a bone chain between the hips, knees and ankles of the human body.

[0031] In a further technical solution, a time window function is introduced to solve the working hours of the working procedure, specifically: taking the minimum time sequence length of the known procedure as the initial time window size, calculating the similarity of the unknown working feature and the feature sequence in the working posture space data set within the range; when the best working feature is matched, the time of the working feature is output; if the working feature in the range is not matched, taking only more than the minimum time sequence length as the new time window size, calculating the similarity of the unknown working feature, and continuing to match until the best working feature is matched, and the corresponding working hours are output.

[0032] A further technical solution maps 3D key points to a virtual human body model. Specifically, a virtual human body model is established in a virtual space, and key points of the human skeleton are defined as key points. Corresponding key points are defined in the virtual human body model. Based on the 3D spatial position of the standardized 3D key points of the operator, the position of the virtual human body model in the 3D space is driven, thereby realizing the mapping between the virtual human body model and the 3D key points in the physical space.

[0033] A personnel operation status recognition system based on twin data includes a video acquisition system, a video analysis system, a data twin system, and a status recognition system. The video acquisition system is installed at the workstations on the production line in the workshop where analysis is required to collect videos of personnel's operation process. The video analysis system is used to identify the workers and perform 3D posture recognition analysis. The data twin system achieves data twinning by mapping the positions of key points between the 3D key points of the human body in physical space and the key points of a virtual human body model. The status recognition system uses similarity to calculate the personnel's operation status.

[0034] The beneficial effects of this invention are as follows:

[0035] (1) Based on the three-dimensional spatial features and time series features of the posture of workshop workers during the operation process, a worker identification method based on spatial feature similarity and time window features was constructed to meet the needs of workshop workers' working time analysis.

[0036] (2) This invention addresses the complex environment of workshops by using deep learning-based identification of workers and analysis of their three-dimensional poses. This solves the problem of the impact of multiple workers and worker movement on the extraction of three-dimensional poses in complex workshop environments, thereby improving the accuracy of three-dimensional pose analysis.

[0037] (3) This invention realizes the digital twin of workshop workers by establishing a visual recognition-based three-dimensional human posture recognition and standardized processing of three-dimensional spatial data. Attached Figure Description

[0038] Figure 1 This is a diagram of the personnel work status recognition system based on twin data described in this invention;

[0039] Figure 2 This is a schematic diagram of the standardization process described in this invention;

[0040] Figure 3 This is a diagram illustrating the personnel operation process of the virtual platform described in this invention.

[0041] Figure 4(a) is a schematic diagram of the three-dimensional key point skeleton described in this invention;

[0042] Figure 4(b) is a schematic diagram of the virtual model skeleton described in this invention;

[0043] Figure 5(a) shows the change of the left elbow angle over time according to the present invention;

[0044] Figure 5(b) shows the change of the right elbow angle over time according to the present invention;

[0045] Figure 6 This is a diagram illustrating the dynamic time window detection process described in this invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0047] like Figure 1 As shown, this invention discloses a personnel operation status recognition system based on twin data, comprising a video acquisition system, a video analysis system, a data twin system, and a status recognition system. The video acquisition system is installed at the workstation on the production line in the workshop where analysis is required. The video analysis system is used to identify and analyze the three-dimensional posture of the workers. The data twin system achieves data twinning by mapping the positions of three-dimensional key points of the human body in physical space to key points of a virtual human body model. The status recognition system calculates the personnel's operation status using similarity.

[0048] The video analytics system can be either a cloud service platform or an edge computing platform.

[0049] The video acquisition system transmits video to the video analysis system via the network. The video analysis system identifies the workers and analyzes their 3D posture, transmitting this data to the data twin system to drive the virtual human model. If multiple video acquisition systems are deployed on-site, they are directly connected to the video analysis system. A deep learning-based posture recognition model of the workers is deployed in the video analysis system to analyze their 3D posture and transmit this data to the data twin system to drive the virtual human model. This resolves the network load pressure caused by multiple video acquisition systems simultaneously transmitting video to the server.

[0050] Based on the aforementioned personnel work status recognition system, this invention provides a personnel work status recognition method based on twin data. This method identifies the postures of production line workers, obtains twin data of workshop workers, and constructs a spatiotemporal feature dataset of personnel work postures. Through this dataset, it extracts the similarity of three-dimensional spatial features of work postures and the time-series features of the work process. By matching spatial and temporal features, it calculates the work status (including processes and working hours) of production line workers.

[0051] I. The posture of the production line operator is identified, including two processes of operator identification based on deep learning and three-dimensional posture identification of the operator.

[0052] The operator identification based on deep learning is to solve the problem of target identification caused by the simultaneous appearance of multiple operators in the video and the movement of personnel during the operation process. The process is as follows:

[0053] (1) Use the video acquisition system to collect the video of the personnel operation process;

[0054] (2) Decompose the operation video into pictures to form a video picture library for the operator at the work station;

[0055] (3) According to different working procedures, label the pictures to form a label file;

[0056] (4) Input the data set formed by the label file and the picture into the deep learning model for model training;

[0057] (5) According to the trained deep learning model, identify the operator in the video and filter out the complex environment.

[0058] The three-dimensional posture identification of the operator is based on the deep learning method to realize the construction of the recognition model of the spatial three-dimensional key points of the operator's posture. The process is as follows:

[0059] (1) When the input is labeled posture data, input the already labeled two-dimensional posture data into the Videopose3d algorithm (a detection algorithm for human three-dimensional key points in video information);

[0060] (2) Get the root joint (human center point) trajectory and the position of all joints relative to the root joint;

[0061] (3) Calculate the error value between the joint position and the labeled result, which is used to train the loss function;

[0062] (4) When the input is unlabeled posture data, after obtaining the root joint trajectory and the position of all joints relative to the root joint, calculate the weighted error value with the bone length calculation in the labeled data training process, and train the loss function of the semi-supervised process;

[0063] (5) In the unlabeled data training, the predicted three-dimensional joint position is projected back to two dimensions to verify the consistency of the input;

[0064] (6) After the above training, a recognition model of the spatial three-dimensional key points of the operator's posture with smaller error is obtained, and the joint position output by the model is used as the human key point.

[0065] II. The process of obtaining the twin data of the postures of the workshop workers is as follows:

[0066] (1) The video collected by the video collection system is transmitted to the video analysis system to realize the identification of the workers and the analysis of the three-dimensional postures of the workers.

[0067] (2) The worker at the specified work station is identified through the deep learning model, and the video of the worker is extracted.

[0068] (3) The three-dimensional key point data of the human skeleton at the specified work station is obtained through the three-dimensional posture recognition algorithm.

[0069] (4) A virtual human model is constructed in the data twin system, and a mapping relationship between the virtual human model and the standard human model is established.

[0070] (5) According to the standard of the standard human body, the three-dimensional key point data is mapped into the standard human model, solving the spatial position change of the human body data caused by the camera angle, the distance between the camera and the measured object, etc. in the visual acquisition process.

[0071] The process of mapping the three-dimensional key point data to the standard human model is as follows: according to the connection order between the joints of the human skeleton, the human structure is divided into 6 bone chains; each bone chain is a set of key points: head b1={8,9,10}, right arm b2={8,14,15,16}, left arm b3={8,11,12,13}, spine b4={8,7,0}, right leg b5={0,1,2,3}, and left leg b6={0,4,5,6}; the numbers in the set correspond to the skeleton key point serial numbers in Figure 4, the direction of the human key point direction vector is determined through the above key point set, and the human key point direction unit vector is determined through the bone chain set (I, J represent two connected key points) is:

[0072]

[0073] In the formula, is the vector formed by the connected key points, is the length of the vector in the pixel space.

[0074] According to the standard size of each joint of an adult human body in the national standard, the vector of each joint in the human bone chain in the actual physical space is obtained as follows:

[0075]

[0076] where L I_J is the length between each joint of the standard human body.

[0077] After standardization, the visual acquisition of human three-dimensional posture spatial position data is mapped to the spatial position of the standard human body.

[0078] (6) The actual acquired standardized worker three-dimensional key point data is mapped to the data twin system to obtain the twin data of the workshop worker posture.

[0079] A virtual human body model is established in a virtual space, and the human body features are actually acquired to define the human body skeletal key points as key points, and the corresponding key points are defined in the virtual human body model; the position in the three-dimensional space of the virtual human body model is driven according to the three-dimensional spatial position of the actual acquired standardized worker three-dimensional key points, so as to realize the mapping between the virtual human body model and the physical space three-dimensional key points.

[0080] III. The operation posture space-time feature data set includes the human body posture three-dimensional space feature sequence and the time sequence of each human body key point position of the personnel operation process obtained through the twin data. In order to reduce the data set which is too large due to similar features and affect the performance of data retrieval, by setting the similarity threshold of the operation posture space-time feature data set, when the maximum similarity of the new space-time data record and the feature data set is less than the set similarity threshold, the data record is saved as the information data set of the operation posture space-time feature data set, otherwise the data record is not saved.

[0081] The human body posture three-dimensional space feature is the angle feature of the upper and lower limbs of the human body, the upper limb angle feature is the angle formed by the skeletal chain between the wrist, elbow and shoulder joint key points of the human body, and the lower limb angle feature is the angle formed by the skeletal chain between the hip, knee and ankle of the human body. The calculation of the angle is obtained by using the cosine law according to the key point space position information.

[0082] IV. The similarity calculation process adopts the cosine similarity calculation method, and the three-dimensional space feature data of the four limb skeletal chains of the acquired working operation state are respectively searched. The twin data in the unknown operation process is matched with the known features in the operation posture space-time feature data set, and the value output by the similarity function is used to determine the process to which the operation process belongs. Among them, the upper limbs are mainly involved in the part positioning and assembly of the work station, which affects the assembly operation, so the distribution of the upper limb key points is studied, which is meaningful for the analysis of the operation process.

[0083] The personnel operation state recognition based on the posture feature similarity matching considers that data of different operation states have different time sequence lengths, and a time window function is introduced to solve the unknown operation time, the process being: taking the minimum time sequence length of the known operation as the initial time window size, calculating the similarity of the unknown operation feature in this range with the feature sequence in the operation posture space-time data set; when the best operation feature is matched, the time of the operation feature is output; if the operation feature in this range is not matched, taking only the minimum time sequence length as the new time window size, calculating the similarity of the unknown operation feature, and continuing to match until the best operation feature is matched, and the corresponding operation time is output.

[0084] A personnel operation state recognition method based on twin data will be specifically introduced below, including the following processes:

[0085] I. Construction of twin data of workshop operation personnel

[0086] Step 1) Construction of data set

[0087] Step 1.1) Operation data acquisition

[0088] A Hikvision network camera is erected at a certain work station in the tractor production workshop, the camera is connected through a network cable and powered, and the operation video is uploaded to the cloud service platform after being acquired.

[0089] Step 1.2) Data set composition

[0090] The operation video in the cloud service platform is taken every 50 frames using OpenCV, and the pictures are imported into the labeling software LabelImag, and workingperson (operation personnel), unworkingperson (non-operation personnel), and workingarea (working area) are defined in data. Then, the working people, non-working people and working areas are selected using a rectangular frame in LabelImag, and LabelImag will generate the four vertex coordinates of the target frame. Finally, the PascalVOC file is formed, including the xml format Annotations folder, ImageSets folder and JPEGImages folder containing the original pictures, and the ImageSets folder contains the documents of the training set and test set directories.

[0091] Step 2) Operation video analysis

[0092] Step 2.1) Operation personnel recognition

[0093] The PascalVOC file is used as the initial model training set and test set data, and the data set is input into the Yolo-v4 model, and the learning rate is set to 1e -3The minimum learning rate is set to one percent of the maximum learning rate; the total training generation is set to 1200 generations, and the training weight is saved once every 100 generations; the training process is divided into frozen training and unfrozen training; the Adam optimizer is used throughout the training process, and the internal parameter Momentum of the optimizer is set to 0.937. The frozen training is a total of 400 generations, and the number of training samples per generation is 8; the unfrozen training is the last 800 generations, and the number of samples per generation is 4. There are a total of 2204 images in the training and test sets, of which 1983 are in the training set. The training effect evaluation of the training set is shown in Table 1, the average precision of workingperson (working personnel) detection is 92.03%, the average precision of unworkingperson (non-working personnel) detection is 93.08%, and the average precision of workingarea (working area) detection is 88.08%.

[0094] Table 1 Performance Statistics

[0095]

[0096] Step 2.2) Three-dimensional pose recognition

[0097] After identifying the working personnel and determining the target, the image pixel values outside the target frame are set to (0, 0, 0) based on Opencv image processing technology. At this time, the video image outside the target frame is black, and only the working personnel at the work station remains in the image, filtering out the complex information in the image.

[0098] The three-dimensional key point positions of the personnel are obtained through the Videopose3d algorithm, and the key point serial number corresponding to each joint is shown in Figure 4(b). There are a total of 17 joints in the human body, and the time domain inflation model in the algorithm converts the two-dimensional coordinate array of consecutive frames into a three-dimensional array, i.e. the spatial coordinate position of the human body key point.

[0099] Step 2.3) Standardization processing

[0100] Considering that in actual factories, due to the different distances between the camera and the working personnel, the sizes of the detected human bodies are different, resulting in that the actual spatial positions of the recognized human body key points are not the actual physical space positions, so the obtained three-dimensional pose data needs to be standardized. The specific process is as follows: considering the connection order between the personnel skeletal joints, the human body structure is divided into 6 skeletal chains, and each skeletal chain is a set of key points:

[0101] b1 = {8, 9, 10}, b2 = {8, 14, 15, 16}, b3 = {8, 11, 12, 13}, b4 = {8, 7, 0}, b5 = {0, 1, 2, 3}, b6 = {0, 4, 5, 6}

[0102] The set b1 represents a skeleton chain composed of key points above the neck, the set b2 represents a right arm skeleton chain, the set b3 represents a left arm skeleton chain, b4 represents a spine skeleton chain, b5 represents a right thigh skeleton chain, and b6 represents a left thigh skeleton chain. The numbers in the set correspond to the sequence numbers of the skeleton key points in FIG. 4(a) (where 0 is hip, 1 is right-hip, 2 is right-knee, 3 is right-foot, 4 is left-hip, 5 is left-knee, 6 is left-foot, 7 is spine, 8 is thorax, 9 is neck, 10 is head, 11 is left-shoulder, 12 is left-elbow, 13 is left-wrist, 14 is right-shoulder, 15 is right-elbow, and 16 is right-wrist), the direction of the personnel key point direction vector is determined by the above key point set, and the direction vector of the human body key point is determined

[0103] According to the human body size of Chinese adults in the national standard, the 50th percentile size from 18 to 60 years old is used as a standard for measurement, and the adjacent point direction vector shown in formula (2) is obtained multiplied by the length (L I_J ) between each joint, so that the distance between the output key points remains unchanged.

[0104] As Figure 2 shown, the personnel key point three-dimensional coordinates are standardized according to the personnel skeleton chain described above, and the skeleton chain is the joint set b3. The skeleton chain is processed step by step, and the direction vector is obtained according to formula (1) by substituting the three-dimensional space coordinates of key points 11 and 8 According to formula (2), the direction vector combined with the size data of the skeleton joint is obtained 11 ′ The point position is the position of the standardized key point 11. Based on the position of 11 ′ and following the above process, the key points 12 and 13 are standardized to obtain the standardized key points 12 ′ and 13 ′ . According to the joint set, the key point space position is standardized, and the standardization of the personnel key point three-dimensional coordinates is completed.

[0105] ​The camera is different from the position of the human body and the output of the human body posture data is different. In order to verify the accuracy of the personnel key point data output by the standardization process, a standardization data verification experiment is set. The experimental process is as follows: the indoor fixed camera position is 1.5 meters high, the nearest distance of the experimental personnel from the camera is 3.5 meters, the farthest distance is 6 meters, the experimental personnel moves every 25 centimeters in a straight line direction with the camera with arms spread out, and 11 groups of human video image information at different positions are collected.

[0106] The video image information of the personnel at different positions is processed by the Videopose3d algorithm, and the three-dimensional posture data obtained is standardized. The standardized posture data contains 17 key points of human body three-dimensional data of 11 different points. According to the human body size of Chinese adult in the national standard, under the experimental conditions, the spatial position of wrist joint and foot joint of the standard body standing at different distances from the camera is calculated as the control group. The error analysis between the standardized wrist key point and foot key point data and the control group is carried out, and the error table shown in Tables 2, 3 and 4 is obtained. The average error of the key points in the X-axis direction is 39.73mm, 39.80mm, 46.79mm and 46.64mm, the average error in the Y-axis direction is 41.96mm, 40.01mm, 46.79mm and 46.64mm, and the average error in the Z-axis direction is 41.6mm, 40.51mm and 39.57mm. The error of the key points in the three directions is between 3-4cm. The actual production workshop space is large, and this error can be ignored in practice. Therefore, the posture standardization process meets the actual personnel posture estimation accuracy.

[0107] Table 2 Standardized X-axis direction error value of part of key points

[0108]

[0109] Table 3 Standardized Y-axis direction error value of part of key points

[0110]

[0111]

[0112] Table 4 Standardized Z-axis direction error value of part of key points

[0113]

[0114] Step 3) Twin data analysis

[0115] Step 3.1) Key point mapping

[0116] As shown in Figure 4(a), based on the human body characteristics in the physical space, the key point positions and names of the human body model's skeleton are defined through a 3D skeleton diagram. Corresponding key points are then defined in the virtual human body model, completing the model's skeletal mapping. By mapping the 3D pose data to the twin space, corresponding to each joint of the virtual human body model, twin pose data information corresponding to each key point in the pose data is formed.

[0117] Step 3.2) Model pose display

[0118] like Figure 3 As shown, the virtual human body model is initialized on the Unity 3D platform. The initialized virtual human body model presents a T-shaped posture. By creating a script in the Unity 3D platform to receive twin data, the twin data is bound to the joints of the virtual human body model. After the joint binding is completed, the twin data drives the joint position update of the virtual human body model, so as to realize the synchronization of the movement of the virtual human body model with the posture of the person working in the video information.

[0119] II. Construction of Assembly Operation Feature Dataset

[0120] The shoulder, elbow, and wrist joints are classified as the upper limb joint chain; the hip, knee, and foot joints are classified as the lower limb joint chain. As shown in Figure 4(a), in the human skeleton map output by 3D pose detection, the upper limb joints correspond to the 11th (left shoulder key point), 12th (left elbow key point), 13th (left wrist key point) and the 14th (right shoulder key point), 15th (right elbow joint), 16th (right wrist joint) key points respectively; the lower limb joints correspond to the 4th (left hip key point), 5th (left knee key point), 6th (left foot key point) key points and the 1st (right hip key point), 2nd (right knee key point), 3rd (right foot key point) key points respectively.

[0121] The 11th (left shoulder key point), 12th (left elbow key point), and 13th (left wrist key point) form a triangle. Let θ be the angle between these three points with the 12th point as the vertex. Let P be the position of the 11th, 12th, and 13th key points. 11 P 12 P 13 The formula for solving using the vector inner product is... Solve for the value of θ:

[0122]

[0123] P 11 (x 11 y 11 , z 11 ), P 12 (x 12 y 12 , z 12), P 13 (x 13 y 13 , z 13 ( ) represents the three-dimensional coordinates of the three key points.

[0124] Will Substitute into equation (3) to obtain the specific value of θ. The same applies to key points 14, 15, and 16.

[0125] Using the law of cosines, the angles formed by key points 11, 12, 13, 14, 15, and 16, with the left and right elbow joints as vertices, are θ1 and θ2, respectively. The variations of θ1 and θ2 are shown in Figures 5(a) and (b). By mapping the different angles and times corresponding to different work processes, feature information of a certain action in the operation process is formed. The feature information mainly includes the angles of the left and right elbow joints and the time node when the action occurs, constituting a feature dataset of known assembly operations.

[0126] III. Work Process Calculation

[0127] By extracting the elbow angle data of different personnel in the feature dataset, it can be found that the elbow angle of the human body has a certain trend of change with the work process. The angle changes within a certain range when performing different assembly operations. Therefore, the angle feature is used as the basis for judging the work status of personnel.

[0128] The time series and the elbow joint angle are mapped onto a two-dimensional coordinate system in vector form. When performing task feature matching, the vector similarity of the time series is used as the matching index to measure the task feature sequence.

[0129] The cosine similarity method is used to calculate the similarity between vectors, and the cosine similarity algorithm is used to calculate the similarity with the elbow joint time series of the known operation process.

[0130] The working principle of the cosine similarity method is as follows: the coordinates of each dimension of the vector are mapped to the vector space, and then the cosine value of the angle between the inner product space of two vectors in the vector space is measured to measure the similarity between the two vectors. The closer the angle is to 0°, that is, the larger the cosine value, the higher the similarity between the two vectors; conversely, the closer the angle is to 180°, that is, the smaller the cosine value, the smaller the similarity between the two vectors.

[0131] As shown in equation (4), the calculation formula of the cosine similarity method is obtained by a corresponding transformation of the vector dot product formula, where and These are represented as two vectors whose similarity needs to be calculated.

[0132] The operation process consists of a two-dimensional array composed of multiple time series and joint angles. The operation feature cosine similarity function (5) based on the time series is constructed by equation (4). In equation (5), Lθ and Rθ are the left and right elbow joint angles corresponding to a single time point in the time series.

[0133] The similarity between the two sets of two-dimensional arrays of the unknown operation process and the known operation process is calculated by formula (4) and summed. The largest value in formula (5) is taken as the best matching result of the unknown operation process, where n and m are the start and end nodes of the time series.

[0134]

[0135]

[0136] IV. Calculation of Working Hours During the Work Process

[0137] like Figure 6 The detection process of the dynamic time window is as follows: First, the time series length of the job feature dataset is retrieved, and the minimum value retrieved is the value of the initial time window T0. When a feature sequence of an unfinished job process is input, the job features of this sequence are matched with T0 as the standard. If the features within T0 are consistent with the features within T0 in the dataset, the job features of the sequence in the initial time window are determined; if the features within T0 are inconsistent with the features within T0 in the dataset, the time series T1 (T1 is only greater than T0) is used as the value of the next stage time window, and the sequence in the time window is matched with features; and so on, until the corresponding job state is matched. Based on the above detection process, the formula for calculating the working time of the job process is as shown in equation (6):

[0138]

[0139] In the formula: T i T represents the time window value where feature inconsistencies occur during the matching process. x is the time window value when the feature matches are consistent, and n is the number of times the matching features are consistent.

[0140] Detection effect verification: based on dynamic time window detection as shown in Table 5, representing the 15 workpieces on the assembly line before painting the operation of the artificial timing analysis of the operation time and based on the operation characteristics matching calculation time. As shown in equation (7), the design operation time is t1, the actual operation time is t0, the relative error of t1 and t0 is γ, and the recognition accuracy is δ. During the continuous processing of the 15 tractors to be assembled, the operation time recognition accuracy of the two operation processes is shown in Table 5, and the average recognition accuracy of assembling gear pump and assembling oil suction pipe is 87.2% and 95.5% respectively, and the average error of the two processes is 3.33s. It can be seen that the personnel operation state recognition method based on twin data can accurately calculate the operation time.

[0141]

[0142] Table 5 Operation time comparison

[0143]

[0144] The embodiments are preferred embodiments of the present application, but the present application is not limited to the above embodiments, and any obvious improvements, replacements or modifications made by those skilled in the art without departing from the essential content of the present application shall fall within the protection scope of the present application.

Claims

1. A method for identifying personnel work status based on twin data, characterized in that: The postures of production line workers are identified to obtain twin data of the postures of workshop workers; By acquiring the three-dimensional spatial feature sequence of human posture and the time series of key points of each human body during the operation process through twin data, a spatiotemporal feature dataset of operation posture is constructed. The twin data of the unknown operation process is matched with the known features in the spatiotemporal feature dataset of the operation posture. Based on the value output by the similarity function, the operation process to which the operation belongs is determined. Then, by introducing a time window function, the working time of the operation process is calculated. The recognition of the posture of production line workers includes worker identification and 3D posture recognition of workers. The process of obtaining twin data of workshop workers involves: establishing a mapping relationship between a virtual human body model and a standard human body model, mapping three-dimensional key point data to the standard human body model, and then mapping the three-dimensional key points to the virtual human body model to obtain twin data of the posture of workshop workers. By introducing a time window function, the working time of a work process is calculated. Specifically, the initial time window size is set to the minimum time series length of the known process. The similarity between the unknown work features within this range and the feature sequences in the work posture spatiotemporal dataset is calculated. When the best work feature is matched, the time of that work feature is output. If the work features within this range do not match, a new time window size is set to be only greater than the minimum time series length. The similarity between the unknown work features is calculated, and the matching continues until the best work feature is matched, at which point the corresponding working time is output.

2. The method for identifying personnel work status based on twin data according to claim 1, characterized in that, The process of identifying workers is as follows: Use a video capture system to collect videos of personnel working in the field; The operation video is broken down into images to create a video and image library for workstation operators. The images are labeled according to the different steps in the work process, forming a label file; The dataset consisting of label files and images is input into a deep learning model for model training. Based on the trained deep learning model, the system identifies workers in videos and filters out complex environments.

3. The method for identifying personnel work status based on twin data according to claim 1, characterized in that, The process of recognizing the three-dimensional pose of workers is as follows: When the input is labeled pose data, the labeled 2D pose data is input into the Videopose3d algorithm to obtain the root joint trajectory and the position of all joints relative to the root joint; the error value between the joint position and the labeling result is calculated and used to train the loss function. When the input is unlabeled pose data, after obtaining the root joint trajectory and the position of all joints relative to the root joint, a weighted error value is calculated with the bone length during the labeled data training process, and the loss function of the semi-supervised process is trained; the 3D joint positions are predicted in the unlabeled data training. After the above training, a spatial three-dimensional key point recognition model of the worker's posture is obtained, and the joint positions output by the model are used as human body key points.

4. The method for identifying personnel work status based on twin data according to claim 1, characterized in that, The process of mapping 3D key point data to a standard human body model is as follows: Based on the connection sequence between the joints of the human skeleton, the human structure is divided into 6 skeletal chains; Each skeletal chain is a set of key points: head b1 = {8,9,10}, right arm b2 = {8,14,15,16}, left arm b3 = {8,11,12,13}, spine b4 = {8,7,0}, right leg b5 = {0,1,2,3} and left leg b6 = {0,4,5,6}; The direction of the human body key point direction vector is determined by the above key point set, and the unit vector of the human body key point direction is determined by the skeletal chain set. In the formula, The vector formed by the key points that are connected. Let I be the length of the vector in pixel space, and let J represent two key points that are connected. Based on the standard dimensions of various joints in the adult human body according to national standards, the vectors of each joint in the human skeletal chain in actual physical space are obtained as follows: Among them, L I_J This refers to the lengths between the various joints in the standard human body.

5. The method for identifying personnel work status based on twin data according to claim 1, characterized in that, The three-dimensional spatial features of human posture are the angular features of the upper and lower limbs. The upper limb angular features are the angles formed by the skeletal chains between the key points of the wrist, elbow and shoulder joints of the two arms. The lower limb angular features are the angles formed by the skeletal chains between the hip, knee and ankle.

6. The method for identifying personnel work status based on twin data according to claim 1, characterized in that, The mapping of 3D key points to a virtual human body model is as follows: a virtual human body model is established in a virtual space, and key points of the human skeleton are defined as key points. Corresponding key points are defined in the virtual human body model. Based on the 3D spatial position of the standardized 3D key points of the operator, the position of the virtual human body model in the 3D space is driven, thereby realizing the mapping between the virtual human body model and the 3D key points in the physical space.

7. A system for implementing the personnel work status recognition method based on twin data as described in any one of claims 1-6, characterized in that, It includes a video acquisition system, a video analysis system, a data twin system, and a status recognition system. The video acquisition system is installed at the workstations on the production line in the workshop where analysis is needed to collect videos of personnel working. The video analysis system is used to identify workers and perform 3D posture recognition analysis. The data twin system achieves data twinning by mapping the positions of key points between the 3D key points of the human body in physical space and the key points of the virtual human body model. The status recognition system uses similarity to calculate the working status of personnel.

Images