Assembly action specification evaluation method fusing pose and semantic understanding

Abstract

The present application belongs to the technical field of industrial intelligent evaluation, and particularly relates to an assembly action standardization evaluation method fusing posture and semantic understanding, comprising: obtaining a human posture sequence based on an assembly video, obtaining a process identification result and a posture quantization index based on the human posture sequence in combination with a standard operation procedure and a pre-defined process template, inputting the process identification result and the posture quantization index into a large language model, then, modeling the assembly process from two aspects of process semantic compliance and posture execution rationality, and realizing comprehensive evaluation in combination with an assembly specification knowledge base and the large language model, so as to output a standardization evaluation result with an overall evaluation conclusion, problem positioning, evidence basis and improvement suggestion. The method is suitable for an industrial assembly scene with clear process constraints and operation specification requirements, and can output a structured, interpretable and traceable evaluation result for a complete assembly process.

Description

Technical Field

[0001] This invention belongs to the technical field of industrial intelligent evaluation, and more specifically, relates to a method for evaluating the standardization of assembly actions by integrating posture and semantic understanding. Background Technology

[0002] In industrial assembly operations, the standardization of human actions directly affects assembly quality, production safety, and operational efficiency. Unlike studies in general scenarios that focus on action category identification or action completion level scoring, industrial assembly scenarios typically have clear process flows, operation sequences, and safety constraints. Therefore, the focus is more on whether the operator's behavior conforms to established process specifications.

[0003] Most existing motion analysis methods revolve around motion recognition or motion quality evaluation. Motion recognition methods mainly answer the question "what action was performed" and can classify videos or skeleton sequences, but they often struggle to determine whether the action conforms to assembly specifications. Motion quality evaluation methods typically output continuous scores or levels to reflect the level of action completion. Their evaluation criteria rely heavily on subjective scoring mechanisms and are suitable for scenarios such as sports, dance, or skill demonstrations, but are difficult to directly transfer to industrial assembly scenarios.

[0004] Furthermore, industrial assembly processes often consist of multiple semantically related and temporally continuous atomic actions, with clear sequences and logical constraints between different procedures. Analyzing only local action segments or single-frame postures makes it difficult to effectively identify high-level semantic anomalies such as missing critical procedures, incorrect procedure sequences, or unauthorized insertions. On the other hand, even if the overall process flow is correct, different operators may still exhibit significantly different body postures and movement patterns when performing the same procedure, leading to risks such as unstable movements, excessive joint misalignment, or compensatory behaviors.

[0005] In existing technologies, analysis methods that rely solely on attitude geometry features are difficult to integrate with process specifications for comprehensive judgment, while relying solely on large models to directly generate evaluation conclusions often lacks stable underlying physical basis and process traceability. Therefore, how to effectively combine process semantic understanding, attitude quantification analysis, and normative knowledge reasoning to construct an assembly action standardization evaluation method that can simultaneously reflect process compliance and the rationality of action execution has become an urgent technical problem to be solved in related fields. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for evaluating the standardization of assembly actions by integrating posture and semantic understanding. The purpose is to overcome the problems in existing technologies that make it difficult to simultaneously characterize the rationality of the assembly process and the standardization of action execution, and that the evaluation results lack interpretability.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for evaluating the standardization of assembly actions that integrates posture and semantic understanding is provided, comprising: Acquire assembly videos and extract chronologically ordered sequences of human postures for each operator. Based on the human posture sequence of each operator, action recognition is performed to obtain candidate action category recognition results corresponding to multiple time windows, including multiple action categories and their corresponding confidence information; using the standard operating procedure as a prior constraint, the action sequence optimization decoding is performed on the candidate action category recognition result sequence composed of all time windows to obtain a continuous action sequence; The continuous action sequence is matched with a predefined process template to obtain process recognition results. Based on the time window distribution corresponding to each action in the continuous action sequence, combined with the window length and window sliding step, the start and end time intervals of each action in the assembly video are determined. The human posture within the corresponding time interval is quantitatively analyzed to obtain posture quantification indicators, including: elbow joint angle deviation, which measures the degree of deviation of the actual action from the reference pattern in terms of joint geometry; wrist trajectory efficiency, which reflects the degree of wrist movement; wrist jerk energy, which describes the intensity of changes in velocity and acceleration during the action; and shoulder height difference, which measures whether the shoulders maintain relative balance during the action execution. The process identification results and posture quantification indicators are represented by text structure. Based on the text structure representation, a retrieval is performed in the assembly specification knowledge base. The text structure representation and retrieval results are input into a large language model. Based on the large language model, the assembly action standardization evaluation results are output, including process semantic compliance and posture execution rationality. Process semantic compliance is used to determine whether there are missing key processes, incorrect process sequence, or non-assembly violations at the action level in the assembly process. Posture execution rationality is used to determine whether there are safety risks, stability risks, or ergonomic risks in the execution of each action.

[0008] Furthermore, the method for performing action recognition is as follows: A fixed-length time window is set on the human posture sequence, and overlapping sliding is performed with a preset step size. Each local time segment is input into the action recognition model to obtain the action category prediction result of the corresponding time window, which includes multiple action categories and their corresponding confidence information. The top k action category prediction results with higher confidence are retained as candidates.

[0009] Furthermore, the method for optimizing the decoding of action sequences is as follows: A state space is constructed with action category sets and skip states as nodes to build a hidden Markov model; where action category nodes are used to represent valid action labels, and skip state nodes are used to represent invalid actions or time windows for identification uncertainty. Based on the confidence information of each action in the candidate action category prediction results for each time window, calculate the emission probability of each action in that time window; Construct state transition probabilities based on the sequence of actions defined in the standard operating procedure; A sequence decoding algorithm based on a hidden Markov model is adopted to obtain the globally optimal action path based on the emission probability of each candidate action in each time window and the state transition probability between time windows, which serves as a continuous action sequence that conforms to the assembly process constraints.

[0010] Furthermore, the method for matching continuous action sequences with predefined process templates is as follows: (1) Starting from the initial action of the continuous action sequence, extract the local action subsequence with the same number of actions as the shortest predefined process template. ; (2) Calculate the relationship between this local action sub-sequence and each predefined process template. Matching distance between Normalizing each matching distance, it is expressed as: In the formula, This represents the number of action steps in a local action subsequence during the matching process; if there is a distance below a preset threshold among the normalized distances, then the predefined process template corresponding to the minimum normalized distance among the normalized distances will be used. As the current local action subsequence For matching processes, record the process category and its start and end times in the assembly video; otherwise, from a continuous action sequence, keep the starting action of the subsequence unchanged, increase the number of actions in the subsequence to obtain a new local action subsequence. Repeat (2) until a matching operation for a subsequence starting with the current starting action is determined; (3) Using the current local action subsequence The number of actions is used as the sliding step size to extract new local action subsequences. , then re-execute (2) until the continuous action sequence has been completely traversed.

[0011] Furthermore, after obtaining the process identification results, the method also includes: determining the time interval of each action contained in each process based on the continuous action sequence, thereby organizing the process identification results into a hierarchical structure representation: process - included actions - time interval of each action; After obtaining the attitude quantization index, the method further includes: aligning the attitude quantization index with the structured representation of the process identification result, and expanding the hierarchical structured representation to: process - included actions - time interval of each action - attitude quantization index of each action. Based on the extended hierarchical structure representation, the posture quantification index and process identification results are represented by text structure.

[0012] Furthermore, the method for quantifying the elbow joint angle deviation is as follows: construct the elbow joint angle based on key points of the shoulder joint, elbow joint, and wrist, and compare the actual elbow joint angle within the current movement range with the standard reference angle range to obtain the elbow joint angle deviation; The method for quantifying wrist trajectory efficiency is as follows: taking the key points of the wrist of the main operating hand as the analysis object, calculate the relationship between the cumulative length of the actual wrist movement trajectory and the initial and final displacements within the current action range; The quantification method of wrist acceleration energy is as follows: based on the positional changes of the key points of the main operating hand wrist in the assembly video between consecutive frames, calculate the velocity, acceleration and jerk, and count the jerk energy in the current action range; The method for quantifying the height difference between the two shoulders is to construct a shoulder balance index by utilizing the difference in the longitudinal coordinates of key points on the left and right shoulders.

[0013] Furthermore, the calculation method for the elbow joint angle deviation is as follows: Let the first t The coordinates of the key points of the left shoulder joint, left elbow joint, and left wrist in the frame are as follows: , and p, then the angle of the left elbow joint Defined as:

[0014] in, This represents the transpose of a vector. The norm of a vector Positive numbers are introduced to prevent the denominator from being zero; Right elbow angle The calculation method is the same as that for the left elbow joint angle. In the current action range elbow joint angle deviation Defined as:

[0015] in, Indicates the starting frame of the current action interval. This indicates the end frame of the current action interval. This indicates the total number of frames in the current action interval. This represents the absolute value operation; and These represent the reference angles of the left and right elbow joints within the current movement range, respectively. The wrist trajectory efficiency is calculated as follows: In the current action range The cumulative length of the actual wrist movement trajectory inside L Defined as:

[0016] In the current action range The Euclidean distance between the initial and final positions is defined as the first and last displacements. D , represented as: ; The wrist trajectory efficiency index Defined as: ,in, The key point of the main operating hand wrist is at the first t Frame position coordinates, A positive number introduced to prevent the denominator from being zero.

[0017] Furthermore, the calculation method for the wrist acceleration energy is as follows: Let the key point of the main operating hand wrist be at the 1st t The position coordinates of the frame are The time interval between two adjacent frames is Then the first t Velocity vector of key points on the wrist in frame , No. t Acceleration vector of key points on the wrist in frame and the t Frame wrist key point acceleration vector They are defined as follows:

[0018] In the current action range Within, the square of the magnitude of the jerk vector is taken as the jerk energy. , represented as: ; The method for calculating the height difference between the two shoulders is as follows: Let the first t The vertical coordinates of the left and right shoulder keypoints in the frame are respectively and Then in the action range Internal, shoulder height difference index Defined as: .

[0019] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0020] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0021] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages: 1. This invention provides a method for evaluating the standardization of assembly actions by integrating posture and semantic understanding. The method obtains human posture sequences from assembly videos. Based on these sequences, and combined with standard operating procedures and predefined process templates, process identification results and posture quantification indicators are obtained and used as input to a large language model. Then, the assembly process is modeled hierarchically from two levels: process semantic compliance and posture execution rationality. A comprehensive evaluation is achieved by combining an assembly standard knowledge base with the large language model, resulting in a standardization evaluation result with overall evaluation conclusions, problem identification, evidence, and improvement suggestions. This invention does not directly input assembly videos into the large language model for end-to-end evaluation. Instead, it addresses the characteristics of small differences in fine-grained actions and complex standardization judgment criteria in assembly scenarios. It first performs hierarchical and detailed modeling of the video content at both the process semantic level and the action posture level. Then, the resulting structured results are converted into text form and combined with assembly standard knowledge before being input into the large language model for comprehensive reasoning. This approach allows the large language model to complete standardization evaluations under clear process semantic constraints and posture quantification evidence, thereby improving the accuracy, stability, and interpretability of the evaluation results. This method is applicable to industrial assembly scenarios with clear process constraints and operational specifications, and can output structured, interpretable, and traceable evaluation results for the complete assembly process.

[0022] 2. For process identification, this invention employs a sliding window approach to extract local action subsequences from a continuous sequence of actions and aligns these subsequences with a set of standard process templates. The window length of the sliding window can be set according to the number of actions contained in the standard process template to be matched, and can be adjusted within a preset range to accommodate variations in process length. Considering that different local action subsequences and different process templates may have different lengths, directly comparing matching distances could easily lead to a situation where "the longer the sequence, the larger the distance," thus affecting the fair comparison between different matching results. Therefore, this invention further normalizes the matching distance, obtaining the process identification result corresponding to the assembly process based on the normalized distance, providing a semantic basis at the process level for subsequent attitude quantization and knowledge reasoning. Attached Figure Description

[0023] Figure 1A flowchart illustrating a method for evaluating the standardization of assembly actions that integrates posture and semantic understanding, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the hierarchical representation of the process identification structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the extended hierarchical representation provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structured representation of text as input to a large language model, provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] Example 1 A method for evaluating the normativity of assembly actions that integrates posture and semantic understanding, such as Figure 1 As shown, it includes: Acquire assembly videos and extract chronologically ordered sequences of human postures for each operator. Based on the human posture sequence of each operator, action recognition is performed to obtain candidate action category recognition results corresponding to multiple time windows, including multiple action categories and their corresponding confidence information; using the standard operating procedure as a prior constraint, the action sequence optimization decoding is performed on the candidate action category recognition result sequence composed of all time windows to obtain a continuous action sequence; The continuous action sequence is matched with a predefined process template to obtain the process recognition result. Based on the time window distribution corresponding to each action in the continuous action sequence, combined with the window length and window sliding step, the start and end time intervals of each action in the assembly video are determined. The human posture within the corresponding time interval is quantitatively analyzed to obtain posture quantification indicators, including: elbow joint angle deviation, which measures the degree of deviation of the actual action from the reference pattern in terms of joint geometry; wrist trajectory efficiency, which reflects the degree of wrist movement; wrist jerk energy, which describes the intensity of changes in velocity and acceleration during the action; and shoulder height difference, which measures whether the shoulders maintain relative balance during the action execution. The process identification results and posture quantification indicators are represented by text structure. Based on the text structure representation, a retrieval is performed in the assembly specification knowledge base. The text structure representation and retrieval results are input into a large language model. Based on the large language model, the assembly action standardization evaluation results are output, including process semantic compliance and posture execution rationality. Process semantic compliance is used to determine whether there are missing key processes, incorrect process sequence, or non-assembly violations at the action level in the assembly process. Posture execution rationality is used to determine whether there are safety risks, stability risks, or ergonomic risks in the execution of each action.

[0026] This invention provides a method for evaluating the standardization of assembly actions by integrating posture and semantic understanding. The method obtains human posture sequences from assembly videos. Based on these sequences, and combined with standard operating procedures and predefined process templates, it generates process identification results and posture quantification indicators, which are then input into a large language model. Subsequently, the assembly process is modeled hierarchically from two perspectives: semantic compliance of the process and rationality of posture execution. A comprehensive evaluation is achieved by combining an assembly standard knowledge base with the large language model, resulting in a standardization evaluation result with overall evaluation conclusions, problem identification, evidence, and improvement suggestions. This method is applicable to industrial assembly scenarios with clear process constraints and operational standard requirements, and can output structured, interpretable, and traceable evaluation results for the complete assembly process.

[0027] This embodiment of the method includes steps such as acquiring assembly videos and human posture sequences, constructing candidate action sequences, optimizing and decoding action sequences, matching work processes, quantifying and analyzing postures, mapping structured text, enhancing knowledge retrieval, and comprehensive reasoning using a large language model. In other words, it divides the evaluation of assembly action standardization into three stages: process semantic analysis, posture quantification analysis, and knowledge-driven reasoning. This approach simultaneously considers the rationality of the assembly process and the rationality of action execution, overcoming the shortcomings of existing methods that only output action categories or single scores.

[0028] Among them, by introducing a motion sequence optimization decoding mechanism constrained by standard operating procedures, the consistency and stability of motion sequences in long assembly videos are improved, and key process omissions, sequence abnormalities and violations can be identified more reliably.

[0029] Posture quantification indicators include at least one or more of the following: elbow joint angle deviation, wrist trajectory efficiency, wrist acceleration energy, and shoulder height difference. These indicators characterize the degree of posture deviation, movement smoothness, and body coordination during motion execution. By constructing posture quantification indicators such as elbow joint angle deviation, wrist trajectory efficiency, wrist acceleration energy, and shoulder height difference, vague assembly specifications are transformed into calculable underlying physical evidence, enhancing the objectivity of the evaluation criteria.

[0030] By employing an assembly specification knowledge base and retrieval enhancement mechanisms, external normative constraints are imposed on the reasoning process of the large language model, thereby improving the reliability, interpretability, and traceability of normative judgments. The final output is a structured evaluation result that includes overall evaluation conclusions, problem identification, evidence, and improvement suggestions, facilitating deployment and application in scenarios such as industrial assembly monitoring, quality traceability, and work training.

[0031] This embodiment does not directly input assembly video into a large language model for end-to-end evaluation. Instead, considering the characteristics of assembly scenarios—small differences in fine-grained movements and complex criteria for standardization judgments—it first performs layered and refined modeling of the video content at both the process semantic level and the action posture level. Then, the resulting structured results are converted into text form and combined with assembly standard knowledge before being input into the large language model for comprehensive reasoning. This approach allows the large language model to complete standardization evaluations under clear process semantic constraints and supported by quantitative posture evidence, thereby improving the accuracy, stability, and interpretability of the evaluation results.

[0032] The following is a detailed explanation of each step in the method of this embodiment: (1) Human posture sequence acquisition.

[0033] First, the assembly video to be evaluated is acquired, and a human pose estimation method is used to extract a sequence of human poses arranged in chronological order from the assembly video (each pose consists of key points). The human pose sequence can be a two-dimensional sequence of human key point coordinates, or an extended representation containing two-dimensional human key point coordinates and key point coordinate confidence information.

[0034] In one optional implementation, the assembly video can be uniformly sampled at a preset frame rate, and the human skeleton result of each frame can be obtained by combining a human detection and pose estimation network. Then, the key point results of the same operator are organized into a temporal pose sequence according to the time sequence, which serves as the basic input for subsequent action recognition and pose quantification analysis.

[0035] (2) Construction of candidate action recognition result sequence based on sliding window Action recognition is performed based on the human posture sequence of each operator. A fixed-length time window is set on the posture sequence, and overlapping sliding is performed with a preset step size (considering that there are usually transition states between adjacent actions during the assembly process, and the action boundaries are difficult to accurately divide). Each local time segment is input into the action recognition model (existing), and the action category recognition result of the corresponding time window (including the category and its corresponding confidence information) is obtained.

[0036] Considering the strong similarity between adjacent actions and the potential introduction of local recognition noise by pose estimation errors, for each time window, the top k action category recognition results with high confidence are retained as candidates, thus forming a sequence of candidate action category recognition results consisting of the candidate action category recognition results corresponding to multiple time windows. Given that human pose sequences can characterize the joint movement changes during assembly action execution, the action recognition model can employ a temporal action recognition network based on skeleton sequences.

[0037] (3) Optimized decoding of continuous action sequences based on standard operating procedures Since directly splicing window-level prediction results can easily lead to discontinuous action sequences, this embodiment introduces standard operating procedures as prior constraints at the action sequence level to perform overall optimization decoding of candidate action category prediction results.

[0038] Specifically, in one optional implementation, a state space can be constructed with action category sets and skipped states as nodes to establish a Hidden Markov Model (HMM). Action category nodes represent valid action labels, and skipped state nodes represent invalid actions or time windows with uncertain identification. Based on the confidence information of each action in the candidate action prediction results for each time window, the emission probability of each action in that time window is calculated. State transition probabilities are constructed according to the action sequence relationships defined in the standard operating procedure. Then, a sequence decoding algorithm based on the HMM is used to obtain the globally optimal path based on the emission probabilities of all candidate actions in each time window and the state transition probabilities between time windows. A continuous action sequence conforming to the assembly process constraints is obtained based on the globally optimal path.

[0039] Based on the emission probability and state transition probability, a hidden Markov model combined with an improved Viterbi algorithm can be used to decode the optimal path, resulting in the globally optimal path, i.e., a continuous sequence of actions. This approach can comprehensively consider the local (time window) confidence of candidate actions, improve the consistency of the action sequence by utilizing process constraints, and allow skipping unreliable time segments.

[0040] (4) Process identification based on process template matching After obtaining the optimized sequence of continuous actions, the sequence is further matched with a predefined process template to obtain the process identification result (i.e., determining which actions constitute a process). Since a process in industrial assembly is usually composed of multiple semantically related and temporally continuous atomic actions, process-level identification needs to be completed at the action sequence level.

[0041] As a preferred implementation, the method for matching the continuous action sequence with the predefined process template is as follows: (1) Starting from the initial action of the continuous action sequence, extract the local action subsequence with the same number of actions as the shortest predefined process template. ; (2) Calculate the relationship between this local action sub-sequence and each predefined process template. Matching distance between Normalizing each matching distance, it is expressed as: In the formula, This represents the number of action steps in a local action subsequence during the matching process; if there is a distance below a preset threshold among the normalized distances, then the predefined process template corresponding to the minimum normalized distance among the normalized distances will be used. As the current local action subsequence For matching processes, record the process category and its start and end times in the assembly video; otherwise, from a continuous action sequence, keep the starting action of the subsequence unchanged, increase the number of actions in the subsequence to obtain a new local action subsequence. Repeat (2) until a matching operation for a subsequence starting with the current starting action is determined; (3) Using the current local action subsequence The number of actions is used as the sliding step size to extract new local action subsequences. , then re-execute (2) until the continuous action sequence has been completely traversed.

[0042] In this preferred embodiment, a sliding window approach is used to extract local action subsequences from a continuous action sequence, and these local action subsequences are then aligned and matched with a set of standard process templates. The window length of the sliding window can be set according to the number of actions contained in the standard process template to be matched, and can be scaled up or down within a preset range to adapt to changes caused by different process lengths. For each local action subsequence obtained by sliding, the Dynamic Time Warping (DTW) method can be used to calculate the matching distance between the subsequence and each process template. Generally, the smaller the matching distance, the more similar the two are. Considering that the lengths of different local action subsequences and different process templates may be different, directly comparing the matching distances can easily lead to a situation where "the longer the sequence, the larger the distance," thus affecting the fair comparison between different matching results. Therefore, this embodiment further normalizes the matching distance. Preferably, the normalized distance... It can be represented as:

[0043] in, S Represents a local action subsequence. P This represents a standard process template. Indicates the matching distance. This represents the length of the optimal alignment path in DTW, i.e., the current number of action steps in the matching process. The meaning of the above normalization process is to convert the total matching error into the average matching error per step, thereby enabling more reasonable comparisons between sequences of different lengths.

[0044] The matching success is determined based on the normalized distance; when the distance is lower than the preset threshold, the current local action subsequence is determined to be a successful match with the corresponding process template, and the corresponding process category and its start and end time range in the original video are recorded.

[0045] Through the above processing, the process identification results corresponding to the assembly process can be obtained, providing semantic basis at the process level for subsequent attitude quantification and knowledge reasoning.

[0046] (5) Hierarchical structure representation of process results To establish a correspondence between the semantics of high-level procedures and the execution process of low-level actions, this embodiment further represents the procedure identification results hierarchically. Specifically, for each identified procedure, it is represented as a set of actions consisting of several consecutive actions, and the start and end time intervals of each action in the original video are recorded.

[0047] In this way, the intermediate representation information obtained during the process of process identification, such as process category, execution order, action composition, and time range, can be further organized into a hierarchical structure of process-action-time interval for each action, such as... Figure 2 As shown. For segments that cannot be attributed to any standard process template, they can be marked as irrelevant actions, abnormal actions, or pending actions, thereby preserving the abnormal information in the original sequence.

[0048] (6) Quantitative analysis of human posture within the action range Based on the hierarchical structure of the process identification results, the time interval of the action is determined, and the human posture sequence within the corresponding action interval is quantitatively analyzed to obtain posture quantification indicators. The posture quantification stage does not directly provide a final normative conclusion, but is used to extract objective physical evidence reflecting the execution state of the action.

[0049] In a preferred embodiment, the attitude quantification index includes at least one of the following: (1) Elbow joint angle deviation. The elbow angle is constructed based on key points of the shoulder, elbow, and wrist joints. The actual elbow joint angle within the current movement range is compared with the standard reference angle range to obtain a joint angle deviation index, which characterizes the degree to which the movement geometry deviates from the reference pattern. Specifically, let the first... t The coordinates of the key points of the left shoulder joint, left elbow joint, and left wrist in the frame are as follows: , and p, then the angle of the left elbow joint Defined as:

[0050] in, This represents the transpose of a vector. The norm of a vector A tiny positive number introduced to prevent the denominator from being zero. Right elbow angle. The same method can be used to define them. If the reference angles of the left and right elbow joints under the current process are obtained from standard operating procedures or advanced technician work data, respectively... and Then in the action range elbow joint angle deviation It can be defined as:

[0051] in, Indicates the starting frame of the current action interval. This indicates the end frame of the current action interval. This represents the total number of frames in the action interval. This represents absolute value calculation. The elbow joint angle deviation index is used to measure the degree of deviation of the actual movement from the reference pattern in terms of joint geometry.

[0052] (2) Wrist trajectory efficiency. Taking the key points of the main operating hand's wrist as the analysis object, the relationship between the total length of the wrist trajectory and the beginning and end displacements within the current movement range is calculated to reflect whether there is obvious detour, retreat, or over-adjustment in the hand trajectory. Specifically, let the key points of the main operating hand's wrist be at the... t The coordinates of the frame are Then in the action range Total length of the trajectory within L Defined as:

[0053] Head and tail displacement D Defined as:

[0054] Based on this, the wrist trajectory efficiency index It can be defined as:

[0055] in, The key point of the main operating hand wrist is at the first t Frame position coordinates, L It represents the cumulative length of the actual wrist movement path within the range of motion. D This represents the Euclidean distance between the start and end positions of the action range. A very small positive number is introduced to prevent the denominator from being zero. The trajectory efficiency index is used to reflect the degree of detour in wrist movement. The larger the index value, the more ineffective movements, repeated adjustments, or path redundancy may exist during the execution of the movement.

[0056] (3) Wrist accelerometer energy. Velocity, acceleration, and accelerometer are calculated based on the positional changes of the wrist key points between consecutive frames. The accelerometer energy within the current movement interval is then statistically analyzed to reflect the drastic changes in velocity and acceleration during the movement, thereby characterizing the smoothness of the movement. Specifically, let the wrist key point of the main operating hand be at the [missing information - likely a frame number]. t The position coordinates of the frame are The time interval between two adjacent frames is Then the first t frame corresponding speed acceleration And accelerometer They are defined as follows:

[0057] Furthermore, within the action range Internal acceleration energy It can be defined as:

[0058] in, Indicates the first t Velocity vectors of key points on the wrist in a frame. Indicates the first t Acceleration vector of key points on the wrist in frame. Indicates the first t The acceleration vector of key points on the wrist in the frame. This represents the square of the magnitude of the jerk vector. The jerk energy is used to characterize the drastic changes in velocity and acceleration during the action. The larger the value, the more unstable the action is, and the more likely there is sudden force, impact, or discontinuous adjustment.

[0059] (4) Shoulder height difference. A shoulder balance index is constructed using the longitudinal coordinate difference of key points on the left and right shoulders to reflect whether there is significant shoulder shrugging, tilting, or compensatory behavior in the upper body. Specifically, let the first... t The vertical coordinates of the left and right shoulder keypoints in the frame are respectively and Then in the action range Internal, shoulder height difference index It can be defined as:

[0060] in, Indicates the first t The vertical coordinates of the key point on the left shoulder of the frame. Indicates the first t Vertical coordinates of key points on the right shoulder of the frame. T This indicates the total number of frames within the current action range. The shoulder height difference index is used to measure whether the shoulders maintain relative balance during the execution of the action. The larger the index value, the more likely the upper body is to have postural imbalance, shrugging, tilting, or compensatory behavior.

[0061] The above indicators can be statistically analyzed separately within the action time interval, and together they constitute a quantitative description of the execution method of a single assembly action.

[0062] (7) Hierarchical alignment representation of attitude quantization results To enhance the traceability of subsequent reasoning, this embodiment aligns the attitude quantification results with the process results and action ranges, forming a hierarchical representation of process-action-attitude indicators, such as... Figure 3 As shown. This method clarifies the process semantics, action segments, and time positions corresponding to each attitude quantification index.

[0063] In this embodiment, the action-level posture indicators are not forcibly compressed or pre-weighted. Instead, the original quantization results are preserved as completely as possible so that the large language model can perform targeted reasoning based on specific specification clauses in the subsequent analysis stage.

[0064] (8) Structured text mapping and assembly specification knowledge retrieval Since large language models primarily use text as input, it is necessary to convert the process recognition results and pose quantization results into structured text descriptions, such as... Figure 4 As shown. Preferably, a templated text organization method is used to organize the process number, process name, time interval of the action, and attitude quantification index into fixed fields.

[0065] For example, a certain process can be described as: process number, process name, start and end time interval, list of included actions, elbow joint angle deviation statistics, wrist trajectory efficiency statistics, wrist acceleration energy statistics, and shoulder height difference statistics, etc.

[0066] Furthermore, an assembly specification knowledge base is constructed, which includes external knowledge documents such as assembly specifications, process standards, and safety constraints. During the evaluation phase, based on the semantics of the current process, the action context, and abnormal posture behavior, relevant specification clauses are retrieved from the assembly specification knowledge base. The retrieval results are then input as additional context into the large language model, thereby forming a retrieval-enhanced knowledge injection mechanism.

[0067] Therefore, as a preferred implementation, the process identification results and posture quantification results can be organized in a hierarchical structure to establish a correspondence between processes, action ranges, and posture indicators; this facilitates subsequent structured text representation, that is, by further adopting a templated text organization method, the process number, process name, action duration range, and key posture indicator statistical results can be mapped into structured text.

[0068] Specifically, after obtaining the process identification results, the method also includes: determining the time interval of each action contained in each process based on the continuous action sequence, thereby organizing the process identification results into a hierarchical structure representation: process - actions contained in - time interval of each action; After obtaining the attitude quantization index, the method further includes: aligning the attitude quantization index with the structured representation of the process identification result, and expanding the hierarchical structured representation to: process - included actions - time interval of each action - attitude quantization index of each action. Based on the extended hierarchical structure representation, the posture quantification index and process identification results are represented by text structure.

[0069] In this preferred embodiment, the attitude quantification results are represented in a hierarchical manner of process-action-attitude index, which enables the attitude quantification evidence to be aligned with the specific process semantics and action execution segments, thereby improving the pertinence and traceability of subsequent normative reasoning.

[0070] (9) Normative comprehensive reasoning based on large language models The process identification results, posture quantification results, and retrieved standard clauses are uniformly input into the large language model. The large language model then performs comprehensive reasoning on the semantic compliance of the process and the rationality of posture execution, and outputs the evaluation results of the assembly action standardization.

[0071] Among them, semantic compliance of the assembly process refers to whether the assembly process conforms to the predefined process specifications in terms of process composition, execution sequence, and process constraints. It is used to determine whether there are any missing key processes, incorrect sequences, or violations in the assembly process. The rationality of posture execution is used to determine whether there are any safety, stability, or ergonomic risks in the execution of actions. The large language model generates explanations of the sources and causes of anomalies by jointly analyzing process anomalies, posture index anomalies, and specification clauses.

[0072] Preferably, the normative evaluation results output by the large language model include at least: an overall evaluation conclusion, problem location information, evidence, and improvement suggestions. The overall evaluation conclusion is used to give a general judgment on whether the assembly process is standardized; the problem location information is used to identify abnormal procedures or abnormal action segments; the evidence is used to explain the corresponding procedure identification results, posture quantification indicators, and the source of the standard clauses; and the improvement suggestions are used to provide corrective opinions for the operation specifications.

[0073] (10) Optional parameters and implementation methods In different implementation scenarios, the sliding window length, window step size, number of candidate actions k, skip state parameters in sequence decoding, DTW matching threshold, and number of knowledge retrieval returned entries can all be set according to the specific assembly object and process flow.

[0074] This invention does not limit the specific network structure and implementation framework of the action recognition model, posture estimation model, large language model, or knowledge retrieval module. As long as the above-mentioned process recognition, posture quantification, and standardized reasoning process can be achieved, they all fall within the protection scope of this invention.

[0075] The method flow of the present invention will be further explained below with reference to a specific assembly scenario.

[0076] For example, in an electrical connector assembly scenario, the process first acquires a video of the operator completing the assembly task and the corresponding human posture sequence. Then, based on a sliding window approach, motion recognition is performed on the human posture sequence to obtain candidate motion results. These results are then combined with standard operating procedures for sequence optimization and decoding to generate a continuous motion sequence. Further, a process template matching method is used to identify processes such as "inspection," "pre-connection," and "installation," along with their corresponding time intervals. Subsequently, for the key motion intervals in the "installation" process, quantitative posture indicators such as elbow joint angle deviation, wrist trajectory efficiency, wrist accelerometer energy, and shoulder height difference are extracted. The process recognition results and posture quantification results are then mapped to structured text, enhanced by retrieval based on assembly specification knowledge, and input into a large language model. Finally, the output includes an overall evaluation conclusion, problem location, evidence, and improvement suggestions for the standardization of the assembly actions. If the recognition results show a lack of a review process, and the insertion stage exhibits significant trajectory detours and large motion fluctuations, it can be determined that the assembly process has standardization issues at both the process semantic level and the posture execution level.

[0077] Example 2 This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0078] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0079] The relevant technical solutions are the same as above, and will not be repeated here.

[0080] Example 3 This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0081] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0082] The relevant technical solutions are the same as above, and will not be repeated here.

[0083] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the standardization of assembly actions by integrating posture and semantic understanding, characterized in that, include: Acquire assembly videos and extract chronologically ordered sequences of human postures for each operator. Based on the human posture sequence of each operator, action recognition is performed to obtain candidate action category recognition results corresponding to multiple time windows, including multiple action categories and their corresponding confidence information; using the standard operating procedure as a prior constraint, the action sequence optimization decoding is performed on the candidate action category recognition result sequence composed of all time windows to obtain a continuous action sequence; The continuous action sequence is matched with a predefined process template to obtain the process recognition result. Based on the time window distribution corresponding to each action in the continuous action sequence, combined with the window length and window sliding step, the start and end time intervals of each action in the assembly video are determined. The human posture within the corresponding time interval is quantitatively analyzed to obtain posture quantification indicators, including: elbow joint angle deviation, which measures the degree of deviation of the actual action from the reference pattern in terms of joint geometry; wrist trajectory efficiency, which reflects the degree of wrist movement; wrist jerk energy, which describes the intensity of changes in velocity and acceleration during the action; and shoulder height difference, which measures whether the shoulders maintain relative balance during the action execution. The process identification results and posture quantification indicators are represented by text structure. Based on the text structure representation, a retrieval is performed in the assembly specification knowledge base. The text structure representation and retrieval results are input into a large language model. Based on the large language model, the assembly action standardization evaluation results are output, including process semantic compliance and posture execution rationality. Process semantic compliance is used to determine whether there are missing key processes, incorrect process sequence, or non-assembly violations at the action level in the assembly process. Posture execution rationality is used to determine whether there are safety risks, stability risks, or ergonomic risks in the execution of each action.

2. The assembly action standardization evaluation method as described in claim 1, characterized in that, The method for action recognition is as follows: A fixed-length time window is set on the human posture sequence, and overlapping sliding is performed with a preset step size. Each local time segment is input into the action recognition model to obtain the action category prediction result of the corresponding time window, which includes multiple action categories and their corresponding confidence information. The top k action category prediction results with higher confidence are retained as candidates.

3. The assembly action standardization evaluation method as described in claim 1, characterized in that, The method for optimizing action sequence decoding is as follows: A state space is constructed with action category sets and skip states as nodes to build a hidden Markov model; where action category nodes are used to represent valid action labels, and skip state nodes are used to represent invalid actions or time windows for identification uncertainty. Based on the confidence information of each action in the candidate action category prediction results for each time window, calculate the emission probability of each action in that time window; Construct state transition probabilities based on the sequence of actions defined in the standard operating procedure; A sequence decoding algorithm based on a hidden Markov model is adopted to obtain the globally optimal action path based on the emission probability of each candidate action in each time window and the state transition probability between time windows, which serves as a continuous action sequence that conforms to the assembly process constraints.

4. The assembly action standardization evaluation method as described in claim 1, characterized in that, The method for matching continuous action sequences with predefined process templates is as follows: (1) Starting from the initial action of the continuous action sequence, extract the local action subsequence with the same number of actions as the shortest predefined process template. ; (2) Calculate the relationship between this local action sub-sequence and each predefined process template. Matching distance between Normalizing each matching distance, it is expressed as: In the formula, This represents the number of action steps in a local action subsequence during the matching process; if there is a distance below a preset threshold among the normalized distances, then the predefined process template corresponding to the minimum normalized distance among the normalized distances will be used. As the current local action subsequence For matching processes, record the process category and its start and end times in the assembly video; otherwise, from a continuous action sequence, keep the starting action of the subsequence unchanged, increase the number of actions in the subsequence to obtain a new local action subsequence. Repeat (2) until a matching operation for a subsequence starting with the current starting action is determined; (3) Using the current local action subsequence The number of actions is used as the sliding step size to extract new local action subsequences. , then re-execute (2) until the continuous action sequence has been completely traversed.

5. The assembly action standardization evaluation method as described in claim 1, characterized in that, After obtaining the process identification results, the method further includes: determining the time interval of each action contained in each process based on the continuous action sequence, thereby organizing the process identification results into a hierarchical structure representation: process - included actions - time interval of each action; After obtaining the attitude quantization index, the method further includes: aligning the attitude quantization index with the structured representation of the process identification result, and expanding the hierarchical structured representation to: process - included actions - time interval of each action - attitude quantization index of each action. Based on the extended hierarchical structure representation, the posture quantification index and process identification results are represented by text structure.

6. The assembly action standardization evaluation method as described in claim 1, characterized in that, The method for quantifying the elbow joint angle deviation is as follows: construct the elbow joint angle based on key points of the shoulder joint, elbow joint, and wrist, and compare the actual elbow joint angle within the current movement range with the standard reference angle to obtain the elbow joint angle deviation. The method for quantifying wrist trajectory efficiency is as follows: taking the key points of the wrist of the main operating hand as the analysis object, calculate the relationship between the cumulative length of the actual wrist movement trajectory and the initial and final displacements within the current action range; The quantification method of wrist acceleration energy is as follows: based on the positional changes of the key points of the main operating hand wrist in the assembly video between consecutive frames, calculate the velocity, acceleration and jerk, and count the jerk energy in the current action range; The method for quantifying the height difference between the two shoulders is to construct a shoulder balance index by utilizing the difference in the longitudinal coordinates of key points on the left and right shoulders.

7. The assembly action standardization evaluation method as described in claim 6, characterized in that, The elbow joint angle deviation is calculated as follows: Let the first t The coordinates of the key points of the left shoulder joint, left elbow joint, and left wrist in the frame are as follows: , and Then the angle of the left elbow joint Defined as: in, This represents the transpose of a vector. The norm of a vector Positive numbers are introduced to prevent the denominator from being zero; Right elbow angle The calculation method is the same as that for the left elbow joint angle. In the current action range elbow joint angle deviation Defined as: in, Indicates the starting frame of the current action interval. This indicates the end frame of the current action interval. This indicates the total number of frames in the current action interval. This represents the absolute value operation; and These represent the reference angles of the left and right elbow joints within the current movement range, respectively. The wrist trajectory efficiency is calculated as follows: In the current action range The cumulative length of the actual wrist movement trajectory within the wrist. L Defined as: In the current action range The Euclidean distance between the initial and final positions is defined as the first and last displacements. D , is represented as: ; The wrist trajectory efficiency index Defined as: ,in, The key point of the main operating hand wrist is at the first t Frame position coordinates, A positive number introduced to prevent the denominator from being zero.

8. The assembly action standardization evaluation method as described in claim 6, characterized in that, The calculation method for the wrist acceleration energy is as follows: Let the key point of the main operating hand wrist be at the 1st t The position coordinates of the frame are The time interval between two adjacent frames is Then the first t Velocity vector of key points on the wrist in frame , No. t Acceleration vector of key points on the wrist in frame and the t Frame wrist key point acceleration vector They are defined as follows: In the current action range Within, the square of the magnitude of the jerk vector is taken as the jerk energy. , is represented as: ; The method for calculating the height difference between the two shoulders is as follows: Let the first t The vertical coordinates of the left and right shoulder keypoints in the frame are respectively and Then within the action range Internal, shoulder height difference index Defined as: 。 9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium resides to perform the steps of the method as described in any one of claims 1 to 8.

Images