A railway joint control operation training action evaluation method and system based on image sequence analysis

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a method and system for evaluating the practical training actions of railway joint control operations based on image sequence analysis. It includes: S1, obtaining the time series of the action features of the trainees, and the action features at any moment include at least two different types of features; aligning the time series of the action features of the trainees and the template standard sequence; S2, calculating the similarity between the action features of any type at any moment in the time series of the action features of the trainees and the action features of the corresponding type at the corresponding moment in the template standard sequence as the individual similarity of the type of feature at that moment; then, weightedly adding the individual similarities of each type of feature at that moment to obtain the comprehensive similarity between the trainee's action at that moment and the standard action of the training; evaluating the practical training actions through the comprehensive similarities at each moment. The present invention solves the technical problem of low accuracy in the recognition of teaching practical training actions in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for evaluating railway joint control operation training actions based on image sequence analysis. Background Art

[0002] Railway joint control operation refers to the use of wireless train dispatching communication joint control equipment to communicate in accordance with the prescribed joint control procedures and standards, to prompt train operation and on-site safety operation information, to ensure the effective implementation of joint control measures such as train operation, crossing operations, and construction operations, so as to be able to control and eliminate safety hazards in advance. Railway joint control operations mainly include vehicle-machine joint control, shunting joint control, crossing joint control, construction joint control, and joint control operations at crossings, off-roads, and station areas.

[0003] In the existing railway joint control operation teaching, image recognition is often used to identify the movements of trainees. The Chinese invention patent application with application publication number CN112818800A and application publication date 2021.05.18 discloses a sports movement evaluation method and system based on the depth image of human skeleton points, which obtains the actual distance of the key skeleton points of the action to be evaluated and the actual angle of the vector between the skeleton points, and then compares the actual distance and the actual angle with the standard distance and standard angle of the action, and evaluates the trainee's sports movement according to the comparison result. In the above technical scheme, since the distance and angle features of each action are compared with the distance and angle features of the standard action separately, when the feature values ​​are actually extracted, due to the error in the calculation of the spatial angle and the data point drift of the key points selected by the angle feature during the movement, there will be certain errors in the angle feature compared with the distance feature; however, if only the distance feature is selected as the action feature, some action information will be ignored, resulting in incorrect evaluation results, and the error in action evaluation will be large when the actions are relatively similar. Therefore, the above technical scheme has the technical problem of low recognition accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a railway joint control operation training action evaluation method and system based on image sequence analysis, so as to solve the technical problem of low recognition accuracy of teaching training action recognition in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a technical solution for evaluating a railway joint control operation training action based on image sequence analysis: a railway joint control operation training action evaluation method based on image sequence analysis, the method comprising:

[0006] S1, obtaining a time series of motion features of a trainee, wherein the motion features at any moment include at least two different types of features; aligning the time series of motion features of the trainee with a template standard series;

[0007] The template standard sequence is a time sequence of motion features generated in advance based on the training standard motion template;

[0008] S2. Calculate the similarity between the action features of any type at any moment in the action feature time series of the trainee and the action features of the corresponding type at the corresponding moment in the template standard sequence as the individual similarity of the feature of this type at that moment; then, perform weighted addition of the individual similarities of each type of feature at that moment to obtain the comprehensive similarity between the trainee's action at that moment and the standard action of training; and evaluate the training action through the comprehensive similarities at each moment.

[0009] The beneficial effect of the above technical solution is that the technical solution of the railway joint control operation training action evaluation method based on image sequence analysis of the present invention belongs to an improved invention. When performing action evaluation, the present invention comprehensively considers a variety of different types of features; at the same time, since different types of action features have different effects on the results, the feature type with calculation errors may have a greater impact on the final result, and weights are set for different types of features to balance their effects on the results. The present invention solves the technical problem of low recognition accuracy of teaching training action recognition in the prior art.

[0010] Furthermore, the action features include distance features and angle features; the distance features are used to represent the relative positions between different key points, and the angle features are used to represent the angles between different key point vectors; the key points are nodes obtained through a skeleton point model and used to represent key points of a human body structure, and the key point vectors are vectors composed of any two key points.

[0011] Furthermore, the similarity is represented by the Euclidean distance; accordingly, the Euclidean distance used to represent the comprehensive similarity is calculated according to the following formula:

[0012]

[0013] Where d(i, j) is the Euclidean distance between the feature at the i-th moment in the trainee's action feature time series and the feature at the j-th moment in the template standard sequence; ω1 and ω2 are the weights corresponding to the angle feature and the distance feature, respectively, and ω1+ω2=1; F m (i, k1) is the k1th angle feature value in the angle feature at the i-th moment in the trainee’s action feature time series; F m ′(j,k1) is the k1th angle feature value in the angle feature at the jth moment in the template standard sequence; F m (i, k2) is the k2th distance feature value in the distance feature at the i-th moment in the trainee’s action feature time series; Fm ′(j,k2) is the k2th distance feature value in the distance feature at the jth moment in the template standard sequence; l1 and l2 are the number of angle feature values ​​and distance feature values, respectively; after alignment, the feature of the i-th moment in the trainee's action feature time series is the feature of the j-th moment in the template standard sequence is the corresponding moment.

[0014] Furthermore, the key points include a reference key point and at least one non-reference key point, the reference key points include a right hip key point, a nose key point, a right shoulder key point and a left hip key point; the distance feature is a relative distance between the reference key point and the non-reference key point in the vertical direction of the image coordinate system.

[0015] Furthermore, the comprehensive similarity is calculated according to the following formula:

[0016]

[0017] Among them, s is the comprehensive similarity between the trainee's action feature sequence and the template standard sequence; k is the length of the longer sequence between the trainee's action feature sequence and the template standard sequence; d is the value obtained by summing the Euclidean distance d(i, j) at all times.

[0018] Furthermore, the distance feature is a distance feature obtained after the original distance feature is normalized.

[0019] Furthermore, the action feature time series of the trainee and the template standard sequence are aligned by a dynamic time warping algorithm.

[0020] The present invention also provides a technical solution for a railway joint control operation practical training action evaluation system based on image sequence analysis: a railway joint control operation practical training action evaluation system based on image sequence analysis, comprising a processor, wherein the processor is used to execute a computer program to implement the steps of the railway joint control operation practical training action evaluation method based on image sequence analysis as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of key point vectors of a human body structure according to an embodiment of a method for evaluating a railway joint control operation training action based on image sequence analysis of the present invention;

[0022] Figure 2 It is a method flow chart of an embodiment of a railway joint control operation training action evaluation method based on image sequence analysis of the present invention;

[0023] Figure 3 It is a schematic diagram of two-track hand signal recognition of an embodiment of a railway joint control operation training action evaluation method based on image sequence analysis of the present invention;

[0024] Figure 4 It is a schematic diagram of five-track hand signal recognition of an embodiment of a railway joint control operation training action evaluation method based on image sequence analysis of the present invention;

[0025] Figure 5 It is a schematic diagram of alignment of a template action sequence in an embodiment of a railway joint control operation training action evaluation method based on image sequence analysis of the present invention, in which the template action sequence is a longer sequence;

[0026] Figure 6 It is a schematic diagram of alignment of a relatively long sequence of action sequences to be estimated in an embodiment of a railway joint control operation training action evaluation method based on image sequence analysis of the present invention. DETAILED DESCRIPTION

[0027] The present invention comprehensively considers a variety of different types of features when evaluating actions; at the same time, since different types of action features have different effects on the results, the feature types with calculation errors may have a greater impact on the final results, and weights are set for different types of features to balance their effects on the results. The present invention solves the technical problem of low recognition accuracy in teaching and training action recognition in the prior art.

[0028] Example of railway joint control operation training action evaluation method based on image sequence analysis:

[0029] A method for evaluating the practical training actions of railway joint control operations based on image sequence analysis, such as Figure 2 As shown, the following steps are included:

[0030] 1. Identify key targets such as flags and flashlights through detection models, such as Figure 3 and Figure 4 shown.

[0031] 2. Identify key points of the human body through the skeleton point model, such as Figure 1 shown.

[0032] 3. Extract multi-dimensional features of the training exercise points, including distance features and angle features. Among them, the characteristic changes of angles can reflect the movement trend and movement information of human body movements, but the angle features can only reflect a movement trend of different people doing different movements. For similar movements, the distance features between different key points are also needed to supplement the description of the details of the movement.

[0033] 3.1 Distance Features

[0034] like Figure 2As shown, four relatively stable key points, including the right hip key point 12, the nose key point 0, the right shoulder key point 6, and the left hip key point 11, are selected as reference key points. Different actions can be represented by the relative positions between these four reference key points and other key points in the Y-axis direction, and the Y-axis is the Y-axis of the photo coordinate system.

[0035] Calculate the distance feature. Take the feature distance from the right wrist key point 10 to the right hip key point 12 as an example. Assume that the current coordinates of the right wrist key point are P 10 =(x 10 ,y 10 ), the coordinates of the right hip key point are P 12 =(x 12 ,y 12 )The calculation formula of the distance feature is:

[0036]

[0037] in, Represents the characteristic distance from the right wrist key point 10 to the right hip key point 12.

[0038] Then, the normalized distance features are calculated. When analyzing the actions of trainees, it is a common method to calculate the distance features by the relative positions of key points. However, factors such as the height, body shape, and relative position of different individuals to the camera can have a significant impact on these distance features. For example, even if two people are the same height, if they are in different positions in the image, the size of the skeleton will be different. This difference can cause significant changes in the distance features, thereby affecting the accuracy of the action evaluation model. Specifically, when a person is far away from the camera, his skeleton will appear smaller in the image; while when another person is closer to the camera, his skeleton will appear larger. This change in skeleton size will lead to inconsistency in distance features, which in turn increases the error in establishing the action evaluation calculation model. In order to eliminate the influence of these factors and ensure the consistency of features in different situations, the distance features need to be normalized.

[0039] Taking the characteristic distance from the right wrist key point 10 to the right hip key point 12 as an example, assuming that the coordinates of the neck key point are P 17 =(x 17 ,y 17 ), the coordinates of the key point at the bottom of the spine are P 18 =(x 18 ,y 18 ), the distance feature and the normalized formula are:

[0040]

[0041] h y =y 18-y 17

[0042]

[0043] Among them, h y Indicates the distance from the neck key point to the spine base key point in the Y-axis direction. express Normalized distance feature.

[0044] 3.2 Angle characteristics

[0045] Every two key points connected can form a human body structure vector, and angle values ​​will be formed between the human body structure vectors. In the trainee's movements, the angles between the key point vectors are basically composed of the vectors between the key points such as the elbow, shoulder, wrist, and hip.

[0046] Calculate the angle feature. Take the right wrist-right shoulder-right hip as an example. Assume that the key point coordinates of the right shoulder in the current frame are P6 = (x6, y6). The angle value α is calculated through the geometric relationship between the joint vectors, as shown in the following formula:

[0047]

[0048] in, Indicates P 10 A vector pointing to P6, Indicates P 12 A vector pointing to P6, α represents the angle between the two vectors.

[0049] 3.3 Fusion of distance features and angle features

[0050] In order to reduce the error of action results and improve the accuracy of action recognition, it is necessary to combine distance features and angle features to form more complete action feature information, which is more efficient and practical.

[0051] Assume that the angle feature of the mth action sequence is:

[0052] A m =(α 1 ,α 2 ,α 3 ,…,α k ) T

[0053] The distance feature is:

[0054]

[0055] Where k represents the number of sampled images, and the combination of the two constitutes the action matrix F m =(A m ,…,Dm ), as follows:

[0056]

[0057] 4. Euclidean distance calculation method based on feature weights. Due to the errors in spatial angle calculation and the data point drift of key points selected for angle features during motion, there will be certain errors in angle features compared to distance features. However, if only distance features are selected as action features, some action information will be ignored, resulting in incorrect evaluation results. When the actions are similar, the error in action evaluation is large. Therefore, a multi-feature weighted Euclidean distance calculation method is proposed to improve the accuracy of action recognition.

[0058] 4.1 Calculation of Euclidean distance

[0059] In actual calculation, when only Euclidean distance is used for calculation, it will obviously cause huge errors. Assume that the feature matrix of the action to be measured is F m , template action feature matrix F m ′, the calculation formula is as follows,

[0060]

[0061] Among them, d(i, j) is the action feature time series of the trainee at the i-th moment (i.e., F m ) features and the template standard sequence (i.e., F m ′), the Euclidean distance between the features, F m (i, k) is the kth feature value of the feature at the i-th moment in the time series of the trainee’s action feature; F m ′(j,k) is the kth eigenvalue in the features at the i-th moment in the template standard sequence; l is the number of eigenvalues.

[0062] 4.2 Euclidean distance of feature weights

[0063] Assume that the weights of angle feature and distance feature in action feature are ω1 and ω2 respectively, and the sum of ω1 and ω2 is 1. The modified formula in 4.1 is as follows:

[0064]

[0065] Where d(i, j) is the Euclidean distance between the feature at the i-th moment in the trainee's action feature time series and the feature at the j-th moment in the template standard sequence; ω1 and ω2 are the weights corresponding to the angle feature and the distance feature, respectively, and ω1+ω2=1; F m (i, k1) is the k1th angle feature value in the angle feature at the i-th moment in the trainee’s action feature time series; F m′(j,k1) is the k1th angle feature value in the angle feature at the jth moment in the template standard sequence; F m (i, k2) is the k2th distance feature value in the distance feature at the i-th moment in the trainee’s action feature time series; F m ′(j,k2) is the k2th distance feature value in the distance feature at the jth moment in the template standard sequence; l1 and l2 are the number of angle feature values ​​and distance feature values, respectively; after alignment, the feature of the i-th moment in the trainee's action feature time series is the feature of the j-th moment in the template standard sequence is the corresponding moment.

[0066] The calculation method of ω1 and ω2 is based on the Lagrange multiplier method to adaptively optimize the two weight values. The objective function (E) is set as follows:

[0067]

[0068] Wherein, d(i,j) represents the expected distance between the feature data at the i-th moment and the j-th moment, and the constraint condition ω1+ω2=1.

[0069] The first step is to set the Lagrangian function, the formula is as follows:

[0070] ξ(ω1,ω2,λ)=E(ω1,ω2)+λ(ω1+ω2-1)

[0071] Among them, λ is the Lagrange coefficient, which introduces the constraint condition.

[0072] In the second step, for the weights ω1, ω2 and λ, their partial derivatives with respect to the function ξ are calculated respectively.

[0073]

[0074] The third step is to solve the system of equations and randomly select the initial values ​​of ω1 and ω2 to be 0.5.

[0075] It should be noted that each of the above-mentioned weight groups corresponds to a template standard sequence, and each template standard sequence represents a type of standard action template for practical training.

[0076] In this embodiment, the same weight ω1 is set for all angle features, and the same weight ω2 is set for all distance features. In other embodiments, weights may be set for each angle feature and distance feature to highlight the impact of angle features or distance features at different positions on the final evaluation result.

[0077] 4.3 Action Timing Alignment

[0078] The dynamic time warping algorithm can handle the situation where the length of time series is different or the speed is inconsistent. It can align two action feature sequences of different lengths. For example, an action sequence to be estimated and a template action sequence are aligned. Figure 5 and Figure 6 shown.

[0079] The algorithm has boundary conditions, continuity and monotonicity. The boundary condition means that the two sequences must match at the beginning and end, and the remaining parts must match in order; continuity means that a feature point has many-to-one and one-to-many situations during the matching and alignment process, and the feature point cannot be matched across points, but can only be aligned with its adjacent points; monotonicity means that the matching increases monotonically over time. When aligning two feature sequences, each matching step in the path must move forward and cannot move backward.

[0080] 5. Use the dynamic time warping method of feature weights to identify the similarity between existing actions and standard actions, and guide the user to make correct training exercises. The similarity calculation formula is as follows:

[0081]

[0082] Among them, s is the comprehensive similarity between the trainee's action feature sequence and the template standard sequence; k is the length of the longer sequence between the trainee's action feature sequence and the template standard sequence; d is the value obtained by summing the Euclidean distance d(i, j) at all times.

[0083] 6. Template standard sequence

[0084] The template sequence contains angle features and distance features, and a unique action template is created for each type of action. Multiple time series are represented by an average time series. This template does not average the data in multiple time series, but uses the size of the dynamic time warping distance as an optimization indicator for obtaining the average sequence.

[0085] Example of railway joint control operation training action evaluation system based on image sequence analysis:

[0086] A railway joint control operation training action evaluation system based on image sequence analysis includes a processor, and the processor is used to execute a computer program to implement the steps of the railway joint control operation training action evaluation method based on image sequence analysis as described above. The specific railway joint control operation training action evaluation method based on image sequence analysis has been introduced in sufficient detail in the above-mentioned railway joint control operation training action evaluation method based on image sequence analysis, and will not be repeated here.

[0087] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments without creative work, or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating the action of railway joint control operation training based on image sequence analysis, characterized in that: The method includes: S1, obtaining a time series of motion features of a trainee, wherein the motion features at any moment include at least two different types of features; aligning the time series of motion features of the trainee with a template standard series; The template standard sequence is a time sequence of motion features generated in advance based on the training standard motion template; S2. Calculate the similarity between the action features of any type at any moment in the action feature time series of the trainee and the action features of the corresponding type at the corresponding moment in the template standard sequence as the individual similarity of the feature of this type at that moment; then, perform weighted addition of the individual similarities of each type of feature at that moment to obtain the comprehensive similarity between the trainee's action at that moment and the standard action of training; and evaluate the training action through the comprehensive similarities at each moment.

2. The method for evaluating the railway joint control operation training action based on image sequence analysis according to claim 1 is characterized in that: The action features include distance features and angle features; the distance features are used to represent the relative positions between different key points, and the angle features are used to represent the angles between different key point vectors; the key points are nodes obtained through a skeleton point model and used to represent key points of the human body structure, and the key point vectors are vectors composed of any two key points.

3. The method for evaluating the railway joint control operation training action based on image sequence analysis according to claim 2 is characterized in that: The similarity is represented by the Euclidean distance; accordingly, the Euclidean distance used to represent the comprehensive similarity is calculated according to the following formula: Where d(i, j) is the Euclidean distance between the feature at the i-th moment in the trainee's action feature time series and the feature at the j-th moment in the template standard sequence; ω1 and ω2 are the weights corresponding to the angle feature and the distance feature, respectively, and ω1+ω2=1; F m (i, k1) is the k1th angle feature value in the angle feature at the i-th moment in the trainee’s action feature time series; F m ′(j,k1) is the k1th angle feature value in the angle feature at the jth moment in the template standard sequence; F m (i, k2) is the k2th distance feature value in the distance feature at the i-th moment in the trainee’s action feature time series; F m ′(j,k2) is the k2th distance feature value in the distance feature at the jth moment in the template standard sequence; l1 and l2 are the number of angle feature values ​​and distance feature values, respectively; after alignment, the feature of the i-th moment in the trainee's action feature time series is the feature of the j-th moment in the template standard sequence is the corresponding moment.

4. The method for evaluating the railway joint control operation training action based on image sequence analysis according to claim 2 is characterized in that: The key points include a reference key point and at least one non-reference key point, the reference key points include a right hip key point, a nose key point, a right shoulder key point and a left hip key point; the distance feature is a relative distance between the reference key point and the non-reference key point in the vertical direction of the image coordinate system.

5. The method for evaluating the railway joint control operation training action based on image sequence analysis according to claim 3 is characterized in that: The comprehensive similarity is calculated according to the following formula: Among them, s is the comprehensive similarity between the trainee's action feature sequence and the template standard sequence; k is the length of the longer sequence between the trainee's action feature sequence and the template standard sequence; d is the value obtained by summing the Euclidean distance d(i, j) at all times.

6. The method for evaluating the railway joint control operation training action based on image sequence analysis according to claim 2 is characterized in that: The distance feature is a distance feature obtained after the original distance feature is normalized.

7. The method for evaluating the railway joint control operation training action based on image sequence analysis according to claim 1 is characterized in that: The action feature time series of the trainee and the template standard sequence are aligned by a dynamic time warping algorithm.

8. A railway joint control operation training action evaluation system based on image sequence analysis, comprising a processor, characterized in that: The processor is used to execute a computer program to implement the steps of the railway joint control operation training action evaluation method based on image sequence analysis as described in any one of claims 1 to 7.

Images