Railway hand signal processing method and system based on multi-modal perception fusion and storage medium

Abstract

The application relates to the technical field of computer vision, in particular to a railway hand signal processing method and system based on multi-modal perception fusion and a storage medium. The multi-modal hand signal instruction category determination is performed by fusing the posture angle parameters, the trajectory parameters and the types and visual attributes of signal equipment, compared with the scheme of single reliance on human posture estimation or pixel-level template comparison, the multi-modal features complement each other; by calling a standard action parameter set corresponding to the recognized hand signal instruction category, the action specification is converted into quantitative evaluation indexes of five dimensions of completion degree, speed, posture angle accuracy, equipment use correctness and visibility, a score result is output, and reliable quantitative evaluation of the hand signal is realized. The application aims to solve the problem of how to realize reliable quantitative evaluation of hand signal actions.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a railway hand signal processing method, system and storage medium based on multimodal perception fusion. Background Technology

[0002] In railway shunting operations, hand signals are an indispensable visual communication method used to direct the coordinated work of shunting locomotives and related personnel, ensuring safe and efficient train operation. Hand signals are categorized into daytime and nighttime hand signals based on the environment, adapting to information transmission needs under varying visibility conditions. Traditional railway hand signal training and assessment primarily rely on manual observation and subjective judgment. Evaluation results are easily influenced by the evaluator's experience and subjective factors, lacking standardized quantitative criteria, and the scoring process is difficult to trace and verify.

[0003] With the development of computer vision technology, some methods for analyzing hand signals using image recognition have emerged. However, traditional gesture recognition methods that rely on single human pose estimation are difficult to meet the evaluation requirements of railway hand signals. In addition, many existing solutions rely on preset standard action templates for pixel-level or feature-level image comparison, which is very sensitive to changes in camera shooting angle, personnel height and body shape, and camera distance. Any slight deviation from the standard template may lead to a significant increase in comparison error, which seriously affects the robustness and generalization ability of the system. Moreover, such methods can usually only provide the similarity between the entire set of actions and the template, and it is difficult to accurately quantify and evaluate the key elements of the action, resulting in weak anti-interference ability.

[0004] In view of this, this application proposes a new method for processing railway hand signals, aiming to achieve reliable quantitative evaluation of hand signal actions. Summary of the Invention

[0005] The main objective of this application is to provide a railway hand signal processing method based on multimodal perception fusion, which aims to solve the problem of how to achieve reliable quantitative evaluation of hand signal actions.

[0006] To achieve the above objectives, this application provides a railway hand signal processing method based on multimodal sensing fusion, the method comprising:

[0007] S10, acquire the video sequence to be identified containing hand signal actions, as well as the type, visual attributes, and position sequence of the signaling device performing the hand signal actions;

[0008] S20, extract feature point coordinate sequences including at least shoulder, elbow and wrist based on the video sequence to be identified, and calculate arm posture angle parameters based on the feature point coordinate sequences;

[0009] S30, perform trajectory tracking and fitting on the position sequence to obtain trajectory parameters, the trajectory parameters including trajectory circularity error, cumulative rotation angle and / or number of round trips;

[0010] S40, determine the target hand signal command category satisfied by the arm posture angle parameter, the trajectory parameter, the type of signal equipment and visual attributes, call the standard action parameter set corresponding to the hand signal command category, quantify and score the posture angle parameter, trajectory parameter and the type of signal equipment and visual attributes based on the standard action parameter set, and output the score result.

[0011] Optionally, S30 includes:

[0012] S31, perform time-series tracking on the position sequence to obtain a trajectory point sequence;

[0013] S32, A smoothing algorithm is used to reduce the detection jitter noise of the trajectory point sequence;

[0014] S33, determine the trajectory circularity error of the denoised trajectory point sequence, and / or determine the cumulative rotation angle of the denoised trajectory point sequence, and / or determine the number of round trips of the denoised trajectory point sequence.

[0015] Optionally, in step S33, determining the trajectory circularity error of the denoised trajectory point sequence includes:

[0016] S331, Fit the center and radius of the trajectory point sequence using the least squares criterion;

[0017] S332, using normalized circular error As the circularity error of the trajectory:

[0018]

[0019] in, ,

[0020] In the formula, RMSE represents the trajectory point. The root mean square error of the distance to the fitted circle (a,b), where r is the fitted radius and N is the number of trajectory points.

[0021] Optionally, the arm posture angle parameters include the elbow joint angle and the arm lifting angle, wherein:

[0022] The elbow joint angle The angle between two vectors determined by the three points of shoulder, elbow, and wrist is calculated as follows:

[0023]

[0024] In the formula, A represents the coordinates of the shoulder point; B represents the coordinates of the elbow point; and C represents the coordinates of the wrist point.

[0025] The arm was raised at an angle The angle between the line vector connecting the shoulder point to the wrist point and the vertical direction of a single frame of the video sequence to be identified is calculated as follows:

[0026]

[0027]

[0028] In the formula, V is the vertical upward unit vector (0, -1), and U is the arm direction vector.

[0029] Optionally, the type of signaling equipment includes signal flags or signal lights, and the visual attributes of the signaling equipment include color category or illumination status.

[0030] Optionally, in step S40, the posture angle parameters, trajectory parameters, and the type and visual attributes of the signaling equipment are quantitatively scored based on the standard motion parameter set, and the scoring results are output, including:

[0031] S41, based on the standard action parameter set, determine the completion score, speed score, posture angle accuracy score, equipment usage correctness score and visibility score of the target hand signal command category respectively;

[0032] S42, based on the weight vector, performs a weighted sum of each sub-score to obtain the total score value as the output score result.

[0033] Optionally, S41 specifically includes:

[0034] The completion score is determined based on the ratio between the trajectory parameters and the reference trajectory parameters in the set of standard motion parameters.

[0035] The motion duration of the video sequence to be identified is obtained, and the speed sub-score is determined based on the relationship between the motion duration and the reference duration interval in the standard motion parameter set.

[0036] The attitude angle accuracy sub-score is determined based on the difference between the attitude angle parameter and the reference angle in the set of standard motion parameters.

[0037] Based on the comparison results between the type and visual attributes of the signal equipment and the standard equipment type and standard visual attributes in the standard action parameter set, the correctness score of equipment use is determined.

[0038] Visible frames in the video sequence to be identified are obtained, and a visibility sub-score is determined based on the proportion of the visible frames in the video sequence to be identified. The visible frames are image frames with a detection confidence level not less than a preset confidence threshold and a brightness level not less than a preset brightness threshold.

[0039] Optionally, the step of determining the weight vector includes:

[0040] Construct a comparison matrix of evaluation indicators ,in, This indicates the importance of indicator i relative to indicator j;

[0041] Based on the evaluation index comparison matrix A, the consistency ratio of the current weight vector w of each evaluation index is calculated. When the consistency ratio is not greater than a preset threshold, the current weight vector w is determined to be the weight vector of the evaluation index.

[0042] Otherwise, increase / decrease the current weight vector w until the consistency ratio is not greater than the preset threshold.

[0043] Furthermore, to achieve the above objectives, this application also provides a railway hand signal processing system, the railway hand signal processing system comprising:

[0044] The acquisition module is used to acquire a video sequence to be identified containing hand signal actions, as well as the type, visual attributes, and position sequence of the signaling device performing the hand signal actions;

[0045] The posture analysis module is used to extract a sequence of feature point coordinates, including at least the shoulder, elbow, and wrist, based on the video sequence to be identified, and to calculate posture angle parameters based on the feature point coordinate sequence. The posture angle parameters include arm posture angle parameters and angular velocity parameters.

[0046] The trajectory tracking and fitting module is used to track and fit the position sequence to obtain trajectory parameters, which include trajectory circularity error, cumulative rotation angle and / or number of round trips;

[0047] The judgment and scoring module is used to determine the target hand signal command category satisfied by the posture angle parameter, the trajectory parameter, the type of signal equipment and the visual attributes, call the standard action parameter set corresponding to the hand signal command category, and perform quantitative scoring on the posture angle parameter, the trajectory parameter and the type of signal equipment and the visual attributes based on the standard action parameter set, and output the scoring result.

[0048] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the railway hand signal processing method based on multimodal perception fusion as described in any of the preceding claims.

[0049] This application has at least the following beneficial effects:

[0050] 1. By fusing posture angle parameters, trajectory parameters, and the type and visual attributes of signaling equipment, multimodal hand signal command category determination is performed. Compared with a single solution that relies on human posture estimation or pixel-level template comparison, the multimodal features complement each other.

[0051] 2. By calling the standard action parameter set corresponding to the identified hand signal instruction category, the action specifications are transformed into five quantitative evaluation indicators in terms of completion, speed, accuracy of posture angle, correct use of equipment, and visibility. The scoring results are output to achieve reliable quantitative evaluation of hand signals. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the railway hand signal processing method based on multimodal perception fusion involved in the embodiments of this application;

[0053] Figure 2 This is a schematic diagram of the architecture of the railway hand signal processing system based on the multimodal perception fusion method of this application.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0056] First Embodiment

[0057] Reference Figure 1 This embodiment provides a railway hand signal processing method based on multimodal sensing fusion, the method including the following steps:

[0058] S10, acquire the video sequence to be identified containing hand signal actions, as well as the type, visual attributes, and position sequence of the signaling device performing the hand signal actions;

[0059] In this embodiment, the video sequence to be identified can be obtained by ordinary video acquisition equipment deployed at the work site or training and assessment site. The video sequence contains continuous image frames of the operator performing hand signal actions.

[0060] Signal equipment refers to the handheld devices / equipment used by workers to perform hand signals. The types of signal equipment may include, but are not limited to, signal flags or signal lights, and visual attributes may include, but are not limited to, color categories or illumination states.

[0061] The position sequence of signaling equipment reflects the movement path of the hand signal action in space. The system performs time-series tracking of this position sequence, extracts the trajectory point sequence, and uses a fitting algorithm to quantify and extract the geometric features of the trajectory, outputting trajectory parameters.

[0062] S20, extract a feature point coordinate sequence including at least shoulder, elbow and wrist based on the video sequence to be identified, and calculate the posture angle parameters based on the feature point coordinate sequence, the posture angle parameters including arm posture angle parameters and angular velocity parameters;

[0063] In some alternative implementations, the extracted hand signal feature point coordinate sequence is preferably constructed by continuously extracting skeletal key point data of the operator, including the shoulder, elbow, and wrist, using a pre-trained deep learning pose estimation model.

[0064] Further, and optionally, the arm posture angle parameters include the elbow joint angle and the arm raise angle. It should be noted that in railway hand signal scenarios, these two angles directly determine the standardized semantics of the hand signal action. For example, when executing a stop signal, industry standards require operators to extend their arms straight and raise them overhead. Biomechanically, this corresponds to a large raise angle (arm nearly vertically upward) and a large elbow joint angle (elbow nearly straight at 180 degrees). If only whether the arm is raised is detected without detecting whether the elbow is straight, the operator might raise the arm with a bent arm, which may look similar visually but does not actually comply with the standard requirements, easily leading to signal transmission distortion. Therefore, extracting the elbow joint angle and arm raise angle together allows for a complete characterization of the hand signal posture standardization from two independent dimensions: limb extension and spatial directionality, thus avoiding the one-sidedness of a single angle feature.

[0065] Specifically, the elbow joint angle The angle between two vectors determined by the three points of shoulder, elbow, and wrist is calculated as follows:

[0066]

[0067] In the formula, A represents the coordinates of the shoulder point; B represents the coordinates of the elbow point; and C represents the coordinates of the wrist point.

[0068] The arm raised at an angle The angle between the line vector connecting the shoulder point to the wrist point and the vertical direction of a single frame of the video sequence to be identified is calculated as follows:

[0069]

[0070]

[0071] In the formula, V is the vertical upward unit vector (0, -1), and U is the arm direction vector.

[0072] It's important to note that in common computer vision image coordinate systems, the y-axis is typically downward, which contradicts the human intuition that upward is positive in physical coordinate systems. Directly using the physical upward vector (0,1) for calculation without processing will result in a completely incorrect lift angle in the image coordinate system. Therefore, when the y-axis of the image coordinate system is downward, the vertical direction vector V must be (0,-1) to represent the upward direction, ensuring that the calculated lift angle conforms to physical intuition and scoring logic. Understandably, if other custom image coordinate systems are used, the direction of vector V should be adaptively adjusted, as long as it points in the physical upward direction.

[0073] The angular velocity parameters can be extracted directly using conventional vision algorithms, and will not be elaborated upon in this embodiment.

[0074] S30, perform trajectory tracking and fitting on the position sequence to obtain trajectory parameters, the trajectory parameters including trajectory circularity error, cumulative rotation angle and / or number of round trips;

[0075] In this embodiment, the two-dimensional positions of each frame output by the signal equipment detection module are concatenated in chronological order to form the original trajectory point sequence.

[0076] Furthermore, and optionally, due to inter-frame jitter during video acquisition, bounding box regression bias in the detection model, and hand tremors of the operator, the original trajectory point sequence usually contains a large amount of high-frequency noise. If geometric fitting is directly performed on such an unsmoothed trajectory, the noise points are prone to deviate from the actual motion path, causing the fitting center to drift and greatly increasing the fitting error, thus failing to truly reflect the standardization of the action. Therefore, this embodiment introduces a smoothing algorithm as a preprocessing step:

[0077] S31, perform time-series tracking on the position sequence to obtain a trajectory point sequence;

[0078] S32, A smoothing algorithm is used to reduce the detection jitter noise of the trajectory point sequence;

[0079] S33, determine the trajectory circularity error of the denoised trajectory point sequence, and / or determine the cumulative rotation angle of the denoised trajectory point sequence, and / or determine the number of round trips of the denoised trajectory point sequence.

[0080] In some alternative implementations, the Kalman filter algorithm can be used to make optimal estimates of the position and velocity of the trajectory points using its state-space model, effectively filtering out random jitter noise; in other implementations, a moving average filter or a Savitzky-Golay smoothing filter can also be used to achieve a similar denoising effect.

[0081] Further and optionally, in S33, determining the trajectory circularity error of the denoised trajectory point sequence includes:

[0082] S331, Fit the center and radius of the trajectory point sequence using the least squares criterion;

[0083] S332, using normalized circular error As the circularity error of the trajectory:

[0084]

[0085] in, ,

[0086] In the formula, RMSE represents the trajectory point. The root mean square error of the distance to the fitted circle (a,b), where r is the fitted radius and N is the number of trajectory points.

[0087] To eliminate the influence of different shooting distances and personnel body sizes on the error dimension, this embodiment creatively uses the normalized circular error Ecircle as the final output, instead of directly using the traditional absolute error RMSE.

[0088] It's important to understand that the principle behind this is as follows: when the camera is far from the worker, the trajectory radius *r* in the image is small. Even if the movement is perfectly executed, the absolute deviation *RMSE* may appear large due to pixel resolution limitations. Conversely, when shooting at close range, *r* is larger, and the same physical deviation corresponds to a smaller *RMSE* in the image. If a non-normalized absolute error *RMSE* is used as the scoring benchmark, different scoring results will be obtained for the same standard movement at different shooting distances. By normalizing by dividing by the fitting radius *r*, *Ecircle* essentially measures the proportion of relative deviation, thus eliminating scale factors and ensuring that the scoring standard remains consistent across most shooting distances.

[0089] Exemplarily, let the threshold endpoints of the normalized circular error Ecircle be: when Ecircle ≤ 0.1, the circularity of the trajectory is determined to be excellent; when 0.1 < Ecircle ≤ 0.3, it is determined to be qualified; when Ecircle > 0.3, it is determined to be unqualified.

[0090] In some alternative embodiments, the polar angle of each point relative to the center of the circle is defined:

[0091] , for phase unwrapping is performed to obtain , and the cumulative rotation angle is obtained:

[0092]

[0093] In some alternative embodiments, the round-trip number N is defined as the zero-crossing count or peak-valley count of the trajectory in the main direction.

[0094] S40. Determine the target hand signal instruction category satisfied by the arm posture angle parameter, the trajectory parameter, the type and visual attributes of the signal equipment, call the standard action parameter set corresponding to the hand signal instruction category, quantify and score the posture angle parameter, the trajectory parameter, and the type and visual attributes of the signal equipment based on the standard action parameter set, and output the scoring result.

[0095] In this embodiment, the hand signal instruction category represents the specific command instructions conveyed by the operator. Compared with the limitations of traditional single-modal recognition, in this embodiment, the features of three dimensions of posture, trajectory, and equipment are fused and determined in a multi-modal manner. In the traditional method, only relying on pose estimation, when the perspective changes drastically or the light is insufficient, the key point detection is prone to deviation; only relying on trajectory tracking, the line will break when the equipment is briefly blocked; and the color of the equipment is also difficult to identify under low light at night. Multi-modal fusion makes each feature complement and verify each other. Even if a certain modality is interfered, other modalities can still provide support, thus significantly improving the robustness and generalization ability of recognition, allowing a certain range of action variations and still being able to correctly identify the instruction category.

[0096] The standard action parameter set is a quantization benchmark extracted for a specific hand signal instruction category according to industry specifications, and includes various standard thresholds and tolerance ranges that the actions of this category should reach. By calling the standard parameter set corresponding to the category, the system converts the action specifications into quantifiable evaluation indicators and quantifies and scores the actually measured posture, trajectory, and equipment features.

[0097] In some alternative embodiments, the standard parameter set at least includes:

[0098] (a) The standard value and tolerance of the posture angle parameter;

[0099] (b) Standard range and tolerance of motion duration parameters;

[0100] (c) Standard thresholds and tolerances for trajectory shape parameters;

[0101] (d) Standard equipment attributes such as signal equipment type and color / luminous status;

[0102] (e) The default set of thresholds used for rating mapping.

[0103] In some alternative implementations, the establishment of the standard action parameter set specifically includes:

[0104] The system establishes a standard action parameter library P={Pc}, where c is the hand signal command category index, and each Pc includes at least the following fields:

[0105] (1) Standard posture angle: , and corresponding tolerance , ;

[0106] (2) Standard motion duration: and speed tolerance

[0107] (3) Standard trajectory parameters: circular error threshold Cumulative rotation angle threshold Or the threshold for the number of round trips ;

[0108] (4) Standard equipment attributes: ToolTypestd (signal flag / signal light / calibration hammer), Colorstd (red / yellow / green, etc.) or LightStatestd (on / off);

[0109] (5) Visibility threshold: Detection confidence threshold c0 and brightness threshold b0; this field can be configured for daytime / nighttime separately to form Pc_day and Pc_night.

[0110] For example, the standard action parameter set can be extracted from the standard text of hand signals for railway train operation and shunting, and statistically calibrated by combining video samples from training / assessment scenarios. Taking "stop signal," "deceleration signal," and "departure signal" as examples, during the day they can correspond to actions such as unfolding a red flag, unfolding a yellow flag, unfolding a green flag and making an upward arc circular rotation, respectively; at night they can correspond to red lights, yellow lights, green lights and corresponding trajectory actions.

[0111] Further and optionally, S40 specifically includes:

[0112] S41, based on the standard action parameter set, determine the completion score, speed score, posture angle accuracy score, equipment usage correctness score and visibility score of the target hand signal command category respectively;

[0113] S42, based on the weight vector, performs a weighted sum of each sub-score to obtain the total score value as the output score result.

[0114] It's important to note that the completion sub-score measures whether the movement is performed completely, such as whether a full circle is drawn or the required number of swings is achieved; the speed sub-score measures whether the movement rhythm conforms to the prescribed duration; the posture angle accuracy sub-score measures whether the limb posture is correct; the equipment usage correctness sub-score measures whether the signal carrier is used correctly; and the visibility sub-score measures whether the signal is clearly discernible under environmental interference. These five dimensions are non-overlapping and cover the core elements of movement execution, avoiding the one-sidedness of scoring based on a single dimension.

[0115] Further and optionally, S41 specifically includes:

[0116] S411, determine the completion score based on the ratio between the trajectory parameters and the reference trajectory parameters in the set of standard action parameters;

[0117] For example, the complete sub-score function F comp (·) Calculated based on at least one of the following: the ratio of the cumulative rotation angle Δφ to 360°, the number of round trips N to the standard number of round trips N std The ratio; and after limiting the ratio ρ to [0,1], the degree score scomp=100·ρ is calculated.

[0118] S412, obtain the action duration of the video sequence to be identified, and determine the speed sub-score based on the relationship between the action duration and the reference duration interval in the standard action parameter set;

[0119] For example, the velocity sub-score function F time (·) Based on the action duration T and the standard action duration range [T] min ,T max ]Calculate when T is located in [T min , T max [Time limit stime=100; when T is less than T] min or greater than T max At that time, let stime = max(0, 100·(1-|TT)). clip | / τT)), where T clip For T in [T min , T max The cutoff value on ], where τT is the velocity tolerance parameter.

[0120] S413, Determine the attitude angle accuracy sub-score based on the magnitude of the difference between the attitude angle parameter and the reference angle in the standard motion parameter set;

[0121] For example, the attitude angle accuracy sub-score function F pose (·) Based on the measured angle θ and the standard angle θ std The difference Δθ is calculated as follows: when Δθ≤τθ, let spose=100·(1-Δθ / τθ), and when Δθ>τθ, let spose=0, where τθ is the angle tolerance parameter.

[0122] S414. Based on the comparison results between the type and visual attributes of the signal equipment and the standard equipment type and standard visual attributes in the standard action parameter set, determine the equipment usage correctness score.

[0123] For example, the correctness sub-score function F tool (·) At least based on the consistency determination of the signal equipment type and standard equipment attributes, when consistent, let tool=100, when inconsistent, let tool=0; and optionally apply a penalty factor γ∈(0,1] to the total score, such that S←γ·S.

[0124] S415, Obtain visible frames in the video sequence to be identified, and determine the visibility sub-score based on the proportion of the visible frames in the video sequence to be identified, wherein the visible frames are image frames with a detection confidence level not less than a preset confidence threshold and a brightness level not less than a preset brightness threshold.

[0125] For example, within the action interval, if a frame satisfies the conditions of device detection confidence conf≥ preset confidence threshold c0 and brightness L≥ preset brightness threshold b0, it is recorded as a visible frame.

[0126] Furthermore, and optionally, the Analytic Hierarchy Process (AHP) is used as the implementation method for determining the weight vector. Specifically, this includes:

[0127] S521, Construct a comparison matrix of evaluation indicators ,in, This indicates the importance of indicator i relative to indicator j;

[0128] In some alternative implementations, the evaluation index comparison matrix A can be constructed by inviting multiple domain experts to conduct pairwise importance comparisons of the five indicators: completion, speed, attitude angle accuracy, correct use of equipment, and visibility.

[0129] S522, calculate the consistency ratio of the current weight vector w of each evaluation index based on the evaluation index comparison matrix A, and determine the current weight vector w as the weight vector of the evaluation index when the consistency ratio is not greater than a preset threshold.

[0130] S523, otherwise, increase / decrease the current weight vector w until the consistency ratio is not greater than the preset threshold.

[0131] In the technical solution provided in this embodiment, multimodal hand signal command category determination is performed by fusing posture angle parameters, trajectory parameters, and the type and visual attributes of signal equipment. Compared with a solution that relies solely on human posture estimation or pixel-level template comparison, the multimodal features complement each other. By calling the standard action parameter set corresponding to the identified hand signal command category, the action specifications are transformed into five quantitative evaluation indicators in terms of completion, speed, posture angle accuracy, correct equipment use, and visibility. The scoring results are then output to achieve reliable quantitative evaluation of hand signals.

[0132] As one implementation, this embodiment provides a railway hand signal processing system, which includes:

[0133] The acquisition module is used to acquire a video sequence to be identified containing hand signal actions, as well as the type, visual attributes, and position sequence of the signaling device performing the hand signal actions;

[0134] The posture analysis module is used to extract a sequence of feature point coordinates, including at least the shoulder, elbow, and wrist, based on the video sequence to be identified, and to calculate posture angle parameters based on the feature point coordinate sequence. The posture angle parameters include arm posture angle parameters and angular velocity parameters.

[0135] The trajectory tracking and fitting module is used to track and fit the position sequence to obtain trajectory parameters, which include trajectory circularity error, cumulative rotation angle and / or number of round trips;

[0136] The judgment and scoring module is used to determine the target hand signal command category satisfied by the posture angle parameter, the trajectory parameter, the type of signal equipment and the visual attributes, call the standard action parameter set corresponding to the hand signal command category, and perform quantitative scoring on the posture angle parameter, the trajectory parameter and the type of signal equipment and the visual attributes based on the standard action parameter set, and output the scoring result.

[0137] Based on the above architecture, this application embodiment further provides a method such as... Figure 2The diagram shows the architecture of a railway hand signal processing system. In this embodiment, the system includes two main parts: a camera acquisition unit 1 and a data processing unit 2. The camera acquisition unit 1 is located at the shunting operation site and is used to capture video of shunting personnel directing the signaling. This camera can be a high-definition digital camera and can be positioned at an appropriate location in the testing area to comprehensively capture the upper body movements of the shunting personnel and the movement of the signaling equipment.

[0138] The captured video stream is transmitted to the data processing unit 2 via wired or wireless network. The data processing unit 2 integrates a high-performance processor or industrial computer, as well as functional modules such as attitude analysis module 21, signal equipment detection module 22, trajectory tracking and fitting module 23, automatic scoring module 25, and report generation module 26.

[0139] First, the video frames are processed by the posture analysis module 21 to extract key points of the skeletal structure of the shunting personnel and determine their posture. Specifically, the posture analysis module 21 loads a pre-trained YOLO human keypoint detection model to identify joints in each frame of the image, outputting the coordinates of key points such as the head, left and right shoulders, elbows, wrists, and hips. Combining this keypoint position information, the module 21 calculates the limb posture angles and range of motion of the shunting personnel. For example, when the current action is waving a flag to signal the locomotive to move forward, the posture analysis module can detect that the shunting personnel's right arm starts from a vertically upward position and makes a circular waving motion, recording the change in the bending angle of the elbow joint during the movement; when the current action is issuing a stop signal, the module 21 will identify the posture of the shunting personnel raising one arm above their head with the arm straight, and calculate whether the angle between the arm and the torso is close to 90 degrees (vertical), etc. The output data of the posture analysis module 21 includes the keypoint coordinates of each time frame, the calculated key angle values, and the time series of the changes in the action posture. This data will be passed to the automatic scoring module 25 for evaluation of aspects such as posture accuracy and movement speed.

[0140] Next, the signal equipment detection module 22 processes the synchronized video frames to identify the type and status of the signal tools held by the shunting personnel. Based on a deep learning-based object detection model or color recognition algorithm, module 22 can detect the signal equipment (flags, signal lights, calibration hammers) area from the image and identify its features: in daytime scenes, it identifies the color of the signal flag (e.g., red or yellow) to determine whether it conforms to the flag color stipulated by the railway department, and whether the flag is unfolded or folded as required; in nighttime scenes, it detects the light source color (e.g., red or white light) and flashing status of the signal equipment, and determines whether the light is on to ensure effective signal transmission. In addition, the signal equipment detection module 22 summarizes and analyzes the detection results of consecutive frames: for nighttime scenes, it counts the proportion of time frames during the entire operation where signal lights are detected (i.e., signal lights are clearly visible in the video), thereby quantifying signal visibility; for daytime scenes, it focuses on whether the flag is always visible in the camera frame and is not obstructed. Through these detections, module 22 can output whether the signal tools are used correctly, whether the color / brightness of the signal tools meets the requirements, and the visibility indicators of the signal tools during the operation.

[0141] Then, the trajectory tracking and fitting module 23 analyzes the trajectory of the gesture. Starting from the position of the signal flag / light in each frame of the image provided by the signal equipment detection module 22, module 23 uses algorithms such as Kalman filtering or optical flow tracking to track the changes in that position over time, obtaining the motion trajectory of the signal equipment in space. To eliminate video jitter and detection noise, the trajectory point sequence can be processed by a smoothing filtering algorithm to obtain a smoother motion path. When it is detected that the shunting personnel are performing a signal that requires waving a flag / light in a circle, module 23 performs a circular fitting calculation on the obtained trajectory points. Specifically, it uses the least squares method to calculate the center coordinates and radius of the circle that fits the trajectory points, and evaluates the degree of fit between the trajectory and the fitted circle (circular similarity). At the same time, module 23 calculates the cumulative rotation angle of the motion based on the position change in each frame. When the cumulative angle is close to or exceeds 360°, the circular motion is considered complete. In addition, if the trajectory deviates from the circular track (low circular similarity) or the rotation angle is insufficient, such as only rotating half a circle, the motion completion is considered substandard. For other types of gesture signals, the trajectory tracking module can also identify the standardization of the action by judging the shape of the trajectory: for example, the "slow down" signal may require the flag to be waved up and down three times, so the number of times and the amplitude of the trajectory going back and forth in the vertical direction can be counted to see if the requirements are met. The analysis results of the trajectory tracking and fitting module 23 will be provided to the automatic scoring module 25 as a basis for scoring the degree of action completion and the standardization of the trajectory.

[0142] The action category determination module 24 integrates the posture angle parameters output by the posture analysis module 21, the trajectory parameters output by the trajectory tracking and fitting module 23, and the equipment type and attributes output by the signal equipment detection module 22 to output the hand signal command category c and its confidence level. The recognition result is used to select the corresponding standard action parameter set Pc and is provided to the automatic scoring module 25 and the report generation module 26 as the basis for scoring and report generation.

[0143] After obtaining the analysis results from the posture analysis module 21, signal equipment detection module 22, and trajectory tracking module 23, and the instruction category c and its confidence level from the action category determination module 24, the system enters the automatic scoring module 25 for comprehensive evaluation. The automatic scoring module 25 has pre-stored parameter thresholds and scoring rule libraries for standard actions of railway shunting operators' signals. Module 25 first calculates the action completion score based on the trajectory completion angle or number of times provided by the trajectory tracking module 23. For example, for the "forward" signal requiring a flag to circle once, if the actual trajectory completes more than 90% of the circle, the completion score is close to full; otherwise, the score is reduced proportionally. For the "decelerate" signal requiring three consecutive flag wavings, if only two wavings are actually performed, the completion score is reduced accordingly. Next, module 24 uses the output of the posture analysis module 21 to calculate the posture angle accuracy score. Taking the "stop" signal as an example, its standard posture is one arm raised straight up. Module 25 checks the angle between the shunting operator's arm and the vertical direction, and whether the elbow is fully extended: if the arm is tilted or bent, points are deducted according to the degree of deviation. Similarly, for signals that require horizontal swinging, the posture angle score will consider whether the height and angle of the raised arm are appropriate.

[0144] Module 25 evaluates the speed and rhythm of the action, calculating the duration of the entire gesture based on the video frame rate and comparing it with the standard action time. If a signal action requires completion within 5 seconds but actually takes longer, the speed score will be lowered; conversely, excessive haste may be considered as unstable rhythm and result in a deduction. For nighttime scenes, Module 25 focuses on calculating the visibility score of traffic lights, using the proportion of visible frames of traffic lights output by the signal equipment detection module 22: for example, if the lights are clearly visible and maintain the correct color for 80% of the entire action duration, the visibility score is high; otherwise, points are deducted for insufficient proportion. The correct use of signal equipment is also evaluated as an independent indicator in Module 25: if shunting personnel do not use the correct signal equipment as required (e.g., misusing traffic lights during the day or not turning on traffic lights at night), Module 25 can directly apply a deduction factor to the total score or lower the upper limit of the relevant indicator score to reflect the impact of such errors on the overall evaluation. The automatic scoring module 25 weights and sums the above scores according to predetermined weights to generate the final total score for the gesture action. For example, in this embodiment, for the "forward" signal at night, the scoring weights can be set as follows: action completion 35%, trajectory circularity (action smoothness) 35%, signal light visibility 15%, and action speed 15% (if the tool is used incorrectly, the total score will be multiplied by a certain coefficient and reduced); while for the "stop" signal during the day, the weights can be set as follows: action completion 40%, posture angle (such as arm extension) 30%, action speed 10%, and correct use of the signal flag 20%, etc. This weighted scoring for different signal actions and scenarios ensures that the evaluation results reflect actual importance and makes the total score 100 points, facilitating intuitive interpretation.

[0145] Finally, the report generation module 26 generates a detailed examination report file based on the scoring results and analysis data provided by the automatic scoring module 25. In this embodiment, the report generation module 26 organizes the scores and evaluation results of each indicator into a report format with both text and graphics. For example, as shown... Figure 2As shown in the flowchart, module 26 extracts the total score, sub-item scores, and determined action category of the shunting personnel's actions, and formats this textual information into the report template. Simultaneously, module 26 obtains a smoothed motion trajectory image from the trajectory tracking module and inserts it into the report to visually demonstrate the flag / light movement path of the shunting personnel. Furthermore, module 26 uses plotting libraries such as Matplotlib to generate a score distribution bar chart, displaying the proportion of each evaluation dimension's score to the corresponding full score, and adding a brief explanation below the chart (e.g., "Signal light visibility: 80%, meets requirements" or "Arm verticality deviation 10 degrees, somewhat lacking"). For indicators that do not meet the standards, the report will include textual suggestions, such as "Flag waving amplitude is insufficient; should reach above shoulder height" or "Signal lights are not visible for extended periods; ensure lights remain on continuously at night." The report generation module 26 integrates the above content and outputs a PDF file. This PDF report can be automatically saved to a specified database or provided to evaluators and test subjects for review. This report allows railway shunting personnel to clearly understand the strengths and weaknesses of their hand signals, and evaluators can also use it to assess the training effectiveness and provide targeted guidance.

[0146] Example scenario: Daytime shunting operation hand signal "stop signal". The system detects that the equipment type is a signal flag and the color is red. The signal category recognition module outputs category c = stop signal.

[0147] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0148] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the railway hand signal processing method based on multimodal perception fusion as described in the above embodiments.

[0149] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0150] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0151] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0156] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A railway hand signal processing method based on multimodal sensing fusion, characterized in that, The method includes the following steps: S10, acquire the video sequence to be identified containing hand signal actions, as well as the type, visual attributes, and position sequence of the signaling device performing the hand signal actions; S20, extract feature point coordinate sequences including at least shoulder, elbow and wrist based on the video sequence to be identified, and calculate arm posture angle parameters based on the feature point coordinate sequences; S30, perform trajectory tracking and fitting on the position sequence to obtain trajectory parameters, the trajectory parameters including trajectory circularity error, cumulative rotation angle and / or number of round trips; S40, determine the target hand signal command category satisfied by the arm posture angle parameter, the trajectory parameter, the type of signal equipment and visual attributes, call the standard action parameter set corresponding to the hand signal command category, quantify and score the posture angle parameter, trajectory parameter and the type of signal equipment and visual attributes based on the standard action parameter set, and output the score result.

2. The railway hand signal processing method based on multimodal sensing fusion as described in claim 1, characterized in that, S30 includes: S31, perform time-series tracking on the position sequence to obtain a trajectory point sequence; S32, A smoothing algorithm is used to reduce the detection jitter noise of the trajectory point sequence; S33, determine the trajectory circularity error of the denoised trajectory point sequence, and / or determine the cumulative rotation angle of the denoised trajectory point sequence, and / or determine the number of round trips of the denoised trajectory point sequence.

3. The railway hand signal processing method based on multimodal sensing fusion as described in claim 2, characterized in that, In step S33, determining the circular error of the trajectory point sequence after noise reduction includes: S331, Fit the center and radius of the trajectory point sequence using the least squares criterion; S332, using normalized circular error As the circularity error of the trajectory: ； in, , ; In the formula, RMSE represents the trajectory point. The root mean square error of the distance to the fitted circle (a,b), where r is the fitted radius and N is the number of trajectory points.

4. The railway hand signal processing method based on multimodal sensing fusion as described in claim 1, characterized in that, The arm posture angle parameters include the elbow joint angle and the arm lifting angle, wherein: The elbow joint angle The angle between two vectors determined by the three points of shoulder, elbow, and wrist is calculated as follows: ； In the formula, A represents the coordinates of the shoulder point; B represents the coordinates of the elbow point; and C represents the coordinates of the wrist point. The arm was raised at an angle The angle between the line vector connecting the shoulder point to the wrist point and the vertical direction of a single frame of the video sequence to be identified is calculated as follows: ； ； In the formula, V is the vertical upward unit vector (0, -1), and U is the arm direction vector.

5. The railway hand signal processing method based on multimodal sensing fusion as described in claim 1, characterized in that, The types of signal equipment include signal flags or signal lights, and the visual attributes of the signal equipment include color category or illumination status.

6. The railway hand signal processing method based on multimodal sensing fusion as described in claim 1, characterized in that, In step S40, the posture angle parameters, trajectory parameters, and the type and visual attributes of the signaling equipment are quantitatively scored based on the standard motion parameter set, and the scoring results are output, including: S41, based on the standard action parameter set, determine the completion score, speed score, posture angle accuracy score, equipment usage correctness score and visibility score of the target hand signal command category respectively; S42, based on the weight vector, performs a weighted sum of each sub-score to obtain the total score value as the output score result.

7. The railway hand signal processing method based on multimodal sensing fusion as described in claim 6, characterized in that, S41 specifically includes: The completion score is determined based on the ratio between the trajectory parameters and the reference trajectory parameters in the set of standard motion parameters. The motion duration of the video sequence to be identified is obtained, and the speed sub-score is determined based on the relationship between the motion duration and the reference duration interval in the standard motion parameter set. The attitude angle accuracy sub-score is determined based on the difference between the attitude angle parameter and the reference angle in the set of standard motion parameters. Based on the comparison results between the type and visual attributes of the signal equipment and the standard equipment type and standard visual attributes in the standard action parameter set, the correctness score of equipment use is determined. Visible frames in the video sequence to be identified are obtained, and a visibility sub-score is determined based on the proportion of the visible frames in the video sequence to be identified. The visible frames are image frames with a detection confidence level not less than a preset confidence threshold and a brightness level not less than a preset brightness threshold.

8. The railway hand signal processing method based on multimodal sensing fusion as described in claim 6, characterized in that, The steps for determining the weight vector include: Construct a comparison matrix of evaluation indicators ,in, This indicates the importance of indicator i relative to indicator j; Based on the evaluation index comparison matrix A, the consistency ratio of the current weight vector w of each evaluation index is calculated. When the consistency ratio is not greater than a preset threshold, the current weight vector w is determined to be the weight vector of the evaluation index. Otherwise, increase / decrease the current weight vector w until the consistency ratio is not greater than the preset threshold.

9. A railway hand signal processing system, characterized in that, The railway hand signal processing system includes: The acquisition module is used to acquire a video sequence to be identified containing hand signal actions, as well as the type, visual attributes, and position sequence of the signaling device performing the hand signal actions; The posture analysis module is used to extract a sequence of feature point coordinates, including at least the shoulder, elbow, and wrist, based on the video sequence to be identified, and to calculate posture angle parameters based on the feature point coordinate sequence. The posture angle parameters include arm posture angle parameters and angular velocity parameters. The trajectory tracking and fitting module is used to track and fit the position sequence to obtain trajectory parameters, which include trajectory circularity error, cumulative rotation angle and / or number of round trips; The judgment and scoring module is used to determine the target hand signal command category satisfied by the posture angle parameter, the trajectory parameter, the type of signal equipment and the visual attributes, call the standard action parameter set corresponding to the hand signal command category, and perform quantitative scoring on the posture angle parameter, the trajectory parameter and the type of signal equipment and the visual attributes based on the standard action parameter set, and output the scoring result.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the railway hand signal processing method based on multimodal perception fusion as described in any one of claims 1 to 8.

Images