Queue training method, system and device for following recognition

Abstract

The invention discloses a queue training method, system and device for following recognition, relates to the technical field of automatic recognition and evaluation of queue training, and realizes dynamic motion capture and recognition of training personnel, especially a single person, a row surface and a square matrix queue in the advancing process by utilizing multi-sensor data acquisition, deep learning and motion tracking technologies. By adopting the queue training method, system and device for following recognition, the actions of trainees are automatically tracked in real time in the training process, various action indexes are objectively quantified, the accuracy and integrity of training data acquisition are improved, manual intervention and subjective errors are reduced, and the training efficiency is improved through real-time feedback and comprehensive evaluation. And an instructor is assisted to adjust the training scheme in a targeted manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic identification and evaluation of queue training, in particular to a following identification queue training method, system and device. BACKGROUND

[0002] At present, the traditional queue training mainly relies on the observation by the eyes and the manual scoring by the instructors to evaluate the training actions. This method has the following disadvantages: 1. Limited by subjective factors, different instructors have different evaluation standards, resulting in inconsistent results; 2. For dynamic and continuous queue training, manual monitoring has blind spots and cannot cover all action details; 3. The traditional static monitoring is difficult to capture the subtle action changes of the trainees in the marching process in real time, especially in the action tracking and posture evaluation.

[0003] In the prior art, although there is a scheme of using fixed cameras combined with image processing and deep learning to detect and analyze actions, in the dynamic training scene, how to realize the following identification of the personnel in the single soldier or the formation, and to capture the whole process of the movement in real time, still has technical problems. Therefore, an automatic system and method capable of tracking, identifying and quantitatively evaluating the queue actions in real time during the training process are urgently needed. SUMMARY

[0004] The purpose of the present application is to provide a following identification queue training method, system and device, to solve the problems raised in the background art, to realize the dynamic action capture and identification of the training personnel, especially the single person, the formation and the square formation in the marching process, by using multi-camera data acquisition, deep learning and motion tracking technology.

[0005] To achieve the above purpose, the present application provides a following identification queue training method, comprising the following steps: S1, data acquisition is performed by using a data acquisition module, and the data is preprocessed by an image processing and deep learning analysis module; S2, posture and action identification is performed on the preprocessed data, including key point detection and skeleton construction, posture abnormality judgment and step length and step speed calculation; S3, formation alignment judgment is performed on the identified data, including instance segmentation detection hat, row and column grouping and reference line fitting and alignment judgment; S4, following identification is performed, including real-time target detection, multi-target tracking and state updating and PID control module; S5, evaluation results are formed by a data processing and evaluation module and real-time feedback is provided, including action index quantification and real-time feedback.

[0006] Preferably, the data acquisition module in S1 deploys sensors on the front and side of the training ground, which can be a single camera, multiple cameras, or a combination of cameras and lidar, to obtain RGB or RGB-D data. Each sensor acquires video frames in real time and performs uniform image preprocessing, including pixel normalization and size adjustment.

[0007] Preferably, the key point detection and skeleton construction in S2 use a deep convolutional neural network architecture, with the following specific steps: S211, multi-branch feature extraction and heat map generation: Multi-resolution feature extraction: The network uses a parallel branch structure to gradually extract global semantic information while maintaining the original resolution, and repeatedly exchanges information between different resolutions. Let the output feature maps of each branch be where branch maintains the highest resolution; Heat map generation: Through feature fusion and subsequent convolution layers, the network finally outputs a set of heat maps Each heat map represents the confidence of the th key point in the human body at each position on the image, with the specific formula being: ; where represents the trained network, is the network parameter, is the input image after normalization and size adjustment, x is the horizontal pixel coordinate on the image plane, y is the vertical pixel coordinate on the image plane; S212, select key point positioning: Non-maximum suppression is performed on each heat map to select the local maximum position as the key point coordinates: where ; where is the position coordinate of the k th key point, is the horizontal pixel coordinate of the key point in the image, is the vertical pixel coordinate of the key point in the image; When the sensor combination provides depth information (such as multiple cameras or a combination of cameras and lidar), the depth value can be introduced based on the two-dimensional pixel coordinates to further extend the key point coordinates to three dimensions: ; ; ; Thus, the three-dimensional coordinates of the key points are obtained wherein is the focal length of the camera, is the principal point coordinate.

[0008] When the sensor combination has no depth information, the key points are in the form of two-dimensional coordinates and participate in subsequent analysis; when depth information is available, more rich three-dimensional motion analysis can be further carried out.

[0009] S213, skeleton construction: According to the pre-defined skeleton connection relationship, the detected key points are connected to construct a human skeleton: ; wherein, is the index set of the skeleton connection, is the two-dimensional pixel coordinate of the i-th key point, i is the two-dimensional pixel coordinate of the j-th key point. j

[0010] Preferably, the specific steps of the posture anomaly judgment in S2 are as follows: S221, shoulder unevenness detection: Key point acquisition: let the left shoulder coordinate be and the right shoulder coordinate ; Calculate the shoulder height difference, calculate the difference between the left shoulder and the right shoulder in the vertical direction: ; Judgment standard: if , it is determined that there is shoulder unevenness, wherein is a preset threshold value; S222, head tilt detection: Key point acquisition: let the nose coordinate be ; Calculate the midpoint of the left and right shoulders: ; Calculate the offset of the nose and the shoulder midpoint in the horizontal x-axis direction: ; Judgment standard: if , it is considered that there is head tilt, wherein is a preset threshold value; ​​S223, left-right tilt detection of upper body: Key point acquisition: let the left hip coordinate be and the right hip coordinate be ; Calculate the midpoint of the hips: ; Calculate the midpoint of the shoulders and the midpoint of the hips , and calculate the angle between the line connecting the two midpoints and the horizontal line to obtain the tilt angle of the upper body: ; Judgment standard: if , it is determined that the left-right tilt of the upper body is abnormal, wherein is a preset angle threshold.

[0011] Preferably, the specific steps of step length and step speed calculation in S2 are as follows: S231, draw the arm swing waveform graph: Key point trajectory acquisition: record the vertical coordinates or relative displacements of the arm-related key points in each frame of image with respect to the shoulder to form a time sequence ; Waveform calculation: Let the wrist point be , the shoulder point be , and the relative displacement of the arm be defined as ; Smooth : ; Wherein, N is the size of the sliding window (frame number), which is used to control the degree of smoothing, t is the time or frame number of the current frame, i is the frame offset index in the window, and the value is ; Waveform drawing: take time as the horizontal axis, as the vertical axis, draw the arm swing waveform graph, perform peak value detection, and obtain the arm swing information; S232, draw the step waveform graph: Foot key point sampling: including selecting the vertical coordinates and relative ground information of the foot key points in each frame as a time sequence ; Step cycle detection: find the time point of each "landing" in , and the step frequency (step / minute) is obtained from the interval between adjacent landing times Obtain: ; Waveform drawing: draw the time as the horizontal axis, and the foot swing process as a step waveform diagram; S233, step and step speed calculation: Camera calibration and coordinate transformation: calibrate the camera to obtain the intrinsic matrix and extrinsic parameters, and transform and correct the pixel coordinates to obtain the world coordinates : ; Step calculation: at the time of consecutive landing , including the horizontal distance of the left and right foot key points and the horizontal displacement of the same foot in the consecutive two landings, the horizontal distance of the left and right foot key points is approximated as: ; The average step is: ; Compare the average step with the standard step to determine whether it meets the requirements; Step speed calculation: including step and step frequency combination calculation and direct calculation according to adjacent frame foot displacement and time difference : If represents the step frequency, then the step speed is: ; Directly calculate the step speed according to the adjacent frame foot displacement and time difference : ; Compare the calculated step speed with the standard step speed to determine whether the walking speed meets the specifications.

[0012] Preferably, the specific steps of S3 are as follows: S31, instance segmentation detects the hat: Use instance segmentation algorithm to process the image collected by the top camera, and real-time detect the hat area of each team member. For each detected hat, the algorithm outputs: bounding box ; segmentation mask ; class confidence; Calculate the centroid of the segmentation result of each hat, the formula is: ; ; Get the set of center points of all hats ; S32, row-column grouping and reference line fitting: Row-column grouping: According to the longitudinal coordinate of the hat center point Sort the players, and then according to the distance threshold Divide the hats into several rows according to the horizontal coordinate Divide into several columns; Reference line fitting: For the set of hat center points in each row Fit a straight line using the least squares method: ; Wherein, the fitting parameter calculation formula is: ; ; Calculate the deviation of the fitted straight line from the reference line: ideal slope Calculate the tilt angle of the fitted straight line: ; At the same time, calculate the perpendicular distance from each hat center point to the fitted straight line: ; S33, alignment judgment: Row alignment: Calculate the average value of the distance from the hats in each row to the fitted straight line Or variance : ; Wherein, n The number of hats in the row; If Or Exceed the preset threshold, it is determined that the row is not aligned, and the system issues an alarm; Longitudinal alignment: Fit a longitudinal reference line for the hat center points in each column, and calculate the deviation to determine whether the queue is longitudinally aligned.

[0013] Preferably, the specific steps of S4 are as follows: S41, real-time target detection module, including target candidate region extraction and feature vector extraction, the specific steps are as follows: Target candidate region extraction: use a deep convolutional neural network to process the whole frame image, divide the image into several grids, and predict whether there is a moving target on each grid, output the boundary box of the candidate target And the corresponding confidence score, non-maximum suppression is performed on the candidate boxes to ensure that the output detection results are less redundant. Feature vector extraction: For each candidate region, a fixed-dimension appearance feature vector is extracted through the intermediate layers of the network . S42, multi-target tracking and state updating module, including state modeling and prediction, data association and state updating, the specific steps are as follows: State modeling and prediction: for each tracking target, the state vector is defined as: . Wherein represents the horizontal and vertical coordinates of the center of the target bounding box, represents the target scale, represents the aspect ratio, is the corresponding velocity component; The linear motion model is used for state prediction, and the Kalman filtering algorithm is used: . Wherein, k is the current time step or frame number, is the state vector predicted in the first k step, is the state transition matrix, and the state covariance update formula is: . Wherein, is the process noise covariance matrix, is the state covariance matrix updated in the first k- 1 step; Data association: Motion information matching: calculate the difference between each detection result and the predicted state, set the observation vector , and define the observation matrix , then use the Kalman filter residual: . Wherein, is the horizontal pixel coordinate of the extracted bounding box center, is the vertical pixel coordinate of the extracted bounding box center; The measurement covariance is: . Wherein, represents the measurement noise covariance, and the Mahalanobis distance is calculated as: . Get the motion similarity; Appearance information matching: For the i-th detected target region, the extracted feature vector The appearance feature of the j-th target in the history record Comparison is made using the cosine distance: ; The comprehensive cost is constructed and matched: The comprehensive cost function is constructed: ; Wherein, The optimal matching method is used to complete the matching between the detection result and the predicted state, and the corresponding relationship of the tracked target is updated, The motion similarity measure between the i-th detection result and the j-th predicted trajectory, usually taking the Mahalanobis distance; i State update: j For the successfully matched target, the Kalman filter update formula is used: ; ; Wherein, The updated state vector, The observation vector; The Kalman gain The calculation formula is: ; Wherein, The state covariance matrix obtained by the Kalman filter in the prediction stage at time step k; For the unmatched target, short-term tracking is maintained according to its predicted state, and if it is not matched for a long time, the tracking is terminated, and for the newly appeared detection result, the state and appearance feature of the new tracking target are initialized; S43, PID control module: Error definition: In the image coordinate system, the image center is set as For the selected tracking target, take its current detected center coordinate The horizontal direction error is defined as: ; PID controller formula: According to the error The control signal Is calculated: ; Wherein, , , The proportional, integral and differential coefficients, ​Real-time error in horizontal direction; Control closed loop: After processing each frame of image, the system recalculates the target center position and error , and updates the PID controller output.

[0014] Preferably, the specific steps of S5 are as follows: S51, action index quantization: According to the preset standard model, the collected data are compared, and each action error and score are outputted to form a quantitative evaluation result; S52, real-time feedback: By using a human-machine interface (HMI), data, images and action waveform information are displayed in real time to the instructor and the trainee, and online labeling and alarm are given to abnormal actions.

[0015] A queue training device of a following-identified queue training method, comprising a front-end multi-camera module, a high-performance data processing server and a data storage and management module, the front-end multi-camera module is used for fixing a camera and a following camera installed on a slide rail, the high-performance data processing server is used for supporting real-time inference of a deep learning model, and the data storage and management module is used for supporting long-term training data recording and query.

[0016] A queue training system of a following-identified queue training method, which is composed of a data acquisition layer, a data processing layer, an analysis and evaluation layer and a user interaction layer, and comprises a single-person training system, a row training system and a square training system.

[0017] Therefore, the following beneficial effects are achieved by adopting the above-mentioned following-identified queue training method, system and device: 1. High-precision dynamic tracking. Through the following identification of the slide rail camera, the dynamic actions of the trainee throughout the training process are captured in real time, which can more comprehensively and finely record the training process compared with the traditional fixed-angle camera.

[0018] 2. Objective and accurate quantitative evaluation. By using deep learning, the automatic quantitative evaluation of each index of the queue action is realized, and the subjective error in manual evaluation is greatly reduced.

[0019] 3. Real-time feedback and guidance. Through intuitive data, images and analysis reports, the action deviation in the training process is reminded in real time, which helps the instructor to adjust the training scheme in time and improves the training quality.

[0020] 4. Data storage and intelligent analysis. The long-term accumulated training data can be used for subsequent data mining and training method improvement, and promote the scientific and standardized process of queue training.

[0021] The technical solutions of the present application will be further described below with reference to the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Method flow chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 2 Single-person training hardware topology chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 3 Single-person front posture detection and analysis example chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 4 Single-person side posture detection and analysis example chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 5 Row training hardware topology chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 6 Row posture detection chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 7 Square training hardware topology chart of the embodiment of the present application for a following-identified queue training method, system and device; Figure 8 Square consistency data analysis chart of the embodiment of the present application for a following-identified queue training method, system and device. DETAILED DESCRIPTION

[0023] The technical solutions of the present application will be further described below with reference to the drawings and examples.

[0024] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the usual meaning understood by a person with ordinary skill in the art to which the present application belongs. The terms "first", "second" and similar words used in the present application do not represent any order, number or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar words mean that the elements or objects before the words cover the elements or objects listed after the words and their equivalents, without excluding other elements or objects. The terms "connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right" and the like are only used to represent relative positional relationships, which may change accordingly when the absolute positions of the described objects change.

[0025] Please refer to Figures 1-8The application provides a following-identified queue training method, comprising the following steps: S1, data acquisition is performed by using a data acquisition module, and the data is preprocessed by an image processing and deep learning analysis module.

[0026] The data acquisition module utilizes a combination of fixed and mobile (slide rail) cameras: two variable-focus fixed cameras are arranged at key positions on the front of the training field, and are used to acquire data such as postures and expressions of training personnel at different distances; a mobile camera installed on a slide rail follows the personnel in real time, captures dynamic actions of the personnel at close range, and realizes following identification. The front camera and the side slide rail camera acquire RGB video frames in real time . Each frame of image is preprocessed (such as normalization and size adjustment), to provide input for subsequent target detection.

[0027] S2, posture and action identification is performed on the preprocessed data, including key point detection and skeleton construction, posture anomaly judgment, and step width and speed calculation.

[0028] S21, key point detection and skeleton construction: In this stage, a deep convolutional neural network architecture is adopted, and the design concept is to maintain high-resolution features and perform multi-scale feature fusion, so as to obtain more fine key point positions. The whole process comprises the following steps: S211, multi-branch feature extraction and heat map generation: First step, multi-resolution feature extraction The network adopts a parallel branch structure, gradually extracts global semantic information while maintaining the original resolution, and repeatedly exchanges information between different resolutions. Let the output feature maps of each branch be , wherein the branch maintains the highest resolution.

[0029] Second step, heat map generation Through feature fusion and subsequent convolution layers, the network finally outputs a group of heat maps , each heat map represents the confidence of the key point in the human body at each position on the image. The formula is expressed as: ; , wherein represents the trained network, is a network parameter, is an input image after normalization and size adjustment, is the horizontal pixel coordinate on the image plane, y is the vertical pixel coordinate on the image plane.

[0030] S212. Select key points for location: For each heatmap Non-maximum suppression is performed, and the locations of local maxima are selected as keypoint coordinates: Wherein ; in, For the first k The location coordinates of the key points Key point The horizontal pixel coordinates in the image, Key point Vertical pixel coordinates in the image; When a sensor combination provides depth information (such as multiple cameras or a camera combined with LiDAR), depth values ​​can be introduced on top of two-dimensional pixel coordinates. Furthermore, the key point coordinates are expanded into a three-dimensional form: ; ; ; Thus, the three-dimensional coordinates of the key points are obtained. ,in For the camera's focal length, The coordinates of the main point.

[0031] When the sensor array lacks depth information, key points participate in subsequent analysis in the form of two-dimensional coordinates; when depth information is available, richer three-dimensional motion analysis can be performed.

[0032] S213, Skeleton Construction: Based on predefined skeletal connections (e.g., left shoulder connected to right shoulder, shoulder connected to hip, etc.), the detected key points are... Connect them to form the human skeleton: ; in A set of indices connected to the skeleton. No. i Two-dimensional pixel coordinates of key points For the first j Two-dimensional pixel coordinates of key points.

[0033] S22. Posture Anomaly Detection As attached Figure 2 As shown, key indicators of human posture are calculated using the detected key point coordinates to determine whether there are any abnormalities.

[0034] S221, shoulder height difference detection First step, key point acquisition Let the left shoulder coordinate be and the right shoulder coordinate be .

[0035] Second step, calculate shoulder height difference Calculate the difference between the two in the vertical direction: ; Third step, judgment standard If ( is a preset threshold value), it is determined that there is a shoulder height difference.

[0036] S222, head tilt detection: First step, key point acquisition Let the nose coordinate be .

[0037] Second step, calculate shoulder midpoint Calculate the midpoint of the left and right shoulders: ; Third step, calculate horizontal offset Calculate the offset between the nose and the shoulder midpoint in the horizontal (x-axis) direction: ; Fourth step, judgment standard If ( is a preset threshold value), it is considered that there is a head tilt.

[0038] S223, upper body left-right tilt detection: First step, key point acquisition Let the left hip coordinate be and the right hip coordinate be .

[0039] Second step, calculate hip midpoint ; Third step, calculate upper body tilt angle Calculate the angle between the line connecting the shoulder midpoint and the hip midpoint and the horizontal line: ; Fourth step, judgment standard If ( is a preset angle threshold value), it is determined that the upper body left-right tilt is abnormal.

[0040] S23, stride and pace calculation: S231, draw arm swing waveform chart: First step, key point trajectory acquisition Record the ordinate of the arm-related key points (such as the elbow and wrist) in each frame of image or the relative displacement from the shoulder to form a time series .

[0041] Second step, waveform calculation If the wrist point is , and the shoulder point is , the relative displacement of the arm can be defined as ; To reduce jitter, the can be smoothed (such as sliding mean filtering): ; Third step, waveform drawing As shown in the accompanying Figure 3 , take time as the horizontal axis and as the vertical axis to draw the arm swing waveform chart. Further peak detection can be performed to obtain information such as swing period and amplitude for training and evaluation.

[0042] S232, draw stride waveform chart: First step, foot key point sampling Select the ankle (or toe) key point The ordinate or relative ground height information of each frame as a time series .

[0043] Second step, step cycle detection Through peak detection or threshold determination method, find the time point of each "landing" in . Step frequency (step / minute) can be derived from the adjacent landing time interval : ; Third step, waveform drawing As shown in the accompanying Figure 3 , take time as the horizontal axis and draw into a stride waveform chart to visually display the up and down swing process of the foot S233, stride and pace calculation: First step, camera calibration and coordinate conversion If the real physical distance (such as stride in cm) needs to be obtained, the camera needs to be calibrated to obtain the intrinsic matrix​ and extrinsic parameters. After perspective transformation or depth correction, the world coordinates of the pixel coordinates can be obtained: ; Second step, stride calculation In consecutive frames or consecutive landing moments , the horizontal distance between the left and right foot key points can be used as an approximation of the stride: ; or the horizontal displacement of the same foot in two consecutive landings.

[0044] Average stride: ; Compare with the standard stride to determine whether it meets the requirements.

[0045] Third step, step speed calculation Step speed can be obtained by combining stride and step frequency. If represents the step frequency (steps / minute), then the step speed (unit m / min or cm / min) is: ; Or directly calculate the speed according to the displacement of the foot between adjacent frames and the time difference : ; Compare with the standard step speed to determine whether the walking speed meets the specifications.

[0046] S3, judge the alignment of the identified data, as shown in the attached Figure 7 , the top camera detects the hat of the team member in real time to judge whether the alignment and queue are neat. The overall process of the scheme is: instance segmentation detects hat → extract hat center coordinates → compare / fit with reference line → calculate deviation and judge neatness.

[0047] S31, instance segmentation detects hat: Use instance segmentation algorithm to process the image collected by the top camera, and detect the hat area of each team member in real time. For each detected hat, the algorithm outputs: bounding box ; segmentation mask ; class confidence.

[0048] Calculate the centroid of the segmentation result of each hat, the formula is: ; ; Get the center point set of all hats .

[0049] S32, Grouping and reference line fitting First step, grouping According to the longitudinal coordinate of the hat center point Sort the team members, and then according to the distance threshold Divide the hats into several rows. Similarly, according to the horizontal coordinate It can be divided into several columns.

[0050] Second step, reference line fitting For each set of hat center points in a row Fit a straight line using least squares method: ; Where the fitting parameter calculation formula is: ; ; Third step, calculate the deviation of the fitted straight line and the reference line Ideally, the row should be a horizontal straight line, i.e. the ideal slope Calculate the inclination angle of the fitted straight line: ; At the same time, calculate the perpendicular distance from each hat center point to the fitted straight line: ; S33, Regularity judgment: First step, row regularity Calculate the average value of the distance from each hat in a row to the fitted straight line Or variance : ; Where, n is the number of hats in the row; If Or Exceed the preset threshold, it is determined that the row is not regular, and the system issues an alarm.

[0051] Second step, longitudinal regularity Similar processing is performed on the hat center points of each column, a longitudinal reference straight line is fitted, and the deviation is calculated to determine whether the queue is longitudinally regular.

[0052] S4, Implement the following steps to realize the following functions: real-time target detection, multi-target tracking and state updating, and PID control module. S41, Real-time target detection module, including target candidate region extraction and feature vector extraction, the specific steps are as follows: First step, target candidate region extraction Process the whole frame image with a deep convolutional neural network, which divides the image into several grids and predicts whether there is a moving target on each grid. Output the bounding box of the candidate target and the corresponding confidence score. Non-maximum suppression is performed on the candidate box to ensure that the output is less redundant detection results.

[0053] Second step, feature vector extraction For each candidate region, extract a fixed dimension appearance feature vector through the intermediate layer of the network . These features will be used for subsequent appearance matching of targets in multi-target tracking S42, multi-target tracking and state update module, including state modeling and prediction, data association and state update, the specific steps are as follows: In order to realize the continuous tracking of each player, the system adopts a multi-target tracking method based on motion prediction and appearance information, the core of which includes state modeling, data association and state update.

[0054] First step, state modeling and prediction For each tracking target, the state vector is defined as ; Where , represents the horizontal and vertical coordinates of the center of the target bounding box, represents the target scale (such as the area of the bounding box), represents the aspect ratio, , , is the corresponding velocity component.

[0055] Linear motion model is used for state prediction. Kalman filter algorithm is used: ; Where, k is the current time step or frame number, is the state vector predicted in the k th step, is the state transition matrix, and the state covariance update formula is: ; Where, is the process noise covariance matrix, is the state covariance matrix updated in the k- th step; Second step, data association, including motion information matching, appearance information matching, and constructing a comprehensive cost function First, motion information matching For each detection result, compute its difference with the predicted state. Let the observation vector and define the observation matrix (typically, the position component is chosen), then the residual of Kalman filter is used: ; where is the extracted horizontal coordinate of the bounding box center pixel, is the extracted vertical coordinate of the bounding box center pixel; The measurement covariance is: ; where denotes the measurement noise covariance. The Mahalanobis distance is computed by: ; and the motion similarity is obtained.

[0056] Second, appearance information matching For the i-th detected target region, its extracted feature vector is compared with the appearance feature of the j-th history record of the same target The cosine distance is usually used: ; The overall cost is constructed and matched: Third, constructing the overall cost function ; where is used to balance the influence of motion and appearance information. The optimal matching method (e.g., Hungarian algorithm) is used to complete the matching between the detection results and the predicted states, and update the corresponding relationship of the tracked targets, is the motion similarity measure between the i-th detection result and the j-th predicted trajectory, and the Mahalanobis distance is usually taken.

[0057] Third step, state update For the successfully matched targets, the Kalman filter update formula is used: ; where is the updated state vector, is the observation vector; where the Kalman gain is calculated by: ; where is the state covariance matrix obtained by the Kalman filter in the prediction stage at time step k; For unmatched targets, short-term tracking is maintained according to their predicted states, and if they are not matched for a long time, the tracking is terminated. For newly detected results, new tracking target states and appearance features are initialized.

[0058] S43, PID control module: In order to ensure that the side slide rail camera always keeps the target in the image center, a PID controller based on error feedback is used.

[0059] First step, error definition In the image coordinate system, the image center is set as For the selected tracking target, take its currently detected center coordinates , and define the horizontal direction error as: ; (The vertical direction can also be processed similarly, but this scheme is mainly for horizontal adjustment.) Second step, PID controller formula According to the error , calculate the control signal : ; Where, , , are the proportional, integral, and differential coefficients, is the real-time error in the horizontal direction. The control signal is sent to the slide rail motor to adjust the camera's horizontal position or speed, so that the target remains relatively centered in the image.

[0060] Third step, control closed loop After processing each frame of image, the system recalculates the target center position and error , and updates the PID controller output. Through continuous feedback and adjustment, the camera continuously tracks the target, ensuring that the picture remains stable and the target is always in the preset position during the team member's march.

[0061] S5, form evaluation results through data processing and evaluation module and real-time feedback. Including action index quantification and real-time feedback, the specific steps are as follows: S51, action index quantification: According to the preset standard model, compare the collected data, output each action error and score, and form a quantitative evaluation result.

[0062] S52, real-time feedback: Use human-machine interface (HMI) to display data, images, action waveforms, etc. to the instructor and trainees in real time, and give online labeling and alarm for abnormal actions.

[0063] A queue training device following identification, comprising a front-end multi-camera module, a high-performance data processing server and a data storage and management module, the front-end multi-camera module is used for fixing cameras and following cameras installed on a slide rail, the high-performance data processing server is used for supporting real-time inference of a deep learning model, and the data storage and management module is used for supporting long-term training data recording and query.

[0064] A queue training system following identification, which is composed of a data acquisition layer, a data processing layer, an analysis and evaluation layer and a user interaction layer, and comprises a single-person training system, a row training system and a square training system.

[0065] The application is further described below in three training scene embodiments: Embodiment 1: single-person training scene (1) Hardware configuration: As shown in the accompanying Figure 1 , it can be used for 10-meter training, 4 sets of marching data acquisition modules, 1 set of full-scene multi-source acquisition module, 1 set of front queue capacity and shape emotion acquisition module, communication module, 8-hour power supply module, 1 computing server, 4 end computing units and 4 mobile positioning modules.

[0066] (2) Training process: a) Front and side posture detection and analysis As shown in the accompanying Figure 2 , the front camera posture detection uses a posture estimation algorithm based on deep learning to accurately estimate the key point positions of the human body (such as shoulders, head, spine, etc.) in 2D space. By calculating the height difference between the shoulder key points, it can be determined whether there is a problem of uneven shoulder height; the front camera can also assist in detecting head tilt and upper body left-right tilt, and calculate the head tilt angle by the relationship between the head and spine positions. At the same time, the system can also detect the shaking amplitude of the waist, chest and neck regions, analyze the relative motion (such as position change, angle deviation, etc.) of these key parts, and determine whether there is instability. The key points of the team member's arms are detected, and the angle between the elbow and hand is analyzed. If the elbow is bent more than a set angle, elbow bending and hooking hands occur.

[0067] b) Side posture detection and analysis The two side cameras are placed in an up-down position and are equipped with a slide rail, as shown in the accompanying Figure 3As shown, using a deep learning-based pose estimation algorithm, the key point positions of the human body are accurately estimated in 2D space, and the pace of the team members is tracked by the sliding rail camera. The step length and speed of each step are calculated and compared with the standard step length (e.g. 75 cm per step) and standard speed (112 steps per minute) to determine whether it meets the standard. According to the step waveform diagram, it can be judged whether the pace rhythm of each step is consistent.

[0068] The side-down camera clearly captures the foot movement of the team member from the angle below the foot, helping to detect the situation of not landing with the whole foot and splayed feet when walking. According to the detected wrist key points, the vertical distance between the wrist and the ground is calculated and compared with the standard height to determine whether the leg kick height off the ground is qualified when walking.

[0069] Example 2: Row training scene (1) Hardware configuration: As shown in the accompanying Figure 4 , it can be used for 50-meter training, 6 sets of marching data acquisition modules, 1 set of key area laser radar acquisition module, 1 set of front team capacity team shape emotion acquisition module, communication module, 8-hour power supply module, 1 computing server, 6 end computing units, 6 mobile positioning modules.

[0070] (2) Training process: As shown in the accompanying Figure 5 , in the row system, leg line consistency can be detected by determining whether the toes of the trainees are on a straight line. Hat line and gun line consistency detection is the same. As shown in the accompanying Figure 5 , the figure below, by normalizing the row of multiple people and superimposing to determine whether the row action is neat. According to the multiple arm key point detection results, a waveform diagram is drawn to identify arm action consistency. Use a deep learning-based target detection model to identify each team member's hat and gun, and extract the top position of the hat, the position of the gun barrel, and determine whether they are on the same straight line to detect position consistency.

[0071] Example 3: Square training scene (1) Hardware configuration: As shown in the accompanying Figure 6 , it can be used for 550-meter training, 5 sets of high-pole data acquisition modules, 2 sets of key area laser radar acquisition modules, 2 sets of front team capacity team shape emotion acquisition modules, communication module, mobile area 8-hour power supply module, two computing servers, 10 end computing units, 20 mobile positioning modules, and 1 set of central routing switching unit.

[0072] (2) Training process: As shown in the accompanying Figure 7For the analysis of square consistency, the system uses an instance segmentation method based on deep learning to extract the pixel coordinates of the trainee's cap, as shown in the red detection box and blue segmentation result in the figure. Then, according to the detection result, each row is connected and aligned with the horizontal reference line. If it is not parallel to the yellow horizontal line, the system will give an alarm that the row is inconsistent. Similarly, for the vertical queue, the system will be aligned with the green vertical reference line. If there is a deviation during the queue, the system will record and evaluate it.

[0073] Some hardware device indicators can be set as follows: 1. High pole data acquisition module: Pole height >= 10 meters, telescopic pole can be used.

[0074] Camera A: resolution >= 4K, frame rate >= 60fps, wide-angle lens angle >= 140o.

[0075] Automatic lightweight gimbal: weight meets high pole bearing range.

[0076] 2. Laser radar: FOV: not less than 120o horizontally and not less than 20o vertically; Horizontal scanning line not less than 126 lines, with ROI function; Detection distance not less than 200 meters @ 10%, ROI area angular resolution accuracy not greater than 0.1 degree x 0.1 degree, point frequency not less than 1,500,000 points / second.

[0077] 3. Front data acquisition module: Camera B: resolution >= 4K, frame rate >= 60fps, lens focal length >= 100mm.

[0078] Camera C: resolution >= 4K, frame rate >= 60fps, variable lens focal length range not weaker than 5mm-100mm.

[0079] 4. Lateral moving track robot: Positioning accuracy: not weaker than 1cm.

[0080] Travel speed: not less than 1m / s.

[0081] Sustainable working time: not less than 10 hours.

[0082] Working temperature condition: -20oC~80oC.

[0083] Camera D: resolution >= 4K, frame rate >= 60fps, variable lens focal length range not weaker than 2.8mm-20mm.

[0084] 5. Computing server: CPU: dual CPU, not less than 2*32 cores.

[0085] Memory: not less than 64G.

[0086] GPU: half-precision floating-point operation (FP16) computing power not less than 256 Tera-FLOPS.

[0087] SSD hard disk: not less than 4T.

[0088] 6, hard disk video recorder: Hard disk capacity: not less than 10T, not less than 144 hours of storage.

[0089] 7, mobile computing module: CPU: not less than 8 cores, 2.4GHZ.

[0090] NPU: INT8 computing power not less than 6TOPS.

[0091] Memory: not less than 16G.

[0092] Therefore, the application adopts the above-mentioned queue training method, system and device for following identification, realizes high-precision dynamic tracking, objective and accurate quantitative evaluation, real-time feedback and guidance, data storage and intelligent analysis.

[0093] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand: it can still modify or replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A method of training a queue following recognition, characterized by, It comprises the following steps: S1, data acquisition is performed by using a data acquisition module, and the data is preprocessed by an image processing and deep learning analysis module; S2, posture and action recognition is performed on the preprocessed data, including key point detection and skeleton construction, posture anomaly judgment, and step length and step speed calculation; S3, alignment judgment is performed on the recognized data, including instance segmentation detection of hats, row and column grouping, reference line fitting, and alignment judgment; S4, following recognition is performed, including real-time target detection, multi-target tracking and state updating, and a PID control module; S5, an evaluation result is formed by a data processing and evaluation module and real-time feedback is provided, including action index quantification and real-time feedback.

2. The method of claim 1, wherein: The data collection module in the S1 respectively deploys sensors on the front and side of the training field, and each sensor collects video frames in real time and carries out unified image preprocessing, which includes pixel normalization and size adjustment.

3. The method of claim 2, wherein, The key point detection and skeleton construction in S2 adopt a deep convolutional neural network architecture, and the specific steps are as follows: S211, multi-branch feature extraction and heat map generation: Multi-resolution feature extraction: The network adopts a parallel branch structure, gradually extracts global semantic information while maintaining the original resolution, and repeatedly exchanges information between different resolutions. Let the feature maps output by each branch be where branch maintains the highest resolution; Heatmap generation: Through feature fusion and subsequent convolutional layers, the network finally outputs a set of heatmaps Each heatmap represents the confidence of the i-th key point in the human body at each position on the image, and the specific formula is: ​ ； wherein, represents a trained network, is a network parameter, is a normalized and resized input image, is a horizontal pixel coordinate on the image plane, y is a vertical pixel coordinate on the image plane; S212, key point positioning is selected: For each heat map Non-maximum suppression is performed to select the local maximum position as the key point coordinate: wherein ; wherein is the position coordinate of the k th key point, is the horizontal pixel coordinate of the key point in the image, is the vertical pixel coordinate of the key point in the image; When the sensor combination provides depth information, a depth value can be introduced on the basis of the two-dimensional pixel coordinates , further extending the keypoint coordinates to three dimensions: ； ； ； Thus, the three-dimensional coordinates of the key points are obtained wherein is the focal length of the camera, is the principal point coordinate; When the sensor combination has no depth information, the key points participate in subsequent analysis in the form of two-dimensional coordinates; S213, skeleton construction: According to the predefined skeleton connection relationship, the detected key points are connected to form a human skeleton: are connected to form a human skeleton: ； wherein, is a set of indices for backbone connections, the first i two-dimensional pixel coordinates of the key point, the first j two-dimensional pixel coordinates of the key point.

4. The method of claim 3, wherein, The specific steps of the posture anomaly judgment in S2 are as follows: S221, shoulder height difference detection: Key point acquisition: let the left shoulder coordinate be and the right shoulder coordinate ; Calculate the height difference of the shoulders, and calculate the difference between the left shoulder and the right shoulder in the vertical direction: ； Criteria: If , then it is determined that there is unevenness in the shoulder, where is a preset threshold value. S222, head tilt detection: Key point acquisition: let the nose coordinate be ; Calculate the midpoint of the left and right shoulders: ； Calculate the offset of the nose and the shoulder midpoint in the horizontal x-axis direction: ； Criteria: if , it is considered that there is a head misalignment phenomenon, wherein is a preset threshold value; S223, left and right body tilt detection: Key point acquisition: let the left hip coordinate be and the right hip coordinate be ; Calculate the hip midpoint: ； By calculating the midpoint of the shoulder Midpoint of the hip Calculate the upper body tilt angle by finding the angle between the line connecting the two sides and the horizontal line. ； Criteria: If then determine that the upper body is left / right tilting abnormally, wherein is a preset angle threshold.

5. The method of claim 4, wherein, The specific steps of the step length and step speed calculation in S2 are as follows: S231, draw the arm swing waveform graph: Key point trajectory acquisition: record the longitudinal coordinates of the arm-related key points in each frame of image or the relative displacement from the shoulder to form a time sequence ; Waveform calculation: Let wrist point be , shoulder point be , define arm relative displacement: ； right Perform smoothing: ； wherein, N is a sliding window size (number of frames) for controlling the smoothing degree, t is a time or frame number of the current frame, i is a frame offset index within the window, taking values ; Waveform plotting: by time The horizontal axis is... Using the vertical axis, plot the arm swing waveform, perform peak detection, and obtain arm swing information; S232, draw the step waveform graph: Foot keypoint sampling: includes selecting foot key points Longitudinal coordinate and information about the ground per frame as a time series ; Step cycle detection: In the time point of each "foot strike" is found , the step frequency (steps / min) is derived from the adjacent foot strike time intervals . ； Waveform plotting: by time Using the horizontal axis as the axis, Draw a waveform diagram of the steps to show the up-and-down swinging process of the feet; S233, step length and step speed calculation: Camera calibration and coordinate transformation: Calibrate the camera and obtain the intrinsic parameter matrix. And extrinsic parameters, for pixel coordinates Perform transformation correction to obtain world coordinates. : ； Step length calculation: horizontal distance between left and right foot key points at consecutive landing times and horizontal displacement of left and right foot key points for the same foot at consecutive landings ； The average step length is: ； Compare the average step length with the standard step length to determine whether it meets the requirements; Step rate calculation: includes stride length and step frequency combination calculation and adjacent frame foot displacement and time difference Direct calculation: If The step rate is given by: ； According to the adjacent frame foot displacement and time difference Directly calculating the step speed is: ； Compare the calculated step speed with the standard step speed to determine whether the marching speed meets the specifications.

6. The method of claim 5, wherein, The specific steps of S3 are as follows: S31, instance segmentation detection of hats: Using an instance segmentation algorithm to process the images captured by the top camera, real-time detection of the hat region of each player, for each detected hat, the algorithm outputs: a bounding box ; Segmentation mask ; class confidence; Calculate the centroid of each hat segmentation result, and the formula is: ； ； obtaining a set of center points of all the caps ; S32, row and column grouping and reference line fitting: Row and column grouping: According to the longitudinal coordinate of the hat center point The players are sorted according to the distance threshold The hats are divided into rows according to the horizontal coordinate Divided into columns; Reference line fitting: For each set of hat center points within a row Fit a straight line with least squares: ； The fitting parameter calculation formula is: ； ； Calculate deviation of fitted straight line from reference line: ideal slope Calculate angle of inclination of fitted straight line: ； At the same time, calculate the perpendicular distance from each hat center point to the fitted straight line: ； S33, alignment judgment: Linearity: Statistics of the average distance of the caps to the fitted straight line in each row or variance : ； wherein, is the number of hats for this row; If or If the preset threshold is exceeded, it is determined that the row is not aligned, and the system issues an alarm. Longitudinal alignment: fit a longitudinal reference straight line for the center points of each column of hats, calculate the deviation, and determine whether the queue is longitudinally aligned.

7. The method according to claim 6, wherein The specific steps of S4 are as follows: S41, the real-time target detection module includes target candidate region extraction and feature vector extraction, and the specific steps are as follows: Target candidate region extraction: a deep convolutional neural network is used to process the whole frame of image, the image is divided into several grids, and whether there is a moving target is predicted on each grid, and the bounding box of the candidate target is output and the corresponding confidence score, non-maximum suppression is performed on the candidate box to ensure that the output detection result is less redundant; Feature vector extraction: For each candidate region, a fixed-dimension appearance feature vector is extracted through the intermediate layers of the network ; S42, the multi-target tracking and state updating module includes state modeling and prediction, data association, and state updating, and the specific steps are as follows: State modeling and prediction: for each tracking target, the state vector is defined as: ； wherein denotes the horizontal and vertical coordinates of the center of the target bounding box, denotes the target scale, denotes the aspect ratio, are the respective velocity components; State prediction is performed using a linear motion model, and the Kalman filter algorithm is used: ； wherein k is the current time step or frame number, is the state vector predicted at step k is the state vector predicted at step is the state transition matrix, and the state covariance update formula is: ； wherein, is the process noise covariance matrix, is the state covariance matrix after the 1-step update, k- 1-step update. Data association: Motion information matching: compute the difference between each detection and the predicted state, set the observation vector and define the observation matrix then use the residual of the Kalman filter: ； wherein, is the extraction of the horizontal pixel coordinate of the center of the bounding box, is the extraction of the vertical pixel coordinate of the center of the bounding box; The measurement covariance is: ； wherein denotes the measurement noise covariance, by computing the Mahalanobis distance: ； Get the motion similarity; Appearance information matching: For the detected first i target region, the extracted feature vector is compared with the appearance feature j of the first target in the history record using the cosine distance: ； Comprehensive cost construction and matching: Build a comprehensive cost function: ； in, To balance the influence of motion and appearance information, an optimal matching method is used to match the detection results with the predicted state and update the correspondence between the tracked targets. For the i-th detection result and the i-th j The motion similarity measure between predicted trajectories is usually the Mahalanobis distance. State update: For the matched targets, use the Kalman filter update formula: ； wherein, is the updated state vector, is the observation vector; where the Kalman gain The calculation formula is: ； wherein, is the state covariance matrix obtained by the Kalman filter at the prediction stage at time step k; For unmatched targets, short-term tracking is maintained according to the predicted state, and if the tracking is not matched for a long time, the tracking is terminated; for newly detected results, new tracking target states and appearance features are initialized; S43, the PID control module: Error definition: In the image coordinate system, let the image center be For the selected tracking target, take its currently detected center coordinate Define the horizontal direction error: ； PID controller formula: According to the error Computing control signals : ； wherein, , , is a proportional, integral, derivative coefficient, is a horizontal real-time error; Control loop: The system recalculates the target center position and error after each frame of image is processed and updates the PID controller output.

8. The method of claim 7, wherein, The specific steps of S5 are as follows: S51, action index quantization: According to the preset standard model, the collected data are compared, and each action error and score are output, so as to form a quantitative evaluation result; S52, real-time feedback: Using a human-machine interaction interface (HMI), data, images and action waveform information are displayed in real time to the instructor and the trainee, and online labeling and alarm are given for abnormal actions.

9. A platoon training device for use in a platoon training method according to any of the preceding claims 1-8, characterized in that It comprises a front-end multi-camera module, a high-performance data processing server and a data storage and management module, the front-end multi-camera module is used for fixing a camera and a following camera installed on a slide rail, the high-performance data processing server is used for supporting real-time inference of a deep learning model, and the data storage and management module is used for supporting long-term training data recording and query.

10. A platoon training system for use with a platooning method according to any of the preceding claims 1-8, characterized in that The overall system is composed of a data acquisition layer, a data processing layer, an analysis and evaluation layer and a user interaction layer, and comprises a single-person training system, a row training system and a square training system.

Images