A method for evaluating the effect of live-fire shooting training

Abstract

The present application discloses a method for evaluating the effect of live-fire shooting training, which relates to the field of military training, including: constructing a live-fire shooting training evaluation system, the system including a target drone, a shooting device and a data processing device; obtaining a shooting action image of a trainee through a camera device; extracting features of the shooting action image using a pre-trained convolutional neural network to obtain a feature vector sequence describing the trainee's gun-holding posture; according to the feature vector sequence, establishing a shooting posture stability evaluation model through a bidirectional long short-term memory neural network BiLSTM‑Attention based on an attention mechanism to obtain shooting posture stability data; analyzing the collected shooting action images using a recurrent neural network model based on an attention mechanism to obtain the moving trajectory and dwell time of the trainee's sight between the front sight and the target, generating aiming trajectory and aiming time data; generating a shooting training effect evaluation report. In view of the single evaluation dimension in the prior art, the present application generates a comprehensive and objective evaluation.

Description

Technical Field

[0001] The present application relates to the field of military training, and in particular to a method for evaluating the effect of live-fire shooting training. Background Art

[0002] At present, the evaluation of the effectiveness of live-fire shooting training mainly adopts methods such as manual observation and shooting performance statistics, which have problems such as single evaluation dimension and strong subjectivity. Traditional evaluation methods mainly focus on shooting performance, such as shooting hit rate and shooting time, and ignore important factors such as posture stability and aiming trajectory during shooting, resulting in one-sided evaluation results that cannot fully reflect the shooting ability and existing problems of the trainees.

[0003] Studies at home and abroad have attempted to apply machine learning and computer vision technology to the evaluation of shooting training effects. Some studies use traditional image processing algorithms and shallow neural networks to analyze shooting postures, but ignore the information of the time dimension and cannot effectively capture the dynamic change characteristics of postures. Some studies also use manual annotation to extract sight aiming trajectories, but this is inefficient and highly subjective. Summary of the invention

[0004] In view of the problem of single dimension in the evaluation of live-fire shooting training effect in the prior art, this application provides a method for evaluating the effect of live-fire shooting training, which uses a pre-trained three-dimensional convolutional neural network to extract the spatiotemporal characteristics of shooting posture, establishes a shooting posture stability evaluation model through a bidirectional long short-term memory neural network based on an attention mechanism, and outputs shooting posture stability data. At the same time, an encoder-decoder structure based on an attention mechanism is used to analyze shooting action images, obtain the aiming trajectory and dwell time data of the trainee's sight between the front sight and the target, and generate a comprehensive and objective shooting training effect evaluation report.

[0005] The purpose of this application is achieved through the following technical solutions.

[0006] The present application provides a method for evaluating the effect of live-fire shooting training, comprising: constructing a live-fire shooting training evaluation system, the system comprising a target drone, a shooting device and a data processing device; wherein a camera device is installed on the shooting device; during the training process, the number of shots, the shooting time and the number of hits are collected; a shooting action image of a trainee is obtained through the camera device; a pre-trained convolutional neural network is used to extract features of the shooting action image to obtain a feature vector sequence describing the trainee's gun holding posture; according to the feature vector sequence, a shooting posture stability evaluation model is established through a bidirectional long short-term memory neural network BiLSTM-Attention based on an attention mechanism to obtain shooting posture stability data; a recurrent neural network model based on an attention mechanism is used to analyze the collected shooting action images to obtain the movement trajectory and residence time of the trainee's sight between the crosshairs and the target, and generate aiming trajectory and aiming time data; and a shooting training effect evaluation report is generated according to the number of shots, the shooting time, the number of hits, the shooting posture stability, the aiming trajectory and the aiming time.

[0007] Furthermore, a feature vector sequence describing the trainee's gun-holding posture is obtained, including: constructing the collected continuous multi-frame shooting action images into a three-dimensional tensor; wherein the first dimension and the second dimension represent the spatial dimension of the shooting action image, and the third dimension represents the time dimension of the shooting action image; using a pre-trained three-dimensional convolutional neural network 3D-CNN to extract spatiotemporal features of the three-dimensional tensor to obtain a spatiotemporal feature map of the shooting action; wherein the three-dimensional convolutional neural network includes multiple three-dimensional convolutional layers and three-dimensional pooling layers; flattening the spatiotemporal feature map into a one-dimensional feature vector as an input of a fully connected layer; using the fully connected layer to perform a nonlinear transformation on the one-dimensional feature vector using a Leaky ReLU activation function, to obtain a feature vector sequence describing the trainee's gun-holding posture.

[0008] Furthermore, a pre-trained three-dimensional convolutional neural network 3D-CNN is used to extract spatiotemporal features from the three-dimensional tensor, including: inputting the three-dimensional tensor into a three-dimensional convolutional layer, and using a separable convolution kernel in the three-dimensional convolutional layer to slide in the time and space dimensions to extract local spatiotemporal features of the shooting action image; wherein the separable convolution kernel includes an attention module, a deep convolution kernel, and a point-by-point convolution kernel; inputting the extracted local spatiotemporal features into a three-dimensional pooling layer, and performing a downsampling operation on the local spatiotemporal features in the time and space dimensions; wherein, maximum pooling is used in the time dimension to extract the temporal features of the shooting action; and average pooling is used in the space dimension to extract the regional features of the shooting action; through the alternating operation and hierarchical structure of multiple three-dimensional convolutional layers and three-dimensional pooling layers of the three-dimensional convolutional neural network, the spatiotemporal features of the shooting action are extracted and fused layer by layer to obtain a spatiotemporal feature map of the shooting action.

[0009] Furthermore, local spatiotemporal features of shooting action images are extracted, including: using an attention module to learn the dependencies of shooting actions in the time dimension and the space dimension, and generating attention weight maps in the time dimension and the space dimension; using a deep convolution kernel to slide in the time dimension, and according to the attention weight map in the time dimension, assigning different weights to feature maps of different time steps to extract the temporal features of the shooting action; using a point-by-point convolution kernel to slide in the space dimension, and according to the attention weight map in the space dimension, assigning different weights to feature maps at different spatial positions to extract the regional features of the shooting action; combining the output feature maps of the deep convolution kernel and the point-by-point convolution kernel to obtain the local spatiotemporal features of the shooting action.

[0010] Furthermore, the shooting posture stability data is obtained, including: inputting the obtained feature vector sequence into the input layers of the forward LSTM and backward LSTM of the bidirectional long short-term memory neural network BiLSTM at the same time, and obtaining the forward and backward temporal dependencies of the shooting posture in chronological order; at each time step of the BiLSTM, using the attention mechanism to calculate the association weights between the current time step and other time steps, and obtaining the attention feature vector of the current time step by weighted summation; jointly training the weight parameters of the BiLSTM and the attention mechanism through an iterative optimization algorithm, and constructing a shooting posture stability evaluation model BiLSTM-Attention; inputting the feature vector sequence data into the trained BiLSTM-Attention model in chronological order: the BiLSTM-Attention model extracts the temporal features of the shooting posture through the bidirectional neural network structure and the attention mechanism; the BiLSTM-Attention model outputs the corresponding shooting posture stability evaluation score at each time step; and generating a shooting posture stability evaluation report according to the shooting posture stability evaluation scores and the corresponding attention weights output by the BiLSTM-Attention model at different time steps.

[0011] Furthermore, a shooting posture stability evaluation model BiLSTM-Attention is constructed, including: initializing the weight parameters of BiLSTM and attention mechanism; using the cross-validation method to divide the feature vector sequence data into training set, validation set and test set; wherein the training set is used for model training, the validation set is used for model performance evaluation, and the test set is used to evaluate the generalization performance of the model; using the training set to train BiLSTM-Attention, and calculating the hidden state, attention weight and shooting posture stability evaluation score of each time step through forward propagation; using the cross entropy loss function to calculate the loss value of the BiLSTM-Attention model on the training set; calculating the gradient of the loss function to the weight parameters of BiLSTM and attention mechanism through the back-propagation algorithm; and updating the weight parameters of BiLSTM and attention mechanism.

[0012] Furthermore, the cross entropy loss function is expressed as follows: , where the parameters are defined as follows: N represents the number of training samples, which indicates the total number of shooting posture stability data participating in the training. T represents the number of time steps, which indicates the sequence length of the shooting posture stability data processed by the BiLSTM-Attention model. It represents the true stability label of the i-th training sample at the t-th time step, and its value is 0 or 1, indicating whether the shooting posture at this time step is stable. It represents the stability evaluation score of the BiLSTM-Attention model for the i-th training sample at the t-th time step. The value range is [0, 1], which indicates the probability that the model predicts that the shooting posture at this time step is stable. Represents the weight coefficient of the positive sample at time step t, which is used to balance the contribution of positive samples at different time steps. It can be set based on prior knowledge or optimized as a learnable parameter. : The hyperparameter of Focal Loss, used to adjust the weight of difficult samples. When , it degenerates into the standard cross entropy loss function; when , the weight of the hard-to-separate samples is increased, making the model pay more attention to the samples that are predicted incorrectly. : Negative sample weight coefficient, used to balance the contribution of positive and negative samples. When the ratio of positive and negative samples in the data set is unbalanced, by setting the appropriate , which can alleviate the problem of class imbalance. The improved loss function introduces the weight coefficient of positive and negative samples based on the original Focal Loss. and , weight the loss terms of positive and negative samples. For positive samples, the weight coefficient is ; For negative samples, the weight coefficient is and . Through reasonable settings and , which can flexibly balance the contribution of samples of different time steps and categories, effectively alleviating the problems of category imbalance and time step importance imbalance. At the same time, the improved loss function has a better effect on the weight adjustment of difficult samples. Based on this, we introduce the Item, suppresses the correctly predicted negative samples, and further improves the model's attention to difficult negative samples.

[0013] Furthermore, the weight parameters of BiLSTM and attention mechanism are updated through the following expressions: , , , , ; Among them, the parameters are defined as follows: : The gradient of the current moment t, through the loss function L about the parameters of the previous moment Derived to get . : The first moment of the gradient (i.e., the exponentially weighted moving average of the gradient), used to estimate the mean of the gradient. The update is done by exponential weighted averaging, which smooths the gradient oscillation and improves the stability of the optimization. : The decay rate of the first-order moment, used to control the update speed of the gradient moving average, with a value range of [0, 1). The larger the value, the slower the update speed and the greater the influence of historical gradients; The smaller the value, the faster the update speed and the greater the influence of the current gradient. : The exponentially weighted moving average of the maximum absolute value of the gradient is used to adaptively adjust the learning rate of each parameter. By comparing with the previous moment and the current absolute value of the gradient The update method takes the maximum value, which can adaptively adjust the learning rate and accelerate convergence. : The decay rate of the maximum moving average, used to control The update speed is in the range of [0, 1). The larger the value, the slower the update speed and the greater the impact of the historical maximum value; The smaller the value, the faster the update speed and the greater the influence of the absolute value of the current gradient. : The corrected gradient first-order moment is used to correct The deviation caused by initialization to 0. As the time step t increases, will gradually decay to 0. Will gradually approach The true value of . : The adaptive learning rate at the current time t, directly used No bias correction is required. : The updated model parameters at the current time t, according to the corrected gradient first-order moment and adaptive learning rate to update. : Initial learning rate, controls the step size of each parameter update. It needs to be adjusted according to the specific task and data set, and the value range is usually The exponentially weighted moving average of the maximum absolute value of the gradient is used to adaptively adjust the learning rate of each parameter. Only the first-order moment and the maximum moving average of the gradient need to be calculated, which has high computational efficiency and is suitable for the optimization of large-scale data sets and complex models.

[0014] Furthermore, aiming trajectory and aiming time data are generated, including: inputting the collected shooting action image into a recurrent neural network model based on an attention mechanism; wherein the recurrent neural network model includes an encoder and a decoder; the encoder adopts a bidirectional long short-term memory neural network BiLSTM structure, and the decoder adopts an attention mechanism and a long short-term memory neural network LSTM structure; the encoder receives the shooting action image, extracts the temporal features of the shooting action image through BiLSTM, and obtains a coded feature sequence; wherein, at each time step, BiLSTM uses a bidirectional LSTM unit to respectively extract the forward and backward temporal dependencies of the shooting action image, and splices the forward and backward hidden states. , as the encoding feature of the current time step; the decoder receives the encoding feature sequence and the hidden state of the previous time step, calculates the attention weights of the current time step and each time step in the encoding feature sequence through the attention mechanism, and obtains the attention feature vector of the current time step; the decoder inputs the attention feature vector and the hidden state of the previous time step into the LSTM unit, updates the hidden state through the gating mechanism, and generates the aiming position coordinates of the current time step; the loop is executed until the decoder generates a complete aiming trajectory coordinate sequence, and the movement trajectory of the trainer's sight between the crosshairs and the target is obtained; according to the aiming trajectory coordinate sequence, the trainer's sight dwell time at each aiming position is calculated to generate aiming time data.

[0015] Furthermore, the aiming time data is generated, including: for each coordinate point in the aiming trajectory coordinate sequence, selecting multiple coordinate points before and after it, calculating weights according to the time interval between the coordinate points, and performing weighted averaging on the selected coordinate points to obtain a smoothed coordinate value of the current coordinate point; traversing each coordinate point in the aiming trajectory coordinate sequence, updating its coordinate value to a smoothed coordinate value, and obtaining a smoothed aiming trajectory coordinate sequence; calculating the Euclidean distance between two adjacent aiming positions according to the smoothed aiming trajectory coordinate sequence, and when the Euclidean distance is less than a threshold, classifying the adjacent aiming positions into the same aiming area; counting the number of aiming positions in each aiming area, multiplying the number of aiming positions by the unit time step, and obtaining the residence time of the trainee's sight in each aiming area; using the central coordinates of the aiming area as the representative coordinates of the corresponding area, and generating aiming time data including the representative coordinates of the aiming area and the corresponding residence time.

[0016] Compared with the prior art, the advantages of this application are:

[0017] The introduction of a three-dimensional convolutional neural network to extract the spatiotemporal features of shooting posture overcomes the limitation of traditional two-dimensional image analysis methods that ignore the time dimension information. By extracting features in both time and space dimensions, it is possible to comprehensively characterize the changing patterns of shooting postures in continuous time periods, providing a richer and more effective feature representation for the evaluation of shooting posture stability.

[0018] A shooting posture stability assessment model is constructed using a bidirectional long short-term memory neural network based on an attention mechanism. The attention mechanism is used to adaptively focus on key moments and regions in the shooting posture sequence, effectively extracting posture features that have a significant impact on stability assessment. At the same time, the bidirectional LSTM can fully explore the forward and backward temporal dependencies of the posture sequence, enhance the model's ability to depict dynamic changes in posture, and make stability assessment more accurate and reliable.

[0019] During the training process of the posture stability assessment model, an improved Focal Loss function is introduced. By setting the sample weight coefficient and the difficult-to-classify sample adjustment factor, the category imbalance problem of the data set can be effectively alleviated, and the model's attention to posture samples with poor stability can be increased, thereby enhancing the model's classification performance and generalization ability.

[0020] The encoder-decoder structure based on the attention mechanism is used to analyze the shooting action image sequence, and automatically learn to extract the aiming trajectory of the trainer's sight between the crosshairs and the target. Compared with the manual labeling method, the efficiency and objectivity of aiming trajectory extraction are improved. The introduction of the attention mechanism enables the model to adaptively focus on the image area that is critical to aiming, improving the accuracy of aiming trajectory and dwell time analysis.

[0021] By smoothing the aiming trajectory coordinate sequence, eliminating tiny eye jitters and noise interference, and automatically classifying the aiming area, a refined analysis of the trainee's sight dwell time can be achieved. The quantified dwell time index provides an objective basis for evaluating the trainee's aiming ability and concentration, and enriches the dimension of shooting training effect evaluation.

[0022] Comprehensive shooting times, time, hit rate, shooting posture stability, aiming trajectory, dwell time and other multi-dimensional indicators form a comprehensive, objective and fine-grained shooting training effect evaluation report. The multi-angle evaluation results can reveal the strengths and weaknesses of the trainees in the shooting process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0024] Figure 1 is an exemplary flow chart of a live-fire shooting training effect evaluation method according to the present application;

[0025] Figure 2 is an exemplary flow chart of obtaining spatiotemporal characteristics of a shooting action according to the present application;

[0026] Figure 3is an exemplary flow chart for constructing a BiLSTM-Attention model according to the present application;

[0027] Figure 4 is an exemplary flow chart for generating aiming time data according to the present application. DETAILED DESCRIPTION

[0028] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 The present invention is an exemplary flow chart of a method for evaluating the effect of live-fire shooting training according to the present application, comprising: constructing a live-fire shooting training evaluation system, the system comprising a target machine, a shooting device and a data processing device; wherein a camera device is installed on the shooting device; during the training process, the number of shots, shooting time and number of hits are collected; the shooting action image of the trainee is obtained through the camera device; the shooting action image is extracted by using a pre-trained convolutional neural network to obtain a feature vector sequence describing the gun holding posture of the trainee; according to the feature vector sequence, a shooting posture stability evaluation model is established by using a bidirectional long short-term memory neural network BiLSTM-Attention based on an attention mechanism to obtain shooting posture stability data; the collected shooting action images are analyzed by using a recurrent neural network model based on an attention mechanism to obtain the movement trajectory and residence time of the trainee's line of sight between the crosshairs and the target, and generate aiming trajectory and aiming time data; according to the number of shots, shooting time, number of hits, shooting posture stability, aiming trajectory and aiming time, a shooting training effect evaluation report is generated.

[0030] Construct a live-fire shooting training evaluation system, which includes a target drone, a shooting device, and a data processing device; a camera device is installed on the shooting device; select a suitable target type according to training needs, such as a fixed target, a mobile target, an electronic target, etc.; place the target drone on a target position that meets the shooting safety and distance requirements; connect the power supply and control system of the target drone to ensure that it can work normally. Select corresponding shooting weapons according to the training subjects, such as rifles, pistols, etc.; check the mechanical reliability and shooting accuracy of the weapons to ensure that they are in the best working condition; install a high-speed camera on the shooting device to shoot the shooting movements of the trainees; debug the camera's field of view, frame rate, and exposure parameters so that it can clearly record the entire shooting process. Select a high-performance workstation as the data processing hardware platform, equipped with a GPU to accelerate computing; build a shooting data acquisition and processing software environment, including image acquisition card drivers, deep learning frameworks, etc.; integrate the data interfaces of the target drone and the camera to realize automatic entry and synchronization of shooting data.

[0031] During the training process, the number of shots, shooting time and number of hits are collected; the trigger signal of each shooting action is recorded through the magazine or firing mechanism sensor of the shooting device; the trigger signals are counted and numbered to obtain the number of shots data. When each shot is triggered, the timestamp of the shooting action is recorded through a high-precision timer; according to the sequence of timestamps, the time interval between adjacent shooting actions is calculated to obtain the shooting time data. The electronic sensing device or acoustic testing device of the target drone is used to detect the position and number of bullets hitting the target; the hit signal detected by the target drone is matched with the number of shots, and the number of hits in each shot is counted.

[0032] The shooting action images of the trainees are acquired through the camera device; the trigger signal of the high-speed camera is synchronized with the shooting signal of the shooting device, and the two are precisely controlled by the same control module; each time the shooting device shoots, the control module sends a synchronization signal to trigger the high-speed camera to automatically shoot a continuous image sequence at the moment of shooting; the shooting frame rate is not less than 1000 frames / second, and the exposure time is not more than 1 millisecond to ensure that the key details of the shooting action are accurately captured; the industrial camera powered by POE is used, and the FPGA chip is used to realize real-time image acquisition and preprocessing; the acquisition card is connected to the industrial camera through 10 Gigabit Ethernet, supporting the transmission of high-speed image data streams of more than 1GB per second; the collected original images are preprocessed in turn by distortion correction, spatial calibration and time synchronization: multiple calibration images containing chessboards of known sizes are taken at different positions and angles at the shooting scene: a black and white chessboard calibration plate with a specification of 12x9 is selected, and the physical size of each chessboard is 30mmx 30mm; fix the calibration plate on a flat bracket to keep the overall flatness error of the chessboard plane within 0.1mm; adjust the shooting parameters of the high-speed camera: set the frame rate to 200 frames / second, the exposure time to 1ms, and the image resolution to 1024 x 1024 pixels; select 6 different positions at the shooting scene, select 4 different angles at each position, and take a total of 24 images containing the complete chessboard; the shooting positions must cover the main working range of the camera in the actual shooting test, and the shooting angles must have certain changes in the horizontal and vertical directions to improve the calibration accuracy.

[0033] The pixel coordinates of the chessboard corners in each calibration image are extracted, and the correspondence between the pixel coordinates and the world coordinates is constructed in combination with the physical size of the chessboard: the find Chessboard Corners function of OpenCV is used to extract the chessboard corners of each calibration image to obtain the pixel coordinates of 11x8 inner corners; since the physical size of the chessboard is known, the world coordinates of the chessboard corners can be expressed as (0, 0, 0), (30, 0, 0), (60, 0, 0), ..., (330, 270, 0); the extracted pixel coordinates and the corresponding world coordinates are constructed into a one-to-one corresponding coordinate pair, each coordinate pair is in the form of ((u, v), (x, y, z)), where (u, v) is the pixel coordinate of the corner point, and (x, y, z) is the world coordinate of the corner point; the coordinate pairs are constructed for the 24 calibration images respectively, and finally 24 sets of coordinate pair data are obtained, each set contains 88 valid pixel coordinate-world coordinate pairs.

[0034] Based on the corresponding relationship of corner points, the Zhang Zhengyou calibration algorithm is used to solve the intrinsic parameter matrix and distortion coefficient of the high-speed camera. According to the mathematical model of the Zhang Zhengyou calibration algorithm, the mapping relationship equation group between pixel coordinates, world coordinates and camera parameters is established: let the pixel coordinates be (u, v), the world coordinates be (X, Y, Z), the camera intrinsic parameter matrix be A, the extrinsic matrix be [R|t], and the distortion coefficient be . Establish the following mapping relationship equation: , where T represents matrix transpose. Considering the influence of distortion, radial distortion and tangential distortion models are introduced to correct the pixel coordinates: , Substitute all the obtained coordinate pair data into the equation system to construct an overdetermined equation system, where the unknowns are the camera's intrinsic parameter matrix and distortion coefficients and other parameters: For each pixel coordinate-world coordinate pair ((u, v), (X, Y, Z)), substitute it into the equation system to obtain two constraint equations; there are a total of 24 sets of coordinate pair data, each containing 88 coordinate pairs, so 24882 = 4224 constraint equations can be listed; the unknowns include the 4 parameters in the intrinsic parameter matrix A , distortion coefficient , and the 6 parameters in the external parameter matrix [R|t] corresponding to each set of data, there are a total of 4 + 4 + 24*6 = 152 unknowns; since the number of constraint equations is much larger than the number of unknowns, an overdetermined system of equations is formed; the Levenberg-Marquardt optimization algorithm is used to solve the overdetermined system of equations to obtain the initial values ​​of the camera internal parameter matrix and distortion coefficients: the residual sum of squares of all constraint equations is used as the objective function to construct a least squares optimization problem; the Levenberg-Marquardt algorithm is used to iteratively optimize and solve the objective function to obtain the initial estimates of the internal parameter matrix A and the distortion coefficients (k1, k2, p1, p2); the initial value of the iteration is set to: , , , ; Set the iteration termination condition as follows: the residual sum of squares is less than the preset threshold, or the number of iterations reaches the preset upper limit; Substitute the initial value into the back projection error equation, calculate the back projection error of each corner point, and construct the error function: According to the initial estimated intrinsic parameters and distortion coefficients, perform distortion correction on each pixel coordinate to obtain the corrected pixel coordinates ; Using the world coordinates (X, Y, Z) and the corresponding external parameter matrix [R|t], calculate the theoretical value of the corrected pixel coordinates through back projection ; Calculate the backprojection error: , , calculate the square sum of the reverse projection errors of all corner points and construct the error function: .

[0035] The least squares method is used to perform nonlinear optimization on the error function, and the optimal values ​​of the camera internal parameter matrix and distortion coefficient are iteratively solved. The back-projection error function is used as the new objective function, and the initial estimate is used as the starting point. The Levenberg-Marquardt algorithm is re-applied for nonlinear optimization: Define the back-projection error function ;in, is the actual coordinate of the i-th chessboard corner point on the image plane, is the known coordinates of the i-th chessboard corner point in the world coordinate system; Indicates that the world coordinates The predicted coordinates are obtained by mapping the camera projection model and the distortion model to the image plane; the initial estimated internal parameter matrix A and the distortion coefficient As the starting point, the Levenberg-Marquardt optimization problem is constructed; in each iteration, the Jacobian matrix of the objective function with respect to the intrinsic parameters and distortion coefficients is calculated, the incremental equation is constructed, and the incremental step size is solved: the four parameters of the objective function E with respect to the intrinsic parameter matrix A are calculated and distortion coefficient Partial derivatives of , construct the Jacobian matrix J:

[0036] . Construct the increment equation: ,in, is the incremental step vector to be determined, including the increments of the intrinsic parameters and distortion coefficients; λ is the damping factor, and I is the unit matrix; solve the incremental equation to obtain the incremental step of the current iteration step ; Use the Gauss-Newton method and trust region strategy to adaptively adjust the damping factor, control the iteration step size, and improve the convergence speed and stability: According to the size of the incremental step size Δx, adaptively adjust the damping factor λ: If If it is large, it means that the current iteration step size is too large, which may cause optimization divergence. It is necessary to increase λ to reduce the step size. is smaller, indicating that the current iteration step size is more appropriate, and λ can be reduced to speed up the convergence speed; Add to the current internal parameters and distortion coefficient estimates to get new estimates; calculate the objective function value under the new estimates , compared with the previous objective function value: if the new objective function value is smaller than the previous value, the new estimate is accepted and λ is reduced to speed up convergence; if the new objective function value is larger than the previous value, the new estimate is rejected, λ is increased to reduce the step size, and the incremental equation is recalculated.

[0037] Iterate and optimize until the objective function decreases to a value less than the preset threshold, or the number of iterations reaches the preset upper limit, and obtain the internal parameter matrix A and the distortion coefficient The optimal estimate of: Set the objective function drop threshold ε and the maximum number of iterations ; After each iteration, calculate the decrease value of the objective function : ;if Or the number of iterations reaches , then stop the iteration and output the current internal parameter matrix A and distortion coefficient As the optimal estimate; otherwise, take the current estimate as the new starting point and continue the next iterative optimization; save the optimized intrinsic parameter matrix and distortion coefficient as the camera calibration parameters for subsequent image distortion correction: The four parameters of the optimal estimated intrinsic parameter matrix A are and distortion coefficient Save as the calibration result of the camera; write the calibration parameters to the configuration file or database for subsequent reading and use; perform distortion correction on each frame of the original image collected by the shooting action using the calibrated intrinsic parameters and distortion parameters: read the calibrated intrinsic parameter matrix A and distortion coefficients ; For each frame of the original image, traverse each pixel in the image :According to the radial distortion model and the tangential distortion model, calculate the corrected coordinates corresponding to the pixel point : ; The corrected coordinates Mapping back to the original image coordinate system to obtain the corrected pixel position; performing sub-pixel interpolation on the corrected pixel position to obtain the pixel value of the corrected image; outputting the corrected shooting action image, effectively eliminating the influence of lens distortion on image quality.

[0038] Figure 2 This is an exemplary flow chart for obtaining the spatiotemporal features of shooting actions according to the present application, using a pre-trained convolutional neural network to extract features from shooting action images to obtain a feature vector sequence describing the trainee's gun holding posture; determining the spatial dimension and time dimension of the shooting action image: spatial dimension: refers to the width and height of a single frame image, corresponding to the first and second dimensions of the tensor respectively; time dimension: refers to the number of frames in the shooting action image sequence, corresponding to the third dimension of the tensor; assuming that the width of a single frame image is 640 pixels and the height is 480 pixels, and the shooting action image sequence contains 100 frames, the spatial dimension is (640, 480) and the time dimension is 100; based on the determined spatial dimension and time dimension, create a three-dimensional array with a shape of (640, 480, 100) for storing the shooting action image sequence ; Each element in the three-dimensional array represents a pixel value, and the initial value can be set to 0; the np.zeros function of the NumPy library can be used to easily create an array of all zeros of a specified shape: tensor=np.zeros((640, 480, 100), dtype=np.uint8). Fill the shooting action image sequence into the three-dimensional tensor: traverse each frame of the shooting action image sequence; for the i-th frame image (i=0, 1, ..., 99), copy its pixel value to the i-th time slice of the three-dimensional tensor, that is, ; Using NumPy's array slicing operation, data filling can be easily implemented: foriinrange(100): img = get_frame(i) #Get the i-th frame image, . Perform data preprocessing on the three-dimensional tensor: perform preprocessing operations such as normalization and standardization on the image data in the tensor as needed; Normalization: Scale the pixel values ​​to the range of [0, 1], which can be achieved using the division operation: tensor = tensor.astype (np.float32) / 255.0; Standardization: Subtract the mean from the pixel value and divide it by the standard deviation to make the data distribution more stable: , , The preprocessed three-dimensional tensor is output: a three-dimensional tensor data representing the shooting action image sequence is obtained; the shape of the tensor is (640, 480, 100), which corresponds to the width, height and number of frames of the image respectively.

[0039] The pre-trained 3D convolutional neural network 3D-CNN is used to extract spatiotemporal features from the 3D tensor to obtain the spatiotemporal feature map of the shooting action. The 3D-CNN model pre-trained on a large-scale video dataset is used, such as C3D, I3D, etc. The pre-trained model has learned a rich spatiotemporal feature representation and can be used for shooting action feature extraction; the network structure and weights of the pre-trained model are loaded using deep learning frameworks such as TensorFlow and PyTorch; the constructed 3D tensor of the shooting action image is sent to the input layer of the 3D-CNN; the shape of the 3D tensor is (height, width, frames, channels), which corresponds to the height, width, number of frames and number of channels of the image respectively; the input layer usually performs pre-processing operations such as normalization and standardization on the tensor.

[0040] Use three-dimensional convolutional layers to extract local spatiotemporal features: the three-dimensional convolutional layer contains multiple separable convolution kernels, which slide in the time and space dimensions to extract local spatiotemporal features; the separable convolution kernel is composed of an attention module, a deep convolution kernel, and a point-by-point convolution kernel: the attention module learns the dependencies of shooting actions in the time and space dimensions and generates an attention weight map; the deep convolution kernel slides in the time dimension, and according to the time attention weight map, the feature maps of different time steps are weighted to extract temporal features; the point-by-point convolution kernel slides in the spatial dimension, and according to the spatial attention weight map, the feature maps of different spatial positions are weighted to extract regional features; the output feature maps of the separable convolution kernel are spliced ​​in the channel dimension to obtain fused local spatiotemporal features.

[0041] The local spatiotemporal features are downsampled using the three-dimensional pooling layer: the local spatiotemporal features extracted by the three-dimensional convolutional layer are sent to the three-dimensional pooling layer; the maximum pooling operation is used in the time dimension to extract the key timing features of the shooting action: the pooling window size and step size of the time dimension are set, such as the window size is 2 and the step size is 2; the pooling window is slid in the time dimension, and the maximum value in each window is taken as the output feature; the average pooling operation is used in the spatial dimension to extract the regional features of the shooting action: the pooling window size and step size of the spatial dimension are set, such as the window size is 2x2 and the step size is 2; the pooling window is slid in the spatial dimension, and the average value in each window is taken as the output feature; the pooling operation reduces the spatiotemporal resolution of the feature map while retaining the key spatiotemporal features.

[0042] Through the alternating operation and hierarchical structure of multiple 3D convolutional layers and 3D pooling layers, the spatiotemporal features of the shooting action are extracted and fused layer by layer: 3D-CNN usually contains a hierarchical structure formed by alternating multiple 3D convolutional layers and 3D pooling layers; as the number of network layers increases, the spatiotemporal resolution of the feature map gradually decreases, but the semantic level of the features gradually increases; the bottom 3D convolutional layer extracts the local spatiotemporal features of the shooting action, such as edges, textures, etc.; the middle 3D convolutional layer extracts the regional spatiotemporal features of the shooting action, such as the movement of arms, guns and other parts; the high-level 3D convolutional layer extracts the global spatiotemporal features of the shooting action, such as the overall pattern and rhythm of the action; the last layer of the network outputs the spatiotemporal feature map of the shooting action, which comprehensively encodes the spatiotemporal feature representation of the action.

[0043] Get the spatiotemporal feature map of the shooting action: The spatiotemporal feature map output by 3D-CNN is usually a four-dimensional tensor with the shape of ; The height and width of the spatiotemporal feature map are smaller than the input tensor, and the number of channels is larger than the input tensor, which reflects the hierarchical abstraction of the features; the spatiotemporal feature map can be used for subsequent shooting action analysis tasks, such as action recognition, action detection, action segmentation, etc.; the spatiotemporal feature map can be input into network structures such as the fully connected layer and LSTM for higher-level semantic understanding and application.

[0044] Flatten the spatiotemporal feature map into a one-dimensional feature vector as the input of the fully connected layer: Get the dimensional information of the spatiotemporal feature map: The spatiotemporal feature map is usually a four-dimensional tensor with a shape of (batch_size, height, width, channels); batch_size represents the number of input shooting action samples, height and width represent the spatial dimension of the feature map, and channels represent the number of channels of the feature map; record the spatial dimension size height and width of the feature map, as well as the number of channels channels; use adaptive pooling operation to adjust the spatial dimension of the spatiotemporal feature map to a fixed size: before flattening the spatiotemporal feature map into a one-dimensional feature vector, first adaptively adjust the spatial dimension of the feature map; use adaptive average pooling (Adaptive Average Pooling) or adaptive maximum pooling (Adaptive Max Pooling) Pooling) operation; set the target space dimension size to (target_height, target_width), such as (7, 7), (14, 14), etc.; the adaptive pooling operation automatically adjusts the size and step size of the pooling window according to the target space dimension size, and adjusts the spatial dimension of the feature map to the specified size; the shape of the adjusted feature map is (batch_size, target_height, target_width, channels); the adaptive pooling operation can process input feature maps of different sizes, so that the length of the flattened one-dimensional feature vector is fixed, which is convenient for subsequent fully connected layer processing.

[0045] The attention mechanism is introduced to assign different weights to different regions and channels of the spatiotemporal feature map: before flattening the feature map, the attention weights are introduced to the spatial dimension and channel dimension of the feature map; generation of spatial attention weights: global average pooling and global maximum pooling are performed on the adjusted feature map in the channel dimension to obtain two spatial attention feature maps; the two spatial attention feature maps are spliced ​​in the channel dimension, and a spatial attention weight map is generated through a convolution layer and a Sigmoid function; the original feature map is multiplied by the spatial attention weight map to obtain a spatially weighted feature map; generation of channel attention weights: global average pooling and global maximum pooling are performed on the spatially weighted feature map in the spatial dimension to obtain two channel attention feature vectors; the two channel attention feature vectors are spliced ​​in the channel dimension, and a channel attention weight vector is generated through two fully connected layers and a Sigmoid function; the spatially weighted feature map is multiplied by the channel attention weight vector to obtain a channel weighted feature map; the introduction of the attention mechanism can adaptively adjust the importance of different regions and channels, highlight key feature information, and suppress non-critical noise information.

[0046] Use the feature pyramid structure to fuse spatiotemporal feature maps of different scales: Before flattening the feature map, perform feature fusion on spatiotemporal feature maps of different levels; use a feature pyramid structure, such as FPN (Feature Pyramid Network); for the spatiotemporal feature map of the i-th layer, upsample the spatial dimension of the spatiotemporal feature map to the size of the (i-1)-th layer feature map: let the spatial dimension of the spatiotemporal feature map of the i-th layer be , the spatial dimension of the spatiotemporal feature map of the (i-1)th layer is ; Use upsampling methods such as bilinear interpolation or deconvolution to enlarge the spatial dimension of the feature map of the i-th layer to ; The upsampling operation increases the spatial resolution of the feature map, making it have the same spatial dimension as the feature map of the (i-1)th layer; the upsampled feature map is concatenated with the feature map of the (i-1)th layer in the channel dimension to obtain the fused feature map: Let the number of channels of the upsampled feature map of the i-th layer be Ci, and the number of channels of the feature map of the (i-1)th layer be ; In the channel dimension, the two feature maps are concatenated to obtain a fused feature map with the number of channels being ; The splicing operation on the channel dimension combines feature maps of different levels and integrates target information at different scales; a convolution layer is applied to the fused feature map to adjust the number of channels and perform feature transformation: a convolution layer is designed with a convolution kernel size of 1x1 and an output channel number of C; the convolution layer is applied to the fused feature map to obtain an adjusted feature map with C channels; the 1x1 convolution layer can be regarded as a feature transformation, which is used to adjust the number of channels of the feature map and perform linear combination and nonlinear transformation on the features; recursively upsample, splice and convolve the feature map of each layer to obtain a series of fused feature maps of different scales: From the high layer to the bottom layer of 3D-CNN, the feature maps of each layer are upsampled, concatenated and convolved in turn; a series of fused feature maps of different scales are obtained, each of which fuses the spatiotemporal feature information of different levels; the spatial resolution of the fused feature map is the same as the original feature map, but the number of channels is C, which represents the feature dimension after fusion; the fused feature maps of different scales are concatenated in the channel dimension to obtain the final multi-scale spatiotemporal feature map: assuming that 3D-CNN has a total of L layers, L fused feature maps of different scales are obtained; in the channel dimension, the L fused feature maps are concatenated to obtain the final multi-scale spatiotemporal feature map, whose number of channels is (L *C); the multi-scale spatiotemporal feature map fuses the spatiotemporal feature information at different levels and scales, and provides rich feature representation; the feature pyramid structure can be used to fuse spatiotemporal features of different levels, capture target information at different scales, and improve the robustness and generalization ability of feature representation.

[0047] Apply position encoding to introduce spatial position information for one-dimensional feature vectors: After flattening the spatiotemporal feature map into a one-dimensional feature vector, introduce the corresponding spatial position encoding for each eigenvalue; spatial position encoding can use a combination of sine and cosine functions, such as the position encoding method in Transformer; for the i-th eigenvalue, its corresponding spatial position encoding is: ,in , d represents the dimension of position encoding, which is usually set to the length of the one-dimensional feature vector; add the one-dimensional feature vector to the corresponding spatial position encoding to obtain the one-dimensional feature vector after position encoding; spatial position encoding can introduce prior knowledge of spatial position to the one-dimensional feature vector, helping the model to better understand and utilize the spatial relationship of features. The one-dimensional feature vector is used as the input of the fully connected layer: the one-dimensional feature vector represents the global feature representation of the shooting action on the spatiotemporal feature map; the one-dimensional feature vector is passed to the fully connected layer as the input data of the fully connected layer; the fully connected layer further transforms and abstracts the one-dimensional feature vector through matrix multiplication and nonlinear transformation.

[0048] The one-dimensional feature vector is nonlinearly transformed using the Leaky ReLU activation function in the fully connected layer to obtain a sequence of feature vectors describing the trainer's gun holding posture: Construct a fully connected layer: The fully connected layer consists of several neurons, each of which is connected to all neurons in the previous layer; the input dimension of the fully connected layer is set to the length of the one-dimensional feature vector height*width*channels; the output dimension of the fully connected layer is set to the length of the target feature vector, such as 128, 256, etc.; the weight matrix W and bias vector b of the fully connected layer are obtained by random initialization or pre-training; Leaky ReLU is a leaky rectified linear unit activation function with the advantages of sparsity and unsaturated gradient; the Leaky ReLU function is defined as: , where x is the input value; when the input value is greater than 0, the output of the Leaky ReLU function is the same as the input; when the input value is less than 0, the output of the Leaky ReLU function is 0.01 times the input value, introducing a small negative gradient.

[0049] Perform nonlinear transformation on the one-dimensional feature vector: perform matrix multiplication on the one-dimensional feature vector x and the weight matrix W of the fully connected layer, and add the bias vector b to obtain the activation value a of the fully connected layer: , apply the Leaky ReLU activation function to the activation value a, and get the output value y of the fully connected layer: The output value y is a feature vector with a length equal to the output dimension of the fully connected layer, which represents the result of the nonlinear transformation of the shooting action features. For each input shooting action sample, a corresponding feature vector is obtained after the nonlinear transformation of the fully connected layer. The feature vectors of all shooting action samples are arranged in chronological order to form a feature vector sequence. Each feature vector in the feature vector sequence describes the characteristics of the trainee’s gun holding posture at the corresponding moment.

[0050] Figure 3 This is an exemplary flow chart for constructing a BiLSTM-Attention model according to the present application. According to the feature vector sequence, a shooting posture stability evaluation model is established through a bidirectional long short-term memory neural network BiLSTM-Attention based on an attention mechanism to obtain shooting posture stability data. The obtained feature vector sequence is simultaneously input into the input layer of the forward LSTM and backward LSTM of the bidirectional long short-term memory neural network BiLSTM, and the forward and backward temporal dependencies of the shooting posture are obtained in chronological order: a bidirectional long short-term memory neural network BiLSTM is constructed: BiLSTM is composed of LSTM networks in two directions, the forward LSTM and the backward LSTM; the forward LSTM processes the feature vector sequence in chronological order to capture the forward temporal dependencies of the shooting posture; the backward LSTM processes the feature vector sequence in reverse chronological order to capture the backward temporal dependencies of the shooting posture; the number of hidden units of the BiLSTM is set, such as 128, 256, etc., indicating the feature extraction capability of the LSTM; the weight matrix and bias vector of the BiLSTM are initialized, such as using the Xavier initialization method.

[0051] Input the feature vector sequence to the input layer of BiLSTM: The feature vector sequence describing the trainer's gun holding posture is used as the input of BiLSTM; let the feature vector sequence be [, where T is the sequence length, is the feature vector of the i-th time step; the feature vector sequence is simultaneously input to the input layer of the forward LSTM and the backward LSTM; the forward LSTM processes the feature vector sequence in chronological order, that is, ; The backward LSTM processes the feature vector sequence in reverse time order, that is, ; At each time step of BiLSTM, calculate the hidden state: For the forward LSTM, at the i-th time step, according to the current input feature vector and the hidden state at the previous time step , calculate the hidden state of the current time step : , for the backward LSTM, at the i-th time step, according to the current input feature vector and the hidden state at the previous time step , calculate the hidden state of the current time step : , the calculation process of LSTM involves the update of input gate, forget gate, output gate and cell state, and selectively retains and updates long short-term memory through the gating mechanism. At each time step of BiLSTM, the forward and backward hidden states are concatenated: for the i-th time step, the hidden state of the forward LSTM is concatenated and the hidden state of the backward LSTM Concatenate on the feature dimension to get the BiLSTM hidden state of the i-th time step : , the hidden state of BiLSTM At the same time, the temporal dependency of the shooting posture before and after the i-th time step is encoded; the BiLSTM hidden states of all time steps are combined into a hidden state sequence , as the output of BiLSTM.

[0052] At each time step of BiLSTM, the attention mechanism is used to calculate the association weights between the current time step and other time steps, and the attention feature vector of the current time step is obtained by weighted summation: At the i-th time step, the current hidden state is calculated Attention weights between the hidden states at all time steps: Use the attention mechanism to calculate the current hidden state and the hidden state at the jth time step The attention score between : The attention score can be calculated using different methods, such as dot product, concatenation, perceptron, etc. Dot product: , splicing: , Perceptron: ,in, , , v and is a learnable parameter of the attention mechanism. Normalize the attention score to get the attention weight: Use the Softmax function to normalize the attention score between the i-th time step and all time steps to get the attention weight : , attention weight Indicates the degree of association between the i-th time step and the j-th time step. The larger the weight, the stronger the association between the two time steps. Calculate the attention feature vector of the i-th time step: Use the attention weight to perform weighted summation on the hidden states of all time steps to obtain the attention feature vector of the i-th time step : , the attention feature vector Aggregates the relevant information of the i-th time step with other time steps, highlighting the temporal dependencies that have a greater impact on the current time step; concatenates the attention feature vector with the current hidden state to obtain the final time step feature representation: With the current hidden state Concatenate on the feature dimension to get the final feature representation of the i-th time step : , the final feature representation It combines the hidden state information and attention aggregation information of the current time step to provide a more comprehensive and effective temporal feature representation; the final feature representation of all time steps is combined into a feature sequence , as input for the next step of analysis.

[0053] The weight parameters of BiLSTM and attention mechanism are jointly trained through iterative optimization algorithm to construct the shooting posture stability evaluation model BiLSTM-Attention, and the weight parameters of BiLSTM and attention mechanism are initialized: the weight matrix and bias vector of BiLSTM are initialized. Common initialization methods include Xavier initialization and He initialization. Xavier initialization ensures that the variance of input and output of each layer is the same through random sampling, which is suitable for tanh activation function. He initialization multiplies a factor on the basis of Xavier initialization and is suitable for ReLU activation function. The weight matrix and bias vector of attention mechanism are initialized, and small random values ​​or zero initialization can be used. The cross-validation method is used to divide the data set: the feature vector sequence data is randomly divided into K mutually exclusive subsets, each containing approximately the same number of samples. One of the subsets is selected as the validation set each time, and the remaining K-1 subsets are selected as the training set, and the experiment is repeated K times. The performance indicators of K experiments are averaged as the performance evaluation result of the model on the validation set. A part of the samples are separately divided from the original data as the test set to evaluate the generalization performance of the model.

[0054] Use the training set to train the BiLSTM-Attention model: Input the feature vector sequence data of the training set into the BiLSTM-Attention model in chronological order; Calculate the hidden state, attention weight and shooting posture stability evaluation score of the BiLSTM through forward propagation: For the tth time step, the forward LSTM is based on the current input feature vector and the hidden state at the previous time step Calculate the current hidden state ; The backward LSTM is based on the current input feature vector and the hidden state at the next time step Calculate the current hidden state ; Concatenate forward hidden states and the backward hidden state Get the hidden state of BiLSTM ; Use the attention mechanism to calculate the attention weight between the current time step and other time steps ; According to the attention weight, the hidden state of all time steps is weighted and summed to obtain the attention feature vector ; The attention feature vector With the current hidden state Splicing, through the fully connected layer to get the shooting posture stability evaluation score .

[0055] The cross entropy loss function is used to calculate the loss value of the model on the training set: The cross entropy loss function expression is as follows: , calculate the gradient of the loss function to the weight parameters through the back-propagation algorithm: Using the back-propagation algorithm, calculate the gradient of the loss function L to the weight parameters of BiLSTM and attention mechanism Gradient ; Gradient calculation involves the gradient transfer of the loss function to the output layer, fully connected layer, attention layer, and BiLSTM hidden layer; For the gated units inside the LSTM, it is necessary to consider the long-term dependency of the gradient, and techniques such as gradient clipping can be used to avoid gradient explosion or disappearance; Update the weight parameters of the BiLSTM and attention mechanism: , , , , . The exponentially weighted moving average of the maximum absolute value of the gradient is used to adaptively adjust the learning rate of each parameter. Only the first-order moment and the maximum moving average of the gradient need to be calculated, which has high computational efficiency and is suitable for the optimization of large-scale data sets and complex models. Use the test set to evaluate the performance of the trained BiLSTM-Attention model: input the feature vector sequence data of the test set into the trained model to obtain the shooting posture stability evaluation score; calculate the performance indicators on the test set, such as accuracy, precision, recall, F1 value, etc.

[0056] The feature vector sequence data is input into the trained BiLSTM-Attention model in chronological order: The BiLSTM-Attention model extracts the temporal features of the shooting posture through a bidirectional neural network structure and an attention mechanism: The feature vector sequence data is represented as ,in Represents the feature vector of the tth time step, T is the sequence length; input the feature vector sequence X to the forward LSTM and backward LSTM of BiLSTM: the forward LSTM processes the input sequence sequentially to obtain the forward hidden state sequence ; The backward LSTM processes the input sequence in reverse order to obtain the backward hidden state sequence ; The forward hidden state and the backward hidden state Splicing to get the hidden state representation of BiLSTM at the tth time step ; Use the attention mechanism to calculate the BiLSTM hidden state sequence Correlation with shooting posture stability assessment: via attention weight matrix , hidden state and context vector Calculating attention score ; Attention score Perform Softmax normalization to obtain attention weight ; According to the attention weight For hidden state Perform weighted summation to obtain the attention feature vector c.

[0057] The BiLSTM-Attention model outputs the corresponding shooting posture stability evaluation score at each time step: the attention feature vector c is combined with the hidden state of the BiLSTM at the last time step Splicing to form a feature representation for shooting posture stability assessment; input the feature representation to the fully connected layer, and output the shooting posture stability assessment score through the Sigmoid activation function ; It is a continuous value in the range of [0, 1], indicating the stability of the shooting posture. The closer it is to 1, the more stable the posture.

[0058] Generate a shooting posture stability assessment report based on the shooting posture stability assessment scores and corresponding attention weights output by the BiLSTM-Attention model at different time steps: Analyze the shooting posture stability assessment scores at different time steps: For the tth time step, the BiLSTM-Attention model outputs the shooting posture stability assessment score ; will evaluate the score Compare with the preset threshold to determine whether the shooting posture at the tth time step is stable; count the number and proportion of stable and unstable time steps in the entire shooting process to evaluate the overall stability of the shooting posture; analyze the attention weights of different time steps: For the tth time step, the BiLSTM-Attention model outputs the attention weight , indicating the importance of this time step to the evaluation of shooting posture stability; according to the attention weight Sort different time steps to find the key time steps that have the greatest impact on shooting posture stability; combine the eigenvectors of key time steps to analyze the key factors that affect shooting posture stability, such as aiming error, jitter amplitude, etc. Generate a shooting posture stability assessment report: Generate a shooting posture stability assessment report based on comprehensive overall stability assessment, key time step analysis and other information; the report includes: overall shooting posture stability score, ratio of stable to unstable time steps, characteristic analysis of key time steps, etc.; based on the assessment report, provide trainees with targeted shooting posture improvement suggestions, such as adjusting the aiming method, controlling breathing rhythm, etc.; the assessment report is presented visually, including shooting posture stability score curve, attention weight distribution diagram, etc., for intuitive understanding and analysis.

[0059] Figure 4 This is an exemplary flow chart for generating aiming time data according to the present application, using a recurrent neural network model based on an attention mechanism to analyze the collected shooting action images, obtain the movement trajectory and dwell time of the trainer's line of sight between the crosshairs and the target, and generate aiming trajectory and aiming time data; the collected shooting action images are input into the recurrent neural network model based on the attention mechanism: continuous action images during the shooting process are collected, each image represents a snapshot of the shooting posture at a time step; the shooting action images are arranged in chronological order to form an image sequence as the input of the recurrent neural network model; the recurrent neural network model includes an encoder and a decoder: the encoder adopts a bidirectional long short-term memory neural network (BiLSTM) structure to extract the temporal features of the image sequence; the decoder adopts an attention mechanism (Attention Mechanism) and a long short-term memory neural network (LSTM) structure to generate an aiming trajectory coordinate sequence.

[0060] The encoder receives the shooting action image, extracts the temporal features through BiLSTM, and obtains the encoded feature sequence: the shooting action image is represented as a feature vector sequence ,in Represents the image feature vector at the tth time step; the encoder adopts a BiLSTM structure. At each time step t, a bidirectional LSTM unit is used to extract the forward and backward temporal dependencies of the image sequence respectively: the forward LSTM unit propagates forward in time, and according to the input of the current time step and the hidden state at the previous time step , calculate the forward hidden state of the current time step ; The backward LSTM unit propagates backwards in time, according to the input of the current time step and the hidden state at the next time step , calculate the backward hidden state of the current time step ; The forward hidden state and the backward hidden state Concatenate on the feature dimension to get the encoded feature vector of the current time step : , which is composed of the encoded feature vector of each time step into the encoded feature sequence .

[0061] The decoder receives the encoded feature sequence and the hidden state of the previous time step, and calculates the attention feature vector of the current time step through the attention mechanism: at each time step t, the decoder receives the hidden state of the previous time step and the encoded feature sequence E; through the attention mechanism, the attention weights of the current decoding time step t and each time step in the encoded feature sequence are calculated : The hidden state of the previous time step With each feature vector in the encoded feature sequence E Calculate the attention score: . Perform Softmax normalization on the attention score to obtain the normalized attention weight : According to the attention weight , perform weighted summation on the encoded feature sequence E to obtain the attention feature vector of the current time step : .

[0062] The decoder inputs the attention feature vector and the hidden state of the previous time step into the LSTM unit, updates the hidden state and generates the aiming position coordinates of the current time step: The hidden state at the previous time step Concatenate to form the input vector of the LSTM unit: ; Input vector Input LSTM unit and update the hidden state through the gating mechanism : Forget Gate ; Input gate ; Candidate memory element ; Output gate ; Memory element update: ;Hide status update: ; Update the hidden state Input the fully connected layer to generate the aiming position coordinates of the current time step : The decoder generates a complete sequence of aiming trajectory coordinates: Initialize the decoder's hidden state is the hidden state of the encoder at the last time step; loop through the above steps to generate the corresponding aiming position coordinates at each time step t ; The loop process continues until the decoder outputs a special end mark or reaches the preset maximum generation length; the aiming position coordinates generated by the decoder are arranged in chronological order to form a complete aiming trajectory coordinate sequence .

[0063] According to the aiming trajectory coordinate sequence, calculate the dwell time of the trainer's sight at each aiming position and generate aiming time data: for each coordinate point in the sequence T , select the R coordinate points before and after it to form a local window: define the radius R of the local window, which means selecting R points before and after the current coordinate point; for the tth coordinate point in the sequence T , select R points before and after it: ; The selected 2R+1 points (including the current point) form a local window ; Calculate the weight based on the time interval between coordinate points , the weight decays as the time interval increases: define the time decay factor α to control the decay speed of the weight with the time interval; for the local window Each coordinate point in , calculate its difference with the current point The time interval between : , where Δt is the unit time step. Calculating weights : . Normalize all weights within the local window so that their sum is 1: , perform weighted averaging on the coordinate points in the local window to obtain the smoothed coordinate values : Using weights Perform weighted average on the x-coordinate and y-coordinate in the local window: ; ; Get the smoothed coordinate points , used to replace the original coordinate points Replace each coordinate point in the sequence T with the smoothed coordinate point to obtain the smoothed aiming trajectory coordinate sequence : Traverse each coordinate point in the sequence T , replace it with the corresponding smoothed coordinate point ; Get the smoothed aiming trajectory coordinate sequence ; According to the smoothed aiming trajectory coordinate sequence , calculate the dwell time of each targeting area: traverse each pair of adjacent coordinate points in the sequence T' and ; Calculate the Euclidean distance between two coordinate points : Use the Euclidean distance formula to calculate the distance between two coordinate points: If the distance If the value is less than the preset threshold ε, the coordinate point and Classify into the same targeting area: Define the distance threshold ε to determine whether two coordinate points belong to the same targeting area; if , then the coordinate point and Marked as the same targeting area; if , then the coordinate point and Marked as different aiming areas.

[0064] Count the number of coordinate points in each aiming area , multiplied by the unit time step , get the residence time of the area : Traverse the sequence For each coordinate point in the target area, count the number of coordinate points in each target area ; According to the unit time step and the number of coordinate points Calculate the dwell time of the area : For each targeting area, calculate the arithmetic mean of all coordinate points in the area as the representative coordinate of the area : For the i-th targeting area, calculate the arithmetic mean of the x-coordinates and y-coordinates of all coordinate points in the area: , ;Will As the representative coordinates of the i-th aiming area; generate aiming time data :Representative coordinates of each aiming area and the corresponding residence time Combine into a tuple ; Arrange all the tuples of the targeting area in order to form the targeting time data D; each element Represents the representative coordinates of the i-th aiming area and the time the trainee's sight stays in this area.

[0065] Generate a shooting training effect evaluation report based on the number of shots, shooting time, number of hits, shooting posture stability, aiming trajectory and aiming time. Count the total number of shots N of the trainee in a training course or a training cycle; count the total shooting time T of the trainee in a training course or a training cycle; calculate the average shooting frequency F of the trainee: F = N / T, and include the number of shots N, shooting time T and average shooting frequency F as evaluation indicators in the report; count the total number of hits H of the trainee in a training course or a training cycle; calculate the hit rate P based on the number of shots N and the number of hits H: P = H / N, and include the number of hits H and the hit rate P as evaluation indicators in the report. Evaluate shooting posture stability: Extract the three-dimensional coordinate sequence of key body parts from the shooting posture data, such as the head, shoulders, elbows, wrists, etc.; calculate the position fluctuation range of each key part : ,in, Respectively represent the position sequence of the i-th key part on the three coordinate axes; calculate the average position fluctuation range of the key parts : , where M is the total number of key parts; the average position fluctuation range Included in the report as an evaluation indicator of shooting posture stability; given a shooting posture stability rating (such as excellent, good, average, poor, etc.) based on the preset stability threshold. Analyze aiming trajectory and aiming time: extract the position distribution and transfer sequence of the aiming area from the aiming trajectory coordinate sequence; combine the aiming time data to calculate the trainee's average stay time in each aiming area ,in, is the number of dwell time samples of the ith targeting area, is the jth dwell time sample value; statistics on the trainee's attention distribution in different aiming areas, and generate an aiming heat map; analyze the trainee's aiming strategy and line of sight movement pattern, and extract key aiming behavior features; include the average dwell time, aiming heat map and key aiming behavior features as evaluation indicators in the report.

[0066] Comprehensively evaluate the effectiveness of shooting training: conduct a comprehensive analysis of evaluation indicators such as the number of shots, shooting time, hit rate, shooting posture stability, aiming trajectory and aiming time; compare with the trainee's historical performance, the performance of other trainees in the same group and the standard performance; summarize the trainee's strengths and weaknesses in various indicators and put forward targeted improvement suggestions; give the trainee a comprehensive performance score and grade based on preset evaluation rules.

[0067] Generate a shooting training effect evaluation report: design a report template, define the structure and format of the report; organically integrate the statistical results, analysis conclusions and visual charts of various evaluation indicators into the report template; automatically fill in the report template to generate a complete shooting training effect evaluation report; export the report in PDF, HTML and other formats for easy reference by trainers and coaches.

Claims

1. A method for evaluating the effect of live-fire shooting training, characterized in that: include: Construct a live-fire shooting training evaluation system, which includes a target drone, a shooting device and a data processing device; wherein a camera device is installed on the shooting device; During the training, the number of shots, shooting time and number of hits were collected; Acquire the shooting action images of the trainee through the camera device; The pre-trained convolutional neural network is used to extract features from the shooting action images to obtain a feature vector sequence describing the trainer's gun holding posture; According to the feature vector sequence, a shooting posture stability evaluation model is established through the bidirectional long short-term memory neural network BiLSTM-Attention based on the attention mechanism to obtain the shooting posture stability data; The collected shooting action images are analyzed using a recurrent neural network model based on the attention mechanism to obtain the movement trajectory and dwell time of the trainer's sight between the crosshairs and the target, and generate aiming trajectory and aiming time data; A shooting training effect evaluation report is generated based on the number of shots, shooting time, number of hits, shooting posture stability, aiming trajectory and aiming time.

2. The live-fire shooting training effect evaluation method according to claim 1, characterized in that: The feature vector sequence describing the trainer's gun holding posture is obtained, including: The collected continuous multi-frame shooting action images are constructed into a three-dimensional tensor, wherein the first dimension and the second dimension represent the spatial dimension of the shooting action image, and the third dimension represents the temporal dimension of the shooting action image; The pre-trained 3D convolutional neural network 3D-CNN is used to extract the spatiotemporal features of the 3D tensor to obtain the spatiotemporal feature map of the shooting action; wherein the 3D convolutional neural network includes multiple 3D convolutional layers and 3D pooling layers; Flatten the spatiotemporal feature map into a one-dimensional feature vector as the input of the fully connected layer; The one-dimensional feature vector is nonlinearly transformed using the Leaky ReLU activation function in the fully connected layer to obtain a feature vector sequence describing the trainer's gun holding posture.

3. The live-fire shooting training effect evaluation method according to claim 2, characterized in that: Use the pre-trained 3D convolutional neural network 3D-CNN to extract spatiotemporal features from the 3D tensor, including: The three-dimensional tensor is input into the three-dimensional convolution layer, and the separable convolution kernel in the three-dimensional convolution layer is used to slide in the time and space dimensions to extract the local spatiotemporal features of the shooting action image; wherein the separable convolution kernel includes an attention module, a depth convolution kernel, and a point-by-point convolution kernel; The extracted local spatiotemporal features are input into the three-dimensional pooling layer, and the local spatiotemporal features are downsampled in the time and space dimensions; wherein, the maximum pooling is used in the time dimension to extract the temporal features of the shooting action; and the average pooling is used in the space dimension to extract the regional features of the shooting action; Through the alternating operations and hierarchical structure of multiple 3D convolutional layers and 3D pooling layers of the 3D convolutional neural network, the spatiotemporal features of the shooting action are extracted and fused layer by layer to obtain the spatiotemporal feature map of the shooting action.

4. The live-fire shooting training effect evaluation method according to claim 3 is characterized in that: Extract local spatiotemporal features of shooting action images, including: Use the attention module to learn the dependencies of shooting actions in the time and space dimensions, and generate attention weight maps in the time and space dimensions; The deep convolution kernel is used to slide on the time dimension, and different weights are assigned to the feature maps of different time steps according to the attention weight map on the time dimension to extract the temporal features of the shooting action; The point-by-point convolution kernel is used to slide on the spatial dimension, and different weights are assigned to feature maps at different spatial positions according to the attention weight map on the spatial dimension to extract the regional features of the shooting action. The output feature maps of the deep convolution kernel and the point-by-point convolution kernel are combined to obtain the local spatiotemporal features of the shooting action.

5. The live-fire shooting training effect evaluation method according to claim 4 is characterized in that: Get shooting posture stability data, including: The obtained feature vector sequence is simultaneously input into the input layers of the forward LSTM and backward LSTM of the bidirectional long short-term memory neural network BiLSTM to obtain the forward and backward temporal dependencies of the shooting posture in chronological order; At each time step of BiLSTM, the attention mechanism is used to calculate the association weights between the current time step and other time steps, and the attention feature vector of the current time step is obtained by weighted summation; The weight parameters of BiLSTM and attention mechanism are jointly trained through iterative optimization algorithm to build a BiLSTM-Attention model for shooting posture stability evaluation. Input the feature vector sequence data into the trained BiLSTM-Attention model in chronological order: The BiLSTM-Attention model extracts the temporal features of shooting posture through a bidirectional neural network structure and attention mechanism; The BiLSTM-Attention model outputs the corresponding shooting posture stability evaluation score at each time step; Generate a shooting posture stability evaluation report based on the shooting posture stability evaluation scores and corresponding attention weights output by the BiLSTM-Attention model at different time steps.

6. The live-fire shooting training effect evaluation method according to claim 5, characterized in that: Construct a BiLSTM-Attention model for shooting posture stability evaluation, including: Initialize the weight parameters of BiLSTM and attention mechanism; The cross-validation method is used to divide the feature vector sequence data into training set, validation set and test set; the training set is used for model training, the validation set is used for model performance evaluation, and the test set is used to evaluate the generalization performance of the model; Use the training set to train the BiLSTM-Attention model, and calculate the hidden state, attention weight, and shooting posture stability evaluation score for each time step through forward propagation; The cross entropy loss function is used to calculate the loss value of the BiLSTM-Attention model on the training set; The gradient of the loss function with respect to the weight parameters of BiLSTM and attention mechanism is calculated through the back-propagation algorithm; Update the weight parameters of BiLSTM and attention mechanism.

7. The live-fire shooting training effect evaluation method according to claim 6 is characterized in that: The cross entropy loss function expression is as follows: , where N represents the number of training samples, which means the total number of shooting posture stability data involved in the training; T represents the number of time steps, which means the sequence length of the shooting posture stability data processed by the BiLSTM-Attention model; represents the true stability label of the i-th training sample at the t-th time step; Represents the stability evaluation score of the BiLSTM-Attention model for the i-th training sample at the t-th time step; Represents the weight coefficient of the positive sample at time step t; Represents the hyperparameter of Focal Loss, which is used to adjust the weight of difficult samples; Represents the negative sample weight coefficient, which is used to balance the contribution of positive and negative samples.

8. The live-fire shooting training effect evaluation method according to claim 7 is characterized in that: Update the weight parameters of BiLSTM and attention mechanism through the following expressions: , , , , ,in, Represents the gradient of the current moment t, by calculating the loss function L with respect to the parameters of the previous moment The derivative is obtained; Represents the first-order moment of the gradient, which is used to estimate the mean of the gradient; represents the decay rate of the first-order moment; represents the exponentially weighted moving average of the maximum absolute value of the gradient; Indicates the decay rate of the maximum moving average; represents the corrected first-order moment of gradient; Represents the adaptive learning rate at the current time t; Represents the updated model parameters at the current time t; Represents the exponentially weighted moving average of the maximum absolute value of the gradient at the previous moment; Represents the first moment of the gradient at the previous moment; Represents the initial learning rate, the value range is ; Represents the decay rate of the first-order moment at time t.

9. The live-fire shooting training effect evaluation method according to any one of claims 2 to 8, characterized in that: Generates aiming trajectory and aiming time data, including: The collected shooting action images are input into a recurrent neural network model based on an attention mechanism; the recurrent neural network model includes an encoder and a decoder; the encoder adopts a bidirectional long short-term memory neural network BiLSTM structure, and the decoder adopts an attention mechanism and a long short-term memory neural network LSTM structure; The encoder receives the shooting action image, extracts the temporal features of the shooting action image through BiLSTM, and obtains the encoding feature sequence; wherein, at each time step, BiLSTM uses the bidirectional LSTM unit to extract the forward and backward temporal dependencies of the shooting action image respectively, and concatenates the forward and backward hidden states as the encoding features of the current time step; The decoder receives the encoded feature sequence and the hidden state of the previous time step, and calculates the attention weights of the current time step and each time step in the encoded feature sequence through the attention mechanism to obtain the attention feature vector of the current time step; The decoder inputs the attention feature vector and the hidden state of the previous time step into the LSTM unit, updates the hidden state through the gating mechanism, and generates the aiming position coordinates of the current time step; Repeat the generation of aiming position coordinates until the decoder generates a complete aiming trajectory coordinate sequence, and obtains the movement trajectory of the trainee's sight between the crosshairs and the target; According to the aiming trajectory coordinate sequence, the dwell time of the trainee's sight at each aiming position is calculated to generate aiming time data.

10. The live-fire shooting training effect evaluation method according to claim 9, characterized in that: Generates aiming time data including: For each coordinate point in the aiming trajectory coordinate sequence, multiple coordinate points before and after it are selected, and the weight is calculated according to the time interval between the coordinate points. The selected coordinate points are weighted averaged to obtain the smoothed coordinate value of the current coordinate point. Traversing each coordinate point in the aiming trajectory coordinate sequence, updating its coordinate value to the smoothed coordinate value, and obtaining the aiming trajectory coordinate sequence after smoothing; According to the smoothed aiming trajectory coordinate sequence, the Euclidean distance between two adjacent aiming positions is calculated. When the Euclidean distance is less than a threshold, the adjacent aiming positions are classified as the same aiming area. Count the number of aiming positions in each aiming area, multiply the number of aiming positions by the unit time step, and obtain the dwell time of the trainee's sight in each aiming area; The center coordinates of the aiming area are used as representative coordinates of the corresponding area, and aiming time data including the representative coordinates of the aiming area and the corresponding dwell time are generated.

Images