An unmanned aerial vehicle simulation training method and system based on action recognition

Abstract

The application belongs to the technical field of action recognition, and particularly relates to a method and system for simulating training of a UAV based on action recognition, which comprises the following steps: acquiring and synchronously processing image sequences of a low-frame-rate space stream and a high-frame-rate time stream; generating a fast path inhibition coefficient based on the brightness homogeneity and motion space discreteness of the images, and performing feature gate modulation on the high-frame-rate time stream; extracting features through a double-stream network and combining an environmental noise index for double verification, and finally generating a control instruction. The application adopts an asymmetric double-stream architecture combined with a physical feature gate mechanism, can distinguish between real limb movement and environmental light mutation, avoids the problem of false touch in a complex light environment, reduces the calculation redundancy, and improves the interaction accuracy and robustness of the simulation training system.

Description

Technical Field

[0001] This invention relates to the field of motion recognition technology. More specifically, this invention relates to a method and system for drone simulation training based on motion recognition. Background Technology

[0002] With the application of drone technology in both military and civilian fields, vision-based drone simulation training systems have become key equipment for training professional pilots. These systems primarily capture the operator's body movements in real time using cameras and convert them into control signals for a virtual drone, thus replacing actual drone training. Drone simulation training systems utilize computer vision technology to capture trainees' body movements to control virtual drones, serving as an important means to replace actual training, reduce wear and tear, and improve pilot skills. Existing technologies mainly acquire video streams through spatial positioning equipment or cameras and employ motion recognition algorithms based on deep convolutional neural networks to directly extract and classify features from continuous video frame sequences, thereby identifying specific control commands.

[0003] However, in specific application scenarios of UAV simulation training, the aforementioned existing algorithms have certain shortcomings. First, UAV control actions have a large time-frequency domain span, including both long-term, subtle static gestures and short-duration, rapid tactical maneuvers. Existing algorithms typically use a single sampling rate. If the sampling rate is too high, it generates a large amount of redundant data when processing static gestures, resulting in wasted computing power; if the sampling rate is too low, it cannot capture high-frequency tactical actions, leading to the loss of key command features. Second, training grounds are often accompanied by complex environmental interference such as simulated lightning and flashing lights. Existing deep network models lack an effective mechanism for identifying physical environmental noise, easily misidentifying global illumination changes as action features and amplifying them nonlinearly in the deep layers of the network. This causes the system to be prone to false triggers or recognition errors under strong lighting interference, affecting the accuracy and immersion of training. Summary of the Invention

[0004] To address the technical problem of poor motion recognition performance during drone simulation training, this invention provides solutions in the following aspects.

[0005] In a first aspect, the present invention provides a drone simulation training method based on action recognition, comprising:

[0006] A parallel, synchronized low-frame-rate spatial stream image sequence and a high-frame-rate temporal stream image sequence are acquired. Based on the pixel difference distribution of adjacent images in the high-frame-rate temporal stream image sequence, the brightness homogeneity of any image in the high-frame-rate temporal stream image sequence is obtained. Based on the pixel difference distribution of adjacent images in the high-frame-rate temporal stream image sequence and the brightness homogeneity of the images, the motion spatial dispersion of any image in the high-frame-rate temporal stream image sequence is obtained. Based on the motion spatial dispersion, the fast path suppression coefficient corresponding to each image in the high-frame-rate temporal stream image sequence is obtained, and feature-gated modulation is performed on the high-frame-rate temporal stream image sequence to obtain a high-frame-rate temporal stream input tensor. Based on the low-frame-rate spatial stream image sequence and the high-frame-rate temporal stream input tensor, a comprehensive motion feature vector is obtained through a dual-stream network to obtain an effective motion category. Based on the effective motion category, control commands for driving the UAV simulation training system are obtained. The sampling rate of the high-frame-rate temporal stream image sequence is a preset integer multiple of the sampling rate of the low-frame-rate spatial stream image sequence.

[0007] This invention constructs a parallel architecture of low-frame-rate spatial flow and high-frame-rate temporal flow, ensuring the capture of minute and rapid tactical movements while avoiding redundant computation. It introduces a gating mechanism based on brightness homogeneity and motion space discreteness, calculating the fast path suppression coefficient before the data enters the neural network to modulate the input data. This effectively distinguishes and filters out discrete background noise such as sudden changes in full-screen illumination caused by simulated thunderstorms and light flickering, as well as swaying leaves. This prevents the deep neural network from nonlinearly amplifying and misidentifying environmental noise, improving the accuracy and robustness of action recognition under complex lighting conditions.

[0008] Preferably, the brightness homogeneity of the arbitrary image satisfies the expression:

[0009] ;

[0010] In the formula, This indicates the homogeneity of brightness in the t-th image; This represents the number of pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image and the (t-1)-th image; Represents the absolute value function; It represents a tiny positive value.

[0011] This invention utilizes the physical principle that changes in illumination typically exhibit global unidirectionality while object motion exhibits local alternation of positive and negative light. It can accurately identify illumination interference frames, providing a reliable physical criterion for subsequently cutting off noise signal transmission and avoiding false triggering caused by changes in simulated ambient lighting.

[0012] Preferably, obtaining the motion space discreteness of the arbitrary image includes:

[0013] Get the The image and the first In the frame difference map of the first image, pixels whose changes exceed a preset change threshold are considered as the first... The active pixels at time t, obtaining the first The set of active pixels at each moment; the motion spatial discreteness of any image in the high frame rate time-stream image sequence is the set of active pixels at each moment; The weighted average distance of each active pixel in the set of active pixels at time t is relative to the weighted centroid; the weighted centroid is obtained by weighting the absolute value of the grayscale change of the active pixels in the frame difference map as weights.

[0014] Preferably, the motion space discreteness of the arbitrary image satisfies the expression:

[0015] ;

[0016] In the formula, This represents the motion space discreteness of the t-th image; This represents the number of active pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image; Let x and y represent the x and y coordinates of the i-th pixel in the t-th image; The weighted centroid coordinates of the set of active pixels in the t-th image are represented. Indicates the brightness coefficient; This indicates the homogeneity of brightness in the t-th image; Represents the absolute value function; The symbol represents the Euclidean distance formula.

[0017] This invention establishes a composite physical index that integrates illumination stability and spatial cohesion by introducing brightness homogeneity as an adjustment term in the calculation of motion space dispersion. When the ambient illumination is extremely unstable, this formula automatically amplifies the dispersion index, making the system's judgment conditions more stringent. This proactively tightens the signal passage threshold in harsh simulated training environments, preventing erroneous training instructions from being generated when the signal-to-noise ratio is low.

[0018] Preferably, the first The weighted centroid coordinates of the set of active pixels in an image satisfy the expression:

[0019] ;

[0020] In the formula, The weighted centroid coordinates of the set of active pixels in the t-th image are represented. This represents the number of active pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image; Let x and y represent the x and y coordinates of the i-th pixel in the t-th image; This represents the absolute value function.

[0021] This invention further refines the calculation logic of the weighted centroid of active pixels. By using the absolute value of grayscale change in the frame difference map as weight, it mathematically ensures that the pixel with stronger motion energy contributes more to the centroid position. This makes the calculated centroid able to resist the influence of low-intensity background noise, ensuring that the discrete features calculated in subsequent calculations truly reflect the degree of aggregation of the main control gestures, and improving the purity of feature extraction.

[0022] Preferably, the fast path suppression coefficient satisfies the expression:

[0023] ;

[0024] In the formula, Represents the fast path suppression coefficient of the t-th image; Indicates the regulating factor; This represents the motion space discreteness of the t-th image; Indicates the critical threshold; This represents the natural exponential function.

[0025] Preferably, the feature-gated modulation includes:

[0026] The fast path suppression coefficient is dimension-expanded to obtain a suppression tensor with the same dimension as the image tensor corresponding to the high frame rate time stream image sequence; the suppression tensor is then multiplied element-wise with the image tensor corresponding to the high frame rate time stream image sequence.

[0027] Preferably, the dual-stream network includes:

[0028] The system comprises a spatial flow network and a temporal flow network. The spatial flow network is used to process the low frame rate spatial flow image sequence to extract static appearance feature vectors, and the temporal flow network is used to process the high frame rate temporal flow input tensor to extract dynamic temporal feature vectors. The combined action feature vector is the fusion result of the static appearance feature vector and the dynamic temporal feature vector.

[0029] This invention employs a dual-stream network architecture that combines spatial flow with a physically gated temporal flow. The spatial flow is responsible for extracting static semantic features such as hand gestures and postures, while the temporal flow focuses on capturing high-frequency dynamic features after noise reduction. This architecture leverages the powerful feature representation capabilities of deep learning while incorporating the anti-interference capabilities of traditional physical features, ensuring the system's reliability in understanding both static gesture commands and dynamic tactical commands.

[0030] Preferably, the acquisition of the effective action category includes:

[0031] The comprehensive action feature vector is input into a fully connected layer to obtain probability values ​​corresponding to all preset action categories. The action category corresponding to the highest probability value is recorded as the current predicted action category. The highest probability value is recorded as the action classification confidence score, and the value obtained by subtracting the fast path suppression coefficient from 1 is recorded as the environmental noise index. When the action classification confidence score is greater than a preset classification confidence threshold and the environmental noise index is less than a preset environmental stability threshold, the current predicted action category is determined to be the effective action category.

[0032] Secondly, the present invention provides a drone simulation training system based on motion recognition, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the aforementioned drone simulation training method based on motion recognition is implemented.

[0033] By adopting the above technical solution, a computer program is generated from the above-mentioned drone simulation training method based on action recognition and stored in the memory so that it can be loaded and executed by the processor. In this way, a terminal device can be made based on the memory and the processor for convenient use.

[0034] The beneficial effects of this invention are as follows: This invention solves the problem of action temporal heterogeneity by capturing dynamic mutations and static representations at high and low frame rates respectively; it constructs suppression coefficients using brightness homogeneity and motion space dispersion, and performs gated modulation on the high frame rate stream before feature extraction to filter out illumination flicker and environmental clutter interference from a physical level; combined with a dual verification strategy, it solves the problem of false touches caused by noise enhancement in deep neural networks under complex simulation environments, and improves the interactive robustness of the training system. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an action recognition-based drone simulation training method according to the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the homogeneity of brightness;

[0037] Figure 3This is a schematic diagram illustrating the dispersion of the motion space;

[0038] Figure 4 This is a schematic diagram illustrating the fast path suppression coefficient. Detailed Implementation

[0039] This invention discloses a drone simulation training method based on action recognition, referring to... Figure 1 This includes steps S1-S4:

[0040] S1: A spatial positioning perception layer built on a spatial positioning device acquires the original video stream data in the training scene; based on the preset sampling rate of the low frame rate spatial stream and the sampling rate of the high frame rate temporal stream, parallel and synchronized low frame rate spatial stream image sequences and high frame rate temporal stream image sequences are acquired from the original video stream data.

[0041] It should be noted that the action space of UAV simulation training has a large frequency domain span, which differs significantly from conventional human posture recognition scenarios. In conventional scenarios, human movements are usually continuous and uniform in speed. However, UAV training scenarios include both subtle static gestures when disassembling precision components, such as prolonged pinching motions, and short-term operations during tactical flight, such as moving the joystick or cutting off power in an emergency. If only a single sampling rate is used, a high frame rate will result in a large number of redundant frames for static gesture data, causing computing power to be wasted on processing repetitive information; while a low frame rate will lead to the loss of features of high-frequency tactical commands due to the limitations of the Nyquist sampling theorem. Therefore, considering the temporal heterogeneity of UAV training actions, this invention constructs a dual-stream channel with asymmetric temporal resolution to separately lock static representations and dynamic abrupt changes.

[0042] Specifically, the spatial positioning perception layer, built based on spatial positioning devices, acquires raw video stream data within the training scene, including:

[0043] A spatial positioning perception layer is constructed using spatial positioning devices deployed around the training ground to collect raw video stream data from the training scene in real time. For example, the spatial positioning devices consist of 26 motion capture cameras. The data processing layer server receives the raw video stream data and decomposes it into two parallel input queues: a low frame rate spatial stream and a high frame rate temporal stream. It should be noted that the low frame rate spatial stream and the high frame rate temporal stream are used to capture static features and subtle motion changes in the scene, respectively.

[0044] It should be noted that, in order to ensure that the parallel data stream can cover high-frequency actions while maintaining the controllability of data volume, this invention establishes a multiplication relationship between the two. By using the low frame rate spatial stream as the baseline and the high frame rate temporal stream as an integer multiple of the baseline, it can be guaranteed that during subsequent feature fusion, multiple high-frequency frames can be strictly aligned within the time window of a low-frequency frame, avoiding timing misalignment.

[0045] Preferably, based on a preset sampling rate for the low frame rate spatial stream and a sampling rate for the high frame rate temporal stream, parallel and synchronized low frame rate spatial stream image sequences and high frame rate temporal stream image sequences are obtained from the original video stream data, including:

[0046] Set sampling rate multiplication factor For example, .like The value is too large, for example This can lead to excessively large amounts of high frame rate streaming data, exceeding the real-time computing load of the processing nodes; if The value is too small, for example If the time resolution is not significantly improved, it will not be able to effectively capture the tiny, rapid tossing operations in polarity checks.

[0047] The sampling rate of the low frame rate spatial stream is multiplied by a sampling rate multiplier, and this product is used as the sampling rate of the high frame rate temporal stream. For example, the sampling rate of the low frame rate spatial stream is 16 fps, and the sampling rate of the high frame rate temporal stream is 64 fps.

[0048] The original video stream data was acquired by using the sampling rate of the low frame rate spatial stream and the sampling rate of the high frame rate temporal stream, respectively, to obtain the low frame rate spatial stream image sequence and the high frame rate temporal stream image sequence.

[0049] Thus, parallel and synchronized low-frame-rate spatial stream image sequences and high-frame-rate temporal stream image sequences were obtained.

[0050] S2: Based on the difference features between adjacent frames of a high frame rate time-stream image sequence, obtain the brightness homogeneity of any image in the high frame rate time-stream image sequence; based on the difference distribution of pixels in adjacent images of the high frame rate time-stream image sequence and the brightness homogeneity of the images, obtain the motion space dispersion of any image in the high frame rate time-stream image sequence; based on the motion space dispersion of the image and the nonlinear gating mapping, obtain the fast path suppression coefficient of the image.

[0051] It should be noted that drone training grounds are typically located in semi-open environments or environments filled with electrical equipment, which differ from standard laboratory environments. Lighting conditions are unstable and the background is complex. For example, the flickering of lights simulating thunderstorms, instructors switching lights on and off, or screen flickering can all produce pixel-level intensity changes in the image. These changes are easily confused with real limb movements numerically, but from a physical distribution perspective, sudden changes in lighting are usually globally consistent, with the entire image brightening or darkening simultaneously, while real limb movements only cause alternating positive and negative changes in pixel grayscale in local areas. Therefore, considering the physical homogeneity of lighting interference, this invention distinguishes pseudo-motion signals caused by sudden changes in lighting by statistically analyzing the ratio of the algebraic sum to the absolute sum of the global differences in high frame rate time-stream image sequences.

[0052] It should be noted that for high frame rate time-stream image sequences, pixel-by-pixel differencing is performed between any image and the previous frame. When a sudden change in global illumination occurs in the scene, the direction of grayscale value change for each pixel is consistent. In this case, the algebraic sum of all differences is large and close to the sum of absolute values. Conversely, when there is only limb movement in the scene, the movement causes part of the background to be occluded and darkened, while another part of the background is exposed and brightened. The direction of pixel change is chaotic, and the algebraic sum of positive and negative values ​​is small after cancellation. Therefore, this invention obtains the brightness homogeneity characterizing the degree of global illumination interference based on the ratio of the algebraic sum of differences to the sum of absolute differences in high frame rate time-stream image sequences.

[0053] Specifically, based on the difference features between adjacent frames in a high frame rate time-stream image sequence, the brightness homogeneity of any image in the high frame rate time-stream image sequence is obtained, including:

[0054] The brightness homogeneity of any image satisfies the expression:

[0055] ;

[0056] In the formula, This indicates the homogeneity of brightness in the t-th image; This represents the number of pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image and the (t-1)-th image; Represents the absolute value function; This represents a small positive value, used to avoid a denominator of 0. For example, .

[0057] In the formula, The absolute value of the algebraic sum of pixel grayscale differences reflects the degree of homogeneity of the overall brightness change of the t-th image. The larger the value, the more consistent the direction of grayscale change of the t-th image, thus reflecting the greater the homogeneity of the image's brightness. This represents the sum of absolute differences in pixel gray levels, reflecting the total amount of gray level changes in the image. The larger this value is, the greater the change in the t-th image is relative to the (t-1)-th image. This represents the proportion of globally consistent illumination changes in the total energy of change. The larger this value, the higher the probability of abrupt changes in illumination.

[0058] It should be noted that, as Figure 2 This diagram illustrates the homogeneity of brightness. The horizontal axis represents the image number, and the vertical axis represents the homogeneity of brightness. The illumination interference range reflects the homogeneity characteristics of global illumination abrupt changes.

[0059] It should be noted that training backgrounds often contain environmental clutter such as swaying leaves and distant people walking. This type of interference exhibits significant spatial dispersion, meaning that the points of change are scattered across multiple disconnected regions of the image. In contrast, effective drone control gestures, such as raising a hand, typically focus on the human hand or arm, resulting in highly concentrated spatial points of change. Furthermore, when illumination interference is strong, the overall signal-to-noise ratio of the image decreases, thus reducing the tolerance for discrete noise. Therefore, this invention calculates the spatial dispersion of the current motion signal by measuring the weighted distance of active pixels relative to the centroid and introducing an illumination index as an adjustment term, thereby distinguishing effective gestures from environmental clutter.

[0060] Preferably, the motion spatial dispersion of any image in a high frame rate time-stream image sequence is obtained based on the difference distribution of pixels in adjacent images of the high frame rate time-stream image sequence and the homogeneity of image brightness, including:

[0061] Obtain the frame difference map of the t-th image and the (t-1)-th image. Pixels in the frame difference map whose change exceeds a preset change threshold are recorded as active pixels at time t, and the set of active pixels at time t is obtained. For example, the image grayscale level is 0-255, and the preset change threshold is 20.

[0062] It should be noted that the intensity of change among different pixels in the set of active pixels is not uniform. If only the geometric center is used as the centroid, edge noise will have the same impact on the centroid position as the core region, thus skewing the centroid. Therefore, this invention weights different active pixels, making the centroid more inclined towards the more dynamic core, thereby providing a more accurate reference benchmark for subsequent discreteness calculations.

[0063] Obtain the weighted centroid coordinates of the active pixel set, where the weight used in the weighting calculation is the absolute value of the grayscale change of each active pixel in the frame difference map.

[0064] The weighted centroid coordinates of the set of active pixels in any image satisfy the expression:

[0065] ;

[0066] In the formula, The weighted centroid coordinates of the set of active pixels in the t-th image are represented. This represents the number of active pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image; Let x and y represent the x and y coordinates of the i-th pixel in the t-th image; This represents the absolute value function.

[0067] A brightness coefficient is set to control the weight of the influence of brightness features on the calculation of motion space dispersion. For example, the brightness coefficient is set to... .

[0068] The motion space discreteness of any image satisfies the expression:

[0069] ;

[0070] In the formula, This represents the motion space discreteness of the t-th image; This represents the number of active pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image; Let x and y represent the x and y coordinates of the i-th pixel in the t-th image; The weighted centroid coordinates of the set of active pixels in the t-th image are represented. Indicates the brightness coefficient; This indicates the homogeneity of brightness in the t-th image; Represents the absolute value function; The symbol represents the Euclidean distance formula.

[0071] In the formula, It represents the weighted discrete moment of the i-th pixel in the t-th image, reflecting the degree and intensity of the deviation of that point from the centroid; This represents the total spatial discrete moments of all pixels in the t-th image; This represents the total amount of grayscale changes in the t-th image; This indicates that the total spatial discrete moments of the t-th image are normalized by the total amount of grayscale changes. This indicates the illumination adjustment item, which is used to amplify the dispersion index when the illumination is unstable, thereby tightening the threshold for noise filtering.

[0072] It should be noted that, as Figure 3This is a schematic diagram of motion space dispersion. The horizontal axis represents the frame number, and the vertical axis represents the motion space dispersion. The effective motion range and the discrete noise range reflect the spatial dispersion of local focused motion and random noise, respectively, and are used to distinguish effective motion from environmental clutter.

[0073] It should be noted that, in order to achieve noise reduction control, the aforementioned physical characteristics need to be converted into a normalized control gain. When excessive motion dispersion or high brightness homogeneity is detected, the system should determine the corresponding image as an invalid frame and output a coefficient close to zero to cut off signal transmission. Conversely, when the signal is focused and the illumination is stable, a coefficient close to 1 is output. Therefore, this invention constructs a soft threshold gating function, dynamically generating fast path suppression coefficients based on the deviation between the dispersion and the threshold.

[0074] Preferably, the fast path suppression coefficient of an image is obtained based on the motion space discreteness and nonlinear gating mapping of the image, including:

[0075] Set the adjustment factor and critical threshold. The adjustment factor controls the steepness of the suppression curve's edge; the critical threshold defines a baseline for ambient noise. For example, the adjustment factor is set to 2, and the critical threshold is set to 50.

[0076] The fast path suppression coefficient of any image satisfies the expression:

[0077] ;

[0078] In the formula, Represents the fast path suppression coefficient of the t-th image; Indicates the regulating factor; This represents the motion space discreteness of the t-th image; Indicates the critical threshold; This represents the natural exponential function.

[0079] In the formula, This represents the deviation of the motion space dispersion of the t-th image from the critical threshold. The larger this value is, the greater the degree to which the motion space dispersion exceeds the critical threshold, and the more it needs to be suppressed. This represents the nonlinear error amplification term; This means that by utilizing the characteristics of the Sigmoid function, the output value can quickly approach 0 when the dispersion of the motion space exceeds a critical threshold.

[0080] It should be noted that, as Figure 4 This diagram illustrates the fast path suppression coefficient, demonstrating the effect of feature-gated modulation and proving the algorithm's ability to suppress environmental interference. The horizontal axis represents the frame number, and the vertical axis represents the fast path suppression coefficient, which is close to 1 in the effective action range, preserving features, and close to 0 in the interference range, suppressing noise.

[0081] Thus, the fast path suppression coefficients of each image in the high frame rate time-stream image sequence were obtained.

[0082] S3: Based on the fast path suppression coefficient, feature-gated modulation is performed on the high frame rate time stream image sequence to obtain the modulated high frame rate time stream input tensor; based on the low frame rate spatial stream image sequence and the modulated high frame rate time stream input tensor, static appearance feature vector and dynamic temporal feature vector are obtained through a dual-stream network respectively, and then concatenated to obtain a comprehensive action feature vector.

[0083] It should be noted that while deep neural networks possess powerful feature extraction capabilities, they are essentially nonlinear mappings that amplify input data layer by layer. If the original input contains high-intensity environmental noise, such as flickering light, this noise will be mistakenly identified as a texture feature in deeper layers of the network and retained or even amplified, ultimately leading to recognition errors. Therefore, to purify the feature stream at its source, this invention uses a fast path suppression coefficient to amplitude modulate high frame rate time-stream image sequences before feeding data into the network, allowing only high-confidence motion signals to pass through.

[0084] It should be noted that the fast path suppression coefficient is expanded to the same dimension as the input image tensor, and element-wise multiplication is performed. When there is significant environmental interference, the suppression coefficient approaches 0, at which point the pixel values ​​of the input image are suppressed as a whole, and the network receives an almost entirely black image, thus avoiding noise activating the convolution kernel; when the environment is stable, the suppression coefficient approaches 1, and the original image information is completely preserved.

[0085] Specifically, feature-gated modulation is performed on high frame rate time-stream image sequences based on the fast path suppression coefficient to obtain the modulated input tensor, including:

[0086] Feature gating is performed before the high frame rate time-stream image sequence is input into the first convolutional layer of the time-stream network.

[0087] The input tensor after modulation of any image satisfies the expression:

[0088] ;

[0089] In the formula, This represents the input tensor after the t-th image is modulated; Let represent the original input tensor of the t-th image; Represents the fast path suppression coefficient of the t-th image; This represents the element-wise dot product symbol.

[0090] In the formula, This indicates that the tensor is weighted element-wise using the fast path suppression coefficient, and this indicates that the overall gain of the t-th image is adjusted.

[0091] Preferably, based on the low frame rate spatial stream image sequence and the modulated input tensor, static appearance feature vectors and dynamic temporal feature vectors are obtained through a two-stream network, and then concatenated to obtain a comprehensive action feature vector, including:

[0092] We utilize spatial flow networks to process low frame rate spatial flow image sequences and extract static appearance feature vectors.

[0093] High frame rate time-stream image sequences are processed using an adaptively modulated time-stream network to extract dynamic temporal feature vectors.

[0094] A multimodal fusion layer is used to concatenate static appearance feature vectors and dynamic temporal feature vectors to generate a comprehensive action feature vector.

[0095] Thus, a comprehensive action feature vector containing robust spatiotemporal information has been obtained.

[0096] S4: Based on the comprehensive action feature vector and the fast path suppression coefficient, obtain the valid action judgment result that has been doubly verified, and combine it with the preset business interaction logic to obtain the control command to drive the UAV simulation training system.

[0097] It should be noted that virtual simulation training systems require high accuracy in interaction; accidental touches can disrupt the trainee's immersion and training effectiveness. Since the output of a neural network is a probability distribution, even in the absence of action, noise may cause the highest probability for a particular category. Therefore, this invention introduces a dual verification mechanism, requiring not only high classification probabilities from the network output but also that environmental stability indicators remain within a safe range, thus making the stability of the physical environment a necessary condition for triggering commands.

[0098] Specifically, based on the comprehensive action feature vector and the fast path suppression coefficient, the system obtains a valid action judgment result that has undergone dual verification, and combines it with the preset business interaction logic to obtain the control commands that drive the UAV simulation training system, including:

[0099] The comprehensive action feature vector is input into the fully connected classification layer, and the probability distribution of all preset action categories is obtained using the softmax function. The action category corresponding to the maximum probability value in the probability distribution is recorded as the current predicted action, and the maximum probability value is recorded as the action classification confidence.

[0100] The environmental noise index of the image is obtained by subtracting the fast path suppression coefficient of any image from 1.

[0101] Set a classification confidence threshold and an environmental stability threshold. If the action classification confidence is greater than the classification confidence threshold and the environmental noise index is less than the environmental stability threshold, the current predicted action is considered valid, and the valid action determination result is output. For example, the classification confidence threshold is set to 0.85, and the environmental stability threshold is set to 0.2.

[0102] Based on the action category determined by the valid action judgment result, a preset instruction mapping table is matched to generate the corresponding control instruction.

[0103] For example, when the effective action judgment result is "lift collective distance", a backup check command is generated to drive the rotor blade angle of the virtual drone model to deflect; when the effective action judgment result is "five fingers open to grasp" and the holding time exceeds the preset duration, a disassembly command is generated to drive the virtual hand to adsorb parts, for example, the preset duration is 1.5 seconds; when the effective action judgment result is "arm rotates significantly", a lifting command is generated to drive the virtual crane arm to execute the deployment logic.

[0104] At this point, the control commands for driving the drone simulation training system were obtained.

[0105] This invention also discloses a drone simulation training system based on motion recognition, including a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a drone simulation training method based on motion recognition according to the present invention.

[0106] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0107] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A drone simulation training method based on action recognition, characterized in that, include: Acquire parallel and synchronized low-frame-rate spatial stream image sequences and high-frame-rate temporal stream image sequences; Based on the pixel difference distribution of adjacent images in the high frame rate time-stream image sequence, the brightness homogeneity of any image in the high frame rate time-stream image sequence is obtained; based on the pixel difference distribution of adjacent images in the high frame rate time-stream image sequence and the brightness homogeneity of the images, the motion space dispersion of any image in the high frame rate time-stream image sequence is obtained. Based on the motion space discreteness, the fast path suppression coefficients corresponding to each image in the high frame rate time stream image sequence are obtained, and feature-gated modulation is performed on the high frame rate time stream image sequence to obtain the high frame rate time stream input tensor. Based on the low frame rate spatial stream image sequence and the high frame rate temporal stream input tensor, a comprehensive action feature vector is obtained through a dual-stream network to obtain the effective action category. Based on the effective action category, control commands for driving the UAV simulation training system are obtained; the sampling rate of the high frame rate temporal stream image sequence is a preset integer multiple of the sampling rate of the low frame rate spatial stream image sequence; The brightness homogeneity of any image satisfies the expression: In the formula, This indicates the homogeneity of brightness in the t-th image; This represents the number of pixels in the t-th image; , This represents the grayscale value of the i-th pixel in the t-th image and the (t-1)-th image; Represents the absolute value function; Indicates a tiny positive value; The motion space discreteness of any image satisfies the expression: In the formula, This represents the motion space discreteness of the t-th image; This represents the number of active pixels in the t-th image; Let x and y represent the x and y coordinates of the i-th pixel in the t-th image; The weighted centroid coordinates of the set of active pixels in the t-th image are represented. Indicates the brightness coefficient; This indicates the homogeneity of brightness in the t-th image; Symbols representing the Euclidean distance formula; No. The weighted centroid coordinates of the set of active pixels in an image satisfy the expression: In the formula, The weighted centroid coordinates of the set of active pixels in the t-th image are represented. The fast path inhibition coefficient satisfies the following expression: In the formula, Represents the fast path suppression coefficient of the t-th image; Indicates the regulating factor; Indicates the critical threshold; This represents the natural exponential function.

2. The UAV simulation training method based on action recognition according to claim 1, characterized in that, Obtaining the motion space discreteness of the arbitrary image includes: Get the The image and the first In the frame difference map of the first image, pixels whose changes exceed a preset change threshold are considered as the first... The active pixels at time t, obtaining the first The set of active pixels at each moment; the motion spatial discreteness of any image in the high frame rate time-stream image sequence is the set of active pixels at each moment; The weighted average distance of each active pixel in the set of active pixels at time t is relative to the weighted centroid; the weighted centroid is obtained by weighting the absolute value of the grayscale change of the active pixels in the frame difference map as weights.

3. The UAV simulation training method based on action recognition according to claim 1, characterized in that, The feature-gated modulation includes: The fast path suppression coefficient is dimension-expanded to obtain a suppression tensor with the same dimension as the image tensor corresponding to the high frame rate time stream image sequence; the suppression tensor is then multiplied element-wise with the image tensor corresponding to the high frame rate time stream image sequence.

4. The UAV simulation training method based on action recognition according to claim 1, characterized in that, The dual-stream network includes: The system comprises a spatial flow network and a temporal flow network. The spatial flow network is used to process the low frame rate spatial flow image sequence to extract static appearance feature vectors, and the temporal flow network is used to process the high frame rate temporal flow input tensor to extract dynamic temporal feature vectors. The combined action feature vector is the fusion result of the static appearance feature vector and the dynamic temporal feature vector.

5. The UAV simulation training method based on action recognition according to claim 1, characterized in that, The acquisition of the valid action category includes: The comprehensive action feature vector is input into a fully connected layer to obtain probability values ​​corresponding to all preset action categories. The action category corresponding to the highest probability value is recorded as the current predicted action category. The highest probability value is recorded as the action classification confidence score, and the value obtained by subtracting the fast path suppression coefficient from 1 is recorded as the environmental noise index. When the action classification confidence score is greater than a preset classification confidence threshold and the environmental noise index is less than a preset environmental stability threshold, the current predicted action category is determined to be the effective action category.

6. A drone simulation training system based on motion recognition, characterized in that, include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement a drone simulation training method based on action recognition according to any one of claims 1-5.

Images