Civil aviation pilot abnormal behavior identification method based on diffusion large model fine tuning

Abstract

The invention relates to a civil aviation pilot abnormal behavior identification method based on diffusion large model fine tuning, and the method comprises the following steps: S1, carrying a visual perception module on a civil aviation cockpit simulation training platform for capturing abnormal behavior data of a pilot in a driving process, S2, giving a video # imgabs0 # by the visual perception module, the method comprises the following steps: S1, obtaining a potential variable expression # imgabs1 # of a video after processing by a pre-trained VAE encoder, the height and the width being h and w respectively, S3, diffusing the potential variable # imgabs2 # obtained in the S2 to obtain potential noise # imgabs3 #, S4, applying a random mask to the potential variable # imgabs4 # obtained in the S2 to obtain a potential mask # imgabs5 #, S5, carrying out feature splicing on the potential noise # imgabs6 # obtained in the S3 and the potential mask # imgabs7 # obtained in the S4 in a channel dimension, and S5, carrying out feature splicing on the potential noise # imgabs6 # obtained in the S3 and the potential mask # imgabs7 # obtained in the S4 in the channel dimension. Inputting the action classification result into a space-time UNet network to obtain an output result of action classification; the method has the advantage of ensuring effective recognition of behavior categories.

Description

Technical Field

[0001] The invention belongs to the technical field of behavior recognition, and in particular relates to a method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model. Background Art

[0002] Using computer vision technology to accurately and quickly identify the driving behavior of civil aviation pilots in real time is of great practical significance for standardizing pilot driving behavior and ensuring aviation safety operations. Traditional methods use physiological sensors (such as electroencephalograms, electrocardiograms, and eye trackers) to monitor people's physical reactions and physiological indicators to diagnose abnormal behaviors. However, this method is too cumbersome and costly, and affects the naturalness of human-computer interaction, making it difficult to apply in practice. The current mainstream method is based on computer vision, which uses cameras to collect video and image data and uses advanced computer algorithms to process and analyze the data. However, these methods do not take into account the unknown behavior categories faced by the model due to the diversity of pilot behavior in actual application scenarios. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model, which can ensure effective identification of behavior categories.

[0004] The technical solution of the present invention is as follows: The abnormal behavior identification method of civil aviation pilots based on fine-tuning of the diffusion large model includes the following steps S1: A visual perception module is installed on the civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving; S2: Visual Perception Module Given a video , represents the set of real numbers, T Indicates the number of frames contained in the video. H , W are height and width respectively. After being processed by the pre-trained VAE encoder, the latent variable representation of the video is obtained. , the height and width are h , w； S3: The latent variables obtained from S2 Diffusion is performed to obtain the potential noise ; S4: The latent variables obtained from S2 Apply a random mask to get the latent mask ; S5: Potential noise obtained by adding S3 and the latent mask obtained by S4 Feature concatenation is performed in the channel dimension and input into the spatiotemporal UNet network to obtain the output result of action classification.

[0005] Further, in step S3, for the potential variable , add independent and identically distributed Gaussian noise, and obtain the potential noise after diffusion , calculated as follows: ; in, is the standard deviation of Gaussian noise, N represents a normal distribution with a mean of 0 and a variance of I .

[0006] Furthermore, in step S4, the potential variable , use the randomly generated mask m to mask, and then mask out some frames to get the potential mask , calculated as follows: ; The ⊙ represents a multiplication operation.

[0007] Further, the step S5 includes the following sub-steps: S51: Potential noise and latent mask The channel dimension feature concatenation is performed and input into the spatiotemporal UNet network, which is used to extract pure potential feature representation from the original input mixed with noise and mask. The mapping function of the spatiotemporal UNet network is set to , the expression is as follows: ; in, represents the parameterized denoising network, represents a skip connection, and represents the scale factor; S52: Introducing a short-term time adapter after the convolutional layer Conv Block of the spatiotemporal UNet network ST- Adapter , that is, add a linear layer between the upper and lower linear layers of the spatiotemporal UNet network 3D-Conv Convolution is used for short-term time series modeling. The process can be expressed as follows: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express ST-Adapter Any input to the module; S53: Short-term adapter ST-Adapter Then introduce a long-term time adapterLT-Adapter , used to model the global temporal relationship between adjacent frames, the long-term temporal adapter LT-Adapter consists of upper and lower linear layers and a temporal self-attention module Self-Attn, and is added to the short-term temporal adapter together with a temporal attention block ST-Adapter After that, the expression is as follows: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express LT-Adapter Any input to the module; S54: Introduce a space adapter after the long-term time adapter LT-Adapter S-Adapter , space adapter S-Adapter Added together with the Spatial Attention Block, it aims to adapt the spatiotemporal UNet network to the spatial distribution of the target dataset. S-Adapter It consists of a downsampling linear layer with a GELU activation function and an upsampling linear layer, expressed as: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express S-Adapter Any input to the module.

[0008] S55: Fine-tune the spatiotemporal UNet network of step S51-step S54 to learn generalized feature representations, and extract compact video representations through the network middle layer of the spatiotemporal UNet network for subsequent behavior classification tasks, i.e. ,in , , Represent the feature height, width and feature dimension information respectively. The compact video is input to the classifier head consisting of an attention pooling and a fully connected layer. , used to predict the video behavior category label, the expression is as follows: ; in, represents the scale factor, and represent the potential noise and the potential mask respectively; S56: Using cross entropy loss To measure the true label and predicted class labels The difference between is calculated as follows: ; in represents the true value label, represents the predicted action category label, Represents the index of all input videos.

[0009] Furthermore, in steps S2, S51-S53, the convolutional layers ConvBlock, Temporal Attention Block and Spatial Attention Block of the VAE encoder and the spatiotemporal Unet network are all weights of the diffusion model pre-trained on large-scale video data. The above parameters are fixed and only the short-term time adapter is trained. ST-Adapter , Long-term time adapter LT-Adapter And space adapter S- Adapter The learnable parameter part of the structure is sufficient.

[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses a diffusion large model to fine-tune the recognition of abnormal pilot behavior. On the one hand, the powerful feature expression ability of the large model is used to effectively learn the behavior characteristics. On the other hand, by fine-tuning and updating a small number of parameters, the feature expression ability of the large model can be transferred to the field of behavior recognition, thereby ensuring effective recognition of the behavior.

[0011] In summary, the present invention has the advantage of ensuring effective identification of behavior categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] like Figure 1 As shown in FIG. 1 , the abnormal behavior identification method of civil aviation pilots based on fine-tuning of the diffusion large model includes the following steps: S1: A visual perception module (Intel RealSense D455 depth camera) is installed on the civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving, including smoking, using mobile phones, etc. S2: Visual Perception Module Given a video , represents the set of real numbers, T Indicates the number of frames contained in the video. H , W are height and width respectively. After being processed by the pre-trained VAE encoder, the latent variable representation of the video is obtained. , the height and width are h , w ; S3: The latent variables obtained from S2 Diffusion is performed to obtain the potential noise ; S4: The latent variables obtained from S2 Apply a random mask to get the latent mask ; S5: Potential noise obtained by adding S3 and the latent mask obtained by S4 Perform feature concatenation in the channel dimension and input it into the spatiotemporal UNet network to obtain the output result of action classification; Furthermore, in step S3, for the potential variable , add independent and identically distributed Gaussian noise, and obtain the potential noise after diffusion , calculated as follows: ; in, is the standard deviation of Gaussian noise, N represents a normal distribution with a mean of 0 and a variance of I ; Furthermore, in step S4, the potential variable , use the randomly generated mask m to mask, and then mask out some frames to get the potential mask , calculated as follows: ; The ⊙ represents a multiplication operation.

[0015] Furthermore, the step S5 includes the following sub-steps: S51: Potential noise and latent mask The channel dimension feature concatenation is performed and input into the spatiotemporal UNet network, which is used to extract pure potential feature representation from the original input mixed with noise and mask. The mapping function of the spatiotemporal UNet network is set to , the expression is as follows: ; in, represents the parameterized denoising network, represents a skip connection, and represents the scale factor; S52: Introducing a short-term time adapter after the convolutional layer Conv Block of the spatiotemporal UNet network ST- Adapter , that is, add a linear layer between the upper and lower linear layers of the spatiotemporal UNet network 3D-Conv Convolution is used for short-term time series modeling. The process can be expressed as follows: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express ST-Adapter Any input to the module; S53: Short-term adapter ST-Adapter Then introduce a long-term time adapter LT-Adapter , used to model the global temporal relationship between adjacent frames, the long-term temporal adapter LT-Adapter consists of upper and lower linear layers and a temporal self-attention module Self-Attn, and is added to the short-term temporal adapter together with a temporal attention block ST-Adapter After that, the expression is as follows: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express LT-Adapter Any input to the module; S54: Introduce a space adapter after the long-term time adapter LT-Adapter S-Adapter , space adapter S-Adapter Added together with the Spatial Attention Block, it aims to adapt the spatiotemporal UNet network to the spatial distribution of the target dataset. S-Adapter It consists of a downsampling linear layer with a GELU activation function and an upsampling linear layer, expressed as: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express S-Adapter Any input to the module.

[0016] S55: Fine-tune the spatiotemporal UNet network of step S51-step S54 to learn generalized feature representations, and extract compact video representations through the network middle layer of the spatiotemporal UNet network for subsequent behavior classification tasks, i.e. ,in , , Represent the feature height, width and feature dimension information respectively. The compact video is input to the classifier head consisting of an attention pooling and a fully connected layer. , used to predict the video behavior category label, the expression is as follows: ; in, represents the scale factor, and represent the potential noise and the potential mask respectively; S56: Using cross entropy loss To measure the true label and predicted class labels The difference between is calculated as follows: ; in represents the true value label, represents the predicted action category label, Represents the index of all input videos.

[0017] S6: Deploy the behavior recognition model trained in the above steps S2-S5 into the civil aviation cockpit to achieve effective recognition of pilot behavior.

[0018] In the steps S2, S51-S53, the convolutional layer Conv Block, Temporal Attention Block and Spatial Attention Block of the VAE encoder and the spatiotemporal Unet network are all weights of the diffusion model pre-trained on large-scale video data. Some of the above parameters are fixed, and only the short-term time adapter is trained. ST-Adapter , Long-term time adapter LT-Adapter And space adapter S-Adapter The learnable parameter part of the structure is sufficient.

[0019] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model, characterized by: The following steps are included S1: A visual perception module is installed on the civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving; S2: Visual Perception Module Given a video , represents the set of real numbers, T Indicates the number of frames contained in the video. H , W are height and width respectively. After being processed by the pre-trained VAE encoder, the latent variable representation of the video is obtained. , the height and width are h , w； S3: The latent variables obtained from S2 Diffusion is performed to obtain the potential noise ; S4: The latent variables obtained from S2 Apply a random mask to get the latent mask ; S5: Potential noise obtained by adding S3 and the latent mask obtained by S4 Feature concatenation is performed in the channel dimension and input into the spatiotemporal UNet network to obtain the output result of action classification.

2. The method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model according to claim 1 is characterized in that: In step S3, for the latent variable , add independent and identically distributed Gaussian noise, and obtain the potential noise after diffusion , calculated as follows: ; in, is the standard deviation of Gaussian noise, N represents a normal distribution with a mean of 0 and a variance of I .

3. The method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model according to claim 1 is characterized in that: In step S4, the latent variable , use the randomly generated mask m to mask, and then mask out some frames to get the potential mask , calculated as follows: ; The ⊙ represents a multiplication operation.

4. The method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model according to claim 1 is characterized in that: The S5 step includes the following sub-steps: S51: Potential noise and latent mask The channel dimension feature concatenation is performed and input into the spatiotemporal UNet network, which is used to extract pure potential feature representation from the original input mixed with noise and mask. The mapping function of the spatiotemporal UNet network is set to , the expression is as follows: ； in, represents the parameterized denoising network, represents a skip connection, and represents the scale factor; S52: Introducing a short-term time adapter after the convolutional layer Conv Block of the spatiotemporal UNet network ST-Adapter , that is, add a linear layer between the upper and lower linear layers of the spatiotemporal UNet network 3D-Conv Convolution is used for short-term time series modeling. The process can be expressed as follows: ; in , Represent up-sampling and down-sampling linear layers, respectively. X express ST-Adapter Any input to the module; S53: Short-term adapter ST-Adapter Then introduce a long-term time adapter LT-Adapter , used to model the global temporal relationship between adjacent frames, the long-term temporal adapter LT-Adapter consists of upper and lower linear layers and a temporal self-attention module Self-Attn, and is added to the short-term temporal adapter together with a temporal attention block ST-Adapter After that, the expression is as follows: ； in , Represent up-sampling and down-sampling linear layers, respectively. X express LT-Adapter Any input to the module; S54: Introduce a space adapter after the long-term time adapter LT-Adapter S-Adapter , space adapter S- Adapter Added together with the Spatial Attention Block, it aims to adapt the spatiotemporal UNet network to the spatial distribution of the target dataset. S-Adapter It consists of a downsampling linear layer with a GELU activation function and an upsampling linear layer, expressed as: ； in , Represent up-sampling and down-sampling linear layers, respectively. X express S-Adapter Any input to the module; S55: Fine-tune the spatiotemporal UNet network of step S51-step S54 to learn generalized feature representations, and extract compact video representations through the network middle layer of the spatiotemporal UNet network for subsequent behavior classification tasks, i.e. ,in , , Represent the feature height, width, and feature dimension information respectively, and input the compact video into the classifier head consisting of an attention pooling and a fully connected layer. , used to predict the video behavior category label, the expression is as follows: ; in, represents the scale factor, and represent the potential noise and the potential mask respectively; S56: Using cross entropy loss To measure the true label and predicted class labels The difference between is calculated as follows: ;in represents the true value label, represents the predicted action category label, Represents the index of all input videos.

5. The method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model according to claim 4 is characterized in that: In the steps S2, S51-S53, the convolutional layer Conv Block, Temporal Attention Block and Spatial Attention Block of the VAE encoder and the spatiotemporal Unet network are all weights of the diffusion model pre-trained on large-scale video data. Some of the above parameters are fixed, and only the short-term time adapter is trained. ST-Adapter , Long-term time adapter LT-Adapter And space adapter S-Adapter The learnable parameter part of the structure is sufficient.

Images