Abnormal behavior recognition method for civil aviation pilots based on fine-tuning of diffusion large model

Abstract

The present invention relates to a method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model, comprising the following steps S1: a visual perception module is installed on a civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving; S2: the visual perception module is given a video, and after being processed by a pre-trained VAE encoder, a latent variable representation of the video is obtained, with a height and a width respectively. h , w， S3: diffuse the latent variable obtained in S2 to obtain latent noise, S4: apply a random mask to the latent variable obtained in S2 to obtain a latent mask, S5: perform feature concatenation on the latent noise obtained in S3 and the latent mask obtained in S4 in the channel dimension, and input them into the spatiotemporal UNet network to obtain the output result of action classification; the present invention has the advantage of ensuring effective recognition of behavior categories.

Description

Technical Field

[0001] The invention belongs to the technical field of behavior recognition, and in particular relates to a method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model. Background Art

[0002] Using computer vision technology to accurately and quickly identify the driving behavior of civil aviation pilots in real time is of great practical significance for standardizing pilot driving behavior and ensuring aviation safety operations. Traditional methods use physiological sensors (such as electroencephalograms, electrocardiograms, and eye trackers) to monitor people's physical reactions and physiological indicators to diagnose abnormal behaviors. However, this method is too cumbersome and costly, and affects the naturalness of human-computer interaction, making it difficult to apply in practice. The current mainstream method is based on computer vision, which uses cameras to collect video and image data and uses advanced computer algorithms to process and analyze the data. However, these methods do not take into account the unknown behavior categories faced by the model due to the diversity of pilot behavior in actual application scenarios. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model, which can ensure effective identification of behavior categories.

[0004] The technical solution of the present invention is as follows:

[0005] The abnormal behavior identification method of civil aviation pilots based on fine-tuning of the diffusion large model includes the following steps

[0006] S1: A visual perception module is installed on the civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving;

[0007] S2: Visual Perception Module Given a video , represents the set of real numbers, T Indicates the number of frames contained in the video. H , W are height and width respectively. After being processed by the pre-trained VAE encoder, the latent variable representation of the video is obtained. , the height and width are h , w；

[0008] S3: The latent variables obtained from S2 Diffusion is performed to obtain the potential noise ;

[0009] S4: The latent variables obtained from S2 Apply a random mask to get the latent mask ;

[0010] S5: Potential noise obtained by adding S3 and the latent mask obtained by S4 Feature concatenation is performed in the channel dimension and input into the spatiotemporal UNet network to obtain the output result of action classification.

[0011] Further, in step S3, for the potential variable , add independent and identically distributed Gaussian noise, and obtain the potential noise after diffusion , calculated as follows:

[0012] ;

[0013] in, is the standard deviation of Gaussian noise, N represents a normal distribution with a mean of 0 and a variance of I .

[0014] Furthermore, in step S4, the potential variable , use the randomly generated mask m to mask, and then mask out some frames to get the potential mask , calculated as follows: ;

[0015] The ⊙ represents a multiplication operation.

[0016] Further, the step S5 includes the following sub-steps:

[0017] S51: Potential noise and latent mask The channel dimension feature concatenation is performed and input into the spatiotemporal UNet network, which is used to extract pure potential feature representation from the original input mixed with noise and mask. The mapping function of the spatiotemporal UNet network is set to , the expression is as follows: ;

[0018] in, represents the parameterized denoising network, represents a skip connection, and represents the scale factor;

[0019] S52: Introducing a short-term time adapter after the convolutional layer Conv Block of the spatiotemporal UNet network ST- Adapter , that is, add a linear layer between the upper and lower linear layers of the spatiotemporal UNet network 3D-Conv Convolution is used for short-term time series modeling. The process can be expressed as follows:

[0020] ;

[0021] in , Represent up-sampling and down-sampling linear layers, respectively. X express ST-Adapter Any input to the module;

[0022] S53: Short-term adapter ST-Adapter Then introduce a long-term time adapter LT-Adapter , used to model the global temporal relationship between adjacent frames, the long-term temporal adapter LT-Adapter consists of upper and lower linear layers and a temporal self-attention module Self-Attn, and is added to the short-term temporal adapter together with a temporal attention block ST-Adapter After that, the expression is as follows:

[0023] ;

[0024] in , Represent up-sampling and down-sampling linear layers, respectively. X express LT-Adapter Any input to the module;

[0025] S54: Introduce a space adapter after the long-term time adapter LT-Adapter S-Adapter , space adapter S-Adapter Added together with the Spatial Attention Block, it aims to adapt the spatiotemporal UNet network to the spatial distribution of the target dataset. S-Adapter It consists of a downsampling linear layer with a GELU activation function and an upsampling linear layer, and the expression is: ;

[0026] in , Represent up-sampling and down-sampling linear layers, respectively. X express S-Adapter Any input to the module.

[0027] S55: Fine-tune the spatiotemporal UNet network of step S51-step S54 to learn generalized feature representations, and extract compact video representations through the network middle layer of the spatiotemporal UNet network for subsequent behavior classification tasks, i.e. ,in , , Represent the feature height, width and feature dimension information respectively. The compact video is input to the classifier head consisting of an attention pooling and a fully connected layer. , used to predict the video behavior category label, the expression is as follows: ;

[0028] in, represents the scale factor, and represent the potential noise and the potential mask respectively;

[0029] S56: Using cross entropy loss To measure the true label and predicted class labels The difference between is calculated as follows: ;

[0030] in represents the true value label, represents the predicted action category label, Represents the index of all input videos.

[0031] Furthermore, in steps S2, S51-S53, the convolutional layers ConvBlock, Temporal Attention Block and Spatial Attention Block of the VAE encoder and the spatiotemporal Unet network are all weights of the diffusion model pre-trained on large-scale video data. The above parameters are fixed and only the short-term time adapter is trained. ST-Adapter , Long-term time adapter LT-Adapter And space adapter S- Adapter The learnable parameter part of the structure is sufficient.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention uses a diffusion large model to fine-tune the recognition of abnormal pilot behavior. On the one hand, the powerful feature expression ability of the large model is used to effectively learn the behavior characteristics. On the other hand, by fine-tuning and updating a small number of parameters, the feature expression ability of the large model can be transferred to the field of behavior recognition, thereby ensuring effective recognition of the behavior.

[0034] In summary, the present invention has the advantage of ensuring effective identification of behavior categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] like Figure 1 As shown in FIG. 1 , the abnormal behavior identification method of civil aviation pilots based on fine-tuning of the diffusion large model includes the following steps:

[0038] S1: A visual perception module (Intel RealSense D455 depth camera) is installed on the civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving, including smoking, using mobile phones, etc.

[0039] S2: Visual Perception Module Given a video , represents the set of real numbers, T Indicates the number of frames contained in the video. H , W are height and width respectively. After being processed by the pre-trained VAE encoder, the latent variable representation of the video is obtained. , the height and width are h , w ;

[0040] S3: The latent variables obtained from S2 Diffusion is performed to obtain the potential noise ;

[0041] S4: The latent variables obtained from S2 Apply a random mask to get the latent mask ;

[0042] S5: Potential noise obtained by adding S3 and the latent mask obtained by S4 Perform feature concatenation in the channel dimension and input it into the spatiotemporal UNet network to obtain the output result of action classification;

[0043] Furthermore, in step S3, for the potential variable , add independent and identically distributed Gaussian noise, and obtain the potential noise after diffusion , calculated as follows:

[0044] ;

[0045] in, is the standard deviation of Gaussian noise, Nrepresents a normal distribution with a mean of 0 and a variance of I ;

[0046] Furthermore, in step S4, the potential variable , use the randomly generated mask m to mask, and then mask out some frames to get the potential mask , calculated as follows:

[0047] ;

[0048] The ⊙ represents a multiplication operation.

[0049] Furthermore, the step S5 includes the following sub-steps:

[0050] S51: Potential noise and latent mask The channel dimension feature concatenation is performed and input into the spatiotemporal UNet network, which is used to extract pure potential feature representation from the original input mixed with noise and mask. The mapping function of the spatiotemporal UNet network is set to , the expression is as follows: ;

[0051] in, represents the parameterized denoising network, represents a skip connection, and represents the scale factor;

[0052] S52: Introducing a short-term time adapter after the convolutional layer Conv Block of the spatiotemporal UNet network ST- Adapter , that is, add a linear layer between the upper and lower linear layers of the spatiotemporal UNet network 3D-Conv Convolution is used for short-term time series modeling. The process can be expressed as follows:

[0053] ;

[0054] in , Represent up-sampling and down-sampling linear layers, respectively. X express ST-Adapter Any input to the module;

[0055] S53: Short-term adapter ST-Adapter Then introduce a long-term time adapter LT-Adapter, used to model the global temporal relationship between adjacent frames, the long-term temporal adapter LT-Adapter consists of upper and lower linear layers and a temporal self-attention module Self-Attn, and is added to the short-term temporal adapter together with a temporal attention block ST-Adapter After that, the expression is as follows:

[0056] ;

[0057] in , Represent up-sampling and down-sampling linear layers, respectively. X express LT-Adapter Any input to the module;

[0058] S54: Introduce a space adapter after the long-term time adapter LT-Adapter S-Adapter , space adapter S-Adapter Added together with the Spatial Attention Block, it aims to adapt the spatiotemporal UNet network to the spatial distribution of the target dataset. S-Adapter It consists of a downsampling linear layer with a GELU activation function and an upsampling linear layer, and the expression is: ;

[0059] in , Represent up-sampling and down-sampling linear layers, respectively. X express S-Adapter Any input to the module.

[0060] S55: Fine-tune the spatiotemporal UNet network of step S51-step S54 to learn generalized feature representations, and extract compact video representations through the network middle layer of the spatiotemporal UNet network for subsequent behavior classification tasks, i.e. ,in , , Represent the feature height, width and feature dimension information respectively. The compact video is input to the classifier head consisting of an attention pooling and a fully connected layer. , used to predict the video behavior category label, the expression is as follows: ;

[0061] in, represents the scale factor, and represent the potential noise and the potential mask respectively;

[0062] S56: Using cross entropy loss To measure the true label and predicted class labels The difference between is calculated as follows: ;

[0063] in represents the true value label, represents the predicted action category label, Represents the index of all input videos.

[0064] S6: Deploy the behavior recognition model trained in the above steps S2-S5 into the civil aviation cockpit to achieve effective recognition of pilot behavior.

[0065] In the steps S2, S51-S53, the convolutional layer Conv Block, Temporal Attention Block and Spatial Attention Block of the VAE encoder and the spatiotemporal Unet network are all weights of the diffusion model pre-trained on large-scale video data. Some of the above parameters are fixed, and only the short-term time adapter is trained. ST-Adapter , Long-term time adapter LT-Adapter And space adapter S-Adapter The learnable parameter part of the structure is sufficient.

[0066] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model, characterized by: The following steps are included S1: A visual perception module is installed on the civil aviation cockpit simulation training platform to capture abnormal behavior data of pilots during driving; S2: Visual Perception Module Given a video represents a set of real numbers, T represents the number of frames contained in the video, H and W are the height and width respectively, and the latent variable representation of the video is obtained after processing by the pre-trained VAE encoder The height and width are h and w respectively; S3: Diffusion of the latent variable z0 obtained in S2 to obtain the latent noise For the latent variable z0, add independent and identically distributed Gaussian noise, and obtain the latent noise after diffusion. The calculation is as follows: Where σ is the standard deviation of Gaussian noise, N represents normal distribution, with mean 0 and variance I; S4: Apply a random mask to the latent variable z0 obtained in S2 to obtain the latent mask For the latent variable z0, use the randomly generated mask m to mask it, and then mask out some frames to get the latent mask The calculation is as follows: The ⊙ represents a multiplication operation; S5: Potential noise obtained by adding S3 and the latent mask obtained by S4 Feature concatenation is performed in the channel dimension and input into the spatiotemporal UNet network to obtain the output result of action classification.

2. The method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model according to claim 1 is characterized in that: The S5 step includes the following sub-steps: S51: Potential noise and latent mask The channel dimension feature concatenation is performed and input into the spatiotemporal UNet network, which is used to extract pure potential feature representation from the original input mixed with noise and mask. The mapping function of the spatiotemporal UNet network is set to F θ , the expression is as follows: Among them, D θ represents the parameterized denoising network, c skip represents a skip connection, c out and c in represents the scale factor; S52: A short-term time adapter ST-Adapter is introduced after the convolution layer ConvBlock of the spatiotemporal UNet network, that is, a 3D-Conv convolution is added between the upper and lower linear layers of the spatiotemporal UNet network for short-term time series modeling. The process can be expressed as follows: ST-Adapter(X)=X+W up (3D-Conv(W down (X))); Where W up , W down They represent upsampling and downsampling linear layers respectively, and X represents any input of the ST-Adapter module; S53: A long-term time adapter LT-Adapter is introduced after the short-term time adapter ST-Adapter to model the global temporal relationship between adjacent frames. The long-term time adapter LT-Adapter consists of upper and lower linear layers and a temporal self-attention module Self-Attn, and is added after the short-term time adapter ST-Adapter together with a temporal attention TemporalAttentionBlock. The expression is as follows: LT-Adapter(X)=X+W up (Self-Attn(W down (X))); Where W up , W down Represent the up-sampling and down-sampling linear layers respectively, and X represents any input of the LT-Adapter module; S54: A spatial adapter S-Adapter is introduced after the long-term time adapter LT-Adapter. The spatial adapter S-Adapter is added together with the spatial attention SpatialAttentionBlock to adapt the spatiotemporal UNet network to the spatial distribution of the target data set. The spatial adapter S-Adapter consists of a downsampling linear layer with a GELU activation function and an upsampling linear layer, and the expression is: S-Adapter(X)=X+W up (GELU(W down (X))); Where W up , W down Represent the up-sampling and down-sampling linear layers respectively, and X represents any input of the S-Adapter module; S55: Fine-tune the spatiotemporal UNet network of step S51-step S54 to learn generalized feature representations, and extract compact video representations through the network intermediate layers of the spatiotemporal UNet network for subsequent behavior classification tasks, i.e. Table, where h', w', and D' represent feature height, width, and feature dimension information, respectively. The compact video is input into the classifier head consisting of an attention pooling and a fully connected layer. Used to predict video behavior category labels, the expression is as follows: Among them, c in represents the scale factor, and represent the potential noise and the potential mask respectively; S56: Using cross entropy loss L D To measure the true label y and the predicted category label The difference between is calculated as follows: Where y represents the true value label, represents the predicted action category label and i represents the index of all input videos.

3. The method for identifying abnormal behavior of civil aviation pilots based on fine-tuning of a diffusion large model according to claim 2 is characterized in that: In the steps S2, S51-S53, the convolutional layer Conv Block, temporal attention Temporal Attention Block and spatial attention Spatial Attention Block of the VAE encoder and the spatiotemporal Unet network are all weights of a large diffusion model pre-trained on large-scale video data. Some of the above parameters are fixed, and only the learnable parameter parts in the short-term time adapter ST-Adapter, long-term time adapter LT-Adapter and spatial adapter S-Adapter structures are trained.

Images