Drug abuse feature detection method, device and computer equipment based on multimodal information

Abstract

The present application belongs to the field of image processing technology, and relates to a method, device and computer equipment for drug abuse feature detection based on multimodal information. The method detects drug abuse features based on deep learning; the collection of voice and facial expression images used is non-contact, the person being tested has no resistance, and the collection process is simple and efficient; the cross-attention mechanism is used to fuse voice modal features and facial expression modal features, so that the model pays more attention to important features and then performs feature enhancement fusion, thereby enhancing the representation ability of each modality and improving the accuracy of drug abuse feature extraction. This method is used for drug addict detection, so that the detection process can be achieved in seconds, the detection efficiency is improved, and it is suitable for rapid screening of drug abuse for a large number of people.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a drug abuse feature detection method, device and computer equipment based on multimodal information. Background Art

[0002] At present, drug detection technology often uses biological samples such as hair, blood, and urine to identify whether the person being tested has taken drugs. However, these methods have the following disadvantages: (1) The process of collecting biological samples for testing requires close cooperation from the person being tested, which is cumbersome and puts a certain psychological burden on the person being tested; (2) The testing process requires the use of special chemical reagents and professional equipment, which is costly; (3) The testing process is time-consuming and inefficient; (4) It is not suitable for rapid screening of drug use among a large number of people.

[0003] Image processing technology has been widely used in feature detection. Through feature detection, key information such as the shape, color, and position of objects can be extracted from images or video sequences to provide support for subsequent target recognition, behavior analysis, and other tasks. At the same time, with the development of deep learning, methods have emerged that use facial expressions such as facial spasms and mental depression of drug addicts to detect drug abuse features. Based on the detection results, drug addiction behaviors can be identified, providing intelligent and automated management methods for drug rehabilitation centers. In the process of research on drug abuse feature detection based on image processing technology, people often only detect drug abuse features from facial expression images, and rarely pay attention to other features in the video, such as speech features. Drug abuse feature detection based only on facial expressions will miss some key features, resulting in inaccurate detection results. Summary of the invention

[0004] Based on this, it is necessary to provide a drug abuse feature detection method, device and computer equipment based on multimodal information to address the above technical problems.

[0005] A drug abuse feature detection method based on multimodal information, the method comprising:

[0006] Input the video image to be detected.

[0007] Obtain the facial expression image and voice signal of the target person in the video image to be detected.

[0008] After preprocessing the speech signal, speech feature extraction is performed to obtain speech modal features.

[0009] A lightweight deep learning neural network is used to extract facial expression modality features from facial expression images.

[0010] According to the speech modality features and facial expression modality features, a cross-attention mechanism is adopted to obtain speech modality interaction features and facial expression modality interaction features.

[0011] The speech modality interaction features, facial expression modality interaction features, speech modality features and facial expression modality features are fused using a gating mechanism to obtain multi-modal comprehensive features.

[0012] Taking multimodal comprehensive features as input, a bidirectional long short-term memory network is used as the classifier to detect drug abuse characteristics.

[0013] In one embodiment, the speech signal is preprocessed and then speech features are extracted to obtain speech modal features, including:

[0014] After pre-emphasis processing, the voice signal is segmented according to a preset time length to obtain multiple frames of voice signals;

[0015] Performing windowing processing on each frame of speech signal to obtain a pre-processed speech signal for each frame;

[0016] Speech feature extraction is performed based on the preprocessed speech signal to obtain a variety of speech features; the various speech features are: formant, MFCC coefficient, fundamental frequency, short-time zero-crossing rate and short-time energy;

[0017] All speech features are concatenated to obtain speech modal features.

[0018] In one embodiment, windowing is performed on each frame of speech signal to obtain a preprocessed speech signal for each frame, including:

[0019] A Hamming window is used to perform windowing processing on each frame of speech signal to obtain a preprocessed speech signal for each frame.

[0020] In one embodiment, the lightweight deep learning neural network includes 1 standard convolution layer, 1 weight sparsification layer, 3 depth-wise separable convolution layers, 1 global average pooling layer, and 1 fully connected layer.

[0021] In one embodiment, according to the speech modality features and the facial expression modality features, a cross attention mechanism is adopted to obtain the speech modality interaction features and the facial expression modality interaction features as follows:

[0022] ;

[0023] ;

[0024] in, , Respectively represent the speech modality interaction features and the facial expression modality interaction features, is the speech modality feature, is the facial expression modality feature, , , , , , are six adaptive coefficient matrices, represents the dot product, represents the dimension of the key vector, softmax represents the normalization function, , Represent the query vectors of speech modality features and facial expression modality features respectively, , , , Represent the key matrices of speech modality features and facial expression modality features, , , , Represent the value matrices of speech modality features and facial expression modality features, , .

[0025] In one embodiment, the speech modality interaction feature, the facial expression modality interaction feature, the speech modality feature, and the facial expression modality feature are fused using a gating mechanism to obtain a multi-modal comprehensive feature:

[0026] ;

[0027] in, is a multimodal comprehensive feature. , Respectively represent the speech modality interaction features and the facial expression modality interaction features, is the speech modality feature, is the facial expression modality feature, represents the gating function, which is defined as , g is the gating parameter.

[0028] A drug taking feature detection device based on multimodal information, the device comprising:

[0029] The multimodal data acquisition module is used to input the video image to be detected; and obtain the facial expression image and voice signal of the target person in the video image to be detected.

[0030] The speech modality feature extraction module is used to extract speech features after preprocessing the speech signal to obtain speech modality features.

[0031] The facial expression modality feature extraction module is used to extract facial expression modality features of facial expression images using a lightweight deep learning neural network.

[0032] The interactive feature extraction module is used to obtain speech modality interaction features and facial expression modality interaction features based on speech modality features and facial expression modality features by adopting a cross-attention mechanism.

[0033] The multimodal comprehensive feature extraction module is used to fuse the speech modal interaction features, the facial expression modal interaction features, the speech modal features and the facial expression modal features using a gating mechanism to obtain multimodal comprehensive features.

[0034] The drug abuse feature detection module is used to detect drug abuse features using multimodal comprehensive features as input and a bidirectional long short-term memory network as a classifier.

[0035] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Input the video image to be detected.

[0037] Obtain the facial expression image and voice signal of the target person in the video image to be detected.

[0038] After preprocessing the speech signal, speech feature extraction is performed to obtain speech modal features.

[0039] A lightweight deep learning neural network is used to extract facial expression modality features from facial expression images.

[0040] According to the speech modality features and facial expression modality features, a cross-attention mechanism is adopted to obtain speech modality interaction features and facial expression modality interaction features.

[0041] The speech modality interaction features, facial expression modality interaction features, speech modality features and facial expression modality features are fused using a gating mechanism to obtain multi-modal comprehensive features.

[0042] Taking multimodal comprehensive features as input, a bidirectional long short-term memory network is used as the classifier to detect drug abuse characteristics.

[0043] The above-mentioned drug-taking feature detection method, device and computer equipment based on multimodal information belong to image processing technology. The method processes video images, extracts facial expression images and voice signals of target persons in video images, and uses voice signals and facial expression images to detect drug-taking features based on deep learning. The acquisition of voice and facial expression images is non-contact, so the detected persons have no resistance, and the acquisition process is simple and efficient. The cross-attention mechanism is used to fuse voice modal features and facial expression modal features, so that the model pays more attention to important features and then performs feature enhancement fusion, thereby enhancing the representation ability of each modality and improving the accuracy of drug-taking feature extraction. This method is used for drug addict detection, so that the detection process reaches the second level, the detection efficiency is improved, and it is suitable for rapid screening of drug use of a large number of people. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a flow chart of a method for detecting drug abuse characteristics based on multimodal information in one embodiment;

[0045] Figure 2 is a flowchart of a method for detecting drug abuse characteristics based on multimodal information in one embodiment;

[0046] Figure 3 A flowchart of speech formant feature extraction in another embodiment;

[0047] Figure 4 A flowchart of extracting Mel frequency cepstral coefficient features of speech in another embodiment;

[0048] Figure 5 A diagram of a lightweight deep learning neural network structure in another embodiment;

[0049] Figure 6 is a structural block diagram of a drug abuse feature detection device based on multimodal information in one embodiment;

[0050] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 1 , Figure 2 As shown, a method for detecting drug abuse characteristics based on multimodal information is provided, and the method comprises the following steps:

[0053] Step 100: input a video image to be detected; obtain a facial expression image and a voice signal of a target person in the video image to be detected.

[0054] Specifically, the collection of voice and facial expression images is non-contact, the persons being tested have no resistance, and the collection process is simple and efficient.

[0055] The voice data set and face image data set of the detected person are divided to form the training set data required for fast recognition algorithm training.

[0056] Step 102: extract speech features after preprocessing the speech signal to obtain speech modal features.

[0057] Specifically, after preprocessing the speech signal, speech feature extraction is performed to obtain multiple speech features such as formant, MFCC coefficient, fundamental frequency, short-time zero-crossing rate and short-time energy. All speech features are connected in series to form speech modal features.

[0058] Step 104: Use a lightweight deep learning neural network to extract facial expression modality features of the facial expression image.

[0059] Specifically, a lightweight deep learning neural network model is used to extract facial expression features. The lightweight deep learning neural network model consists of modules such as standard convolution, weight sparsification, depthwise separable convolution, global average pooling, and full connection. Compared with the traditional deep learning convolutional neural network (CNN) model, the advantages of this lightweight deep learning neural network model are: (1) reduced storage requirements, reduced computational complexity, and improved detection efficiency; (2) the model training cycle is shorter and easier to maintain; (3) more suitable for deployment on devices with relatively limited hardware resources.

[0060] Step 106: Based on the speech modality features and the facial expression modality features, a cross attention mechanism is used to obtain speech modality interaction features and facial expression modality interaction features.

[0061] Specifically, the problem of insufficient emotional information in a single modality is reduced through the cross-attention mechanism between the speech modality and the facial expression image. The cross-attention mechanism is used to fuse speech features and facial expression features, so that the model pays more attention to important features and then performs feature enhancement fusion, thereby enhancing the representation ability of each modality.

[0062] Step 108: The speech modality interaction features, the facial expression modality interaction features, the speech modality features, and the facial expression modality features are fused using a gating mechanism to obtain a multi-modal comprehensive feature.

[0063] Specifically, features of different modalities contain information from different aspects. The features of different modalities complement each other and interactively fuse, which can make the feature representation richer, reduce the model's excessive dependence on a single modality, and help improve the accuracy of prediction and the generalization ability of the model.

[0064] Step 110: Using the multimodal comprehensive features as input, a bidirectional long short-term memory network is used as a classifier to detect drug abuse features.

[0065] Specifically, this method uses a bidirectional long short-term memory (BiLSTM) network as a classifier.

[0066] BiLSTM is an improvement on the Long Short-Term Memory (LSTM) network. The LSTM network is a specially designed artificial neural network that introduces a special memory unit and three types of gating structures: input gate, forget gate, and output gate to effectively capture and process long-term dependencies in sequence data. t The memory unit stores input data at every moment; the forget gate controls the storage and forgetting of information and decides which historical information to discard; the output gate determines t Which part of the memory unit is output at each moment. The calculation process of the LSTM model is as follows.

[0067] First, use the following formula to calculate the values ​​of the input gate, forget gate, and output gate:

[0068] (1)

[0069] (2)

[0070] (3)

[0071] in, , , Respectively t The input gate activation value, forget gate activation value and output gate activation value at the moment, represents the Sigmoid activation function, tanh represents the hyperbolic tangent activation function, express t The hidden state at time -1.

[0072] Then, determine the information in the cell state, that is, the candidate cell state value for:

[0073] (4)

[0074] in, , , , , , , , Both represent weight matrices, express t Input at the moment, , , , Represents the hidden layer bias vector.

[0075] Then calculate the current state value based on the cell state value at the previous moment and the current candidate value:

[0076] (5)

[0077] in, express t The cell state value at the moment, express t- The cell state value at time 1, Represents bitwise multiplication.

[0078] Calculated by the result of the output gate and the updated cell state value t Hidden state at the moment for:

[0079] (6)

[0080] The method of constructing a BiLSTM network based on LSTM is: the output value of the forward LSTM is , the output of the reverse LSTM is , BiLSTM outputs the forward LSTM Follow the reverse LSTM output The obtained vectors are concatenated, that is .

[0081] LSTM can only learn the dependency of current data on previous data, while BiLSTM can better capture contextual information in the sequence through bidirectional calculation, thereby improving model performance. BiLSTM can memorize the association between speech features over a longer time span, thereby effectively representing and processing long-term dependencies in long sequences. At the same time, BiLSTM is also highly versatile and suitable for classifying various types of sequence data. In addition, BiLSTM also has strong anti-interference capabilities against noise in the data.

[0082] In the above-mentioned drug-taking feature detection method based on multimodal information, the method detects the drug-taking features of the target person in the video image based on deep learning; the collection of voice and facial expression images is non-contact, the person being tested has no resistance, and the collection process is simple and efficient; the cross-attention mechanism is used to fuse the voice modal features and the facial expression modal features, so that the model pays more attention to important features and then performs feature enhancement fusion, thereby enhancing the representation ability of each modality and improving the accuracy of drug-taking feature extraction. This method is used for drug addict detection, so that the detection process reaches the second level, the detection efficiency is improved, and it is suitable for rapid screening of drug use for a large number of people.

[0083] In one embodiment, step 102 includes: performing pre-emphasis processing on the speech signal and then segmenting it according to a preset time length to obtain multiple frames of speech signals; performing windowing processing on each frame of the speech signal to obtain each frame of the preprocessed speech signal; performing speech feature extraction based on the preprocessed speech signal to obtain multiple speech features; the multiple speech features are: resonance peak, MFCC coefficient, fundamental frequency, short-time zero-crossing rate and short-time energy; all speech features are connected in series to obtain speech modal features.

[0084] Specifically, the principles of speech signal preprocessing and speech feature extraction are as follows:

[0085] 1) Speech signal preprocessing

[0086] The low-frequency part of the speech signal accounts for the majority of the energy. In order to increase the amplitude of the high-frequency part of the spectrum, the speech signal Before subsequent processing, it is usually necessary to use a first-order high-pass filter to pre-emphasize the speech signal. The formula for pre-emphasis is:

[0087] (7)

[0088] In a long period of time, speech signals have non-stationary characteristics, and their characteristics change over time. However, in a very small time segment, speech signals can be approximately regarded as stationary signals, and the characteristic parameters of speech are approximately regarded as unchanged in a short period of time. In other words, speech signals can be regarded as short-term stationary. For this reason, when processing speech signals, they are usually divided into small segments, called frames. Indicates the first i Frame speech signal. The speech signal is processed in frames, and the length of each frame is generally in the range of 10~30ms.

[0089] After framing, discontinuous mutations may occur at the edge of each frame, resulting in spectrum distortion. For this reason, it is necessary to perform windowing on the voice signal after framing to achieve the purpose of introducing a smooth transition at the edge of the frame. In the voice signal preprocessing stage, the windowing process of this embodiment uses a Hamming window, and its expression is:

[0090] (8)

[0091] in, represents the Hamming window function, is the length of the frame.

[0092] Using the Hamming window The expression for windowing is:

[0093] (9)

[0094] 2) Formant characteristics of speech

[0095] After preprocessing, the speech signal Perform a Fast Fourier Transform (FFT):

[0096] (10)

[0097] in, For speech signal The fast Fourier transform result of .

[0098] right Take the magnitude and then the logarithm:

[0099] (11)

[0100] right Perform Fourier transform to obtain the inverse spectrum :

[0101] (12)

[0102] right Add a rectangular window function ,get , where the rectangular window function is defined as:

[0103] (13)

[0104] in, is the width of the rectangular window function.

[0105] right Calculate the fast Fourier transform to get the envelope for:

[0106] (14)

[0107] By finding the maximum amplitude on the envelope, the parameters of the corresponding formant feature can be obtained. The speech formant feature extraction process is as follows: Figure 3 As shown in Figure 2, the resonance peak contains rich speech information, which is conducive to improving the accuracy of feature detection. At the same time, the resonance peak is not easily disturbed by noise, which is conducive to improving the anti-noise performance of feature detection.

[0108] 3) Mel frequency cepstral coefficients

[0109] In the process of obtaining the Mel Frequency Cepstrum Coefficient (MFCC), the first four steps are the same as the first four steps of the speech formant feature extraction process, that is, pre-emphasis, framing, Hamming windowing, and fast Fourier transform of the speech signal to obtain the frequency components. . Find the frequency component Later, calculate the corresponding spectral line energy for:

[0110] (15)

[0111] Next, use the Mel filter bank to The Mel scale is a logarithmic compression scale designed based on the human hearing characteristics. The Mel scale filter bank has filters, each with a triangular filtering characteristic. m The center frequency of the filter is , The transfer function of each filter is:

[0112] (16)

[0113] The filtering process is expressed as . Then Perform discrete cosine transform (DCT) to obtain the MFCC coefficients:

[0114] (17)

[0115] The process of extracting the Mel frequency cepstral coefficient features of speech is as follows Figure 4 shown.

[0116] 4) Fundamental frequency

[0117] The period of vocal cord vibration is the fundamental frequency, and its reciprocal is the fundamental frequency. The fundamental frequency feature mainly represents the speaker's timbre information and is very important in speech emotion recognition. After the three preprocessing steps of pre-emphasis, framing, and Hamming windowing of the speech signal, we can get , Indicates The first frame of the speech signal after frame preprocessing Sample values. Based on this, the present embodiment uses the short-time average amplitude difference function Find the fundamental frequency, which is defined as:

[0118] (18)

[0119] The short-time average amplitude difference function takes a minimum value at each integer point of the fundamental pitch period. The fundamental pitch period can be determined by the number of sampling points between adjacent minimum values, and the fundamental pitch frequency can be obtained by taking the inverse.

[0120] 5) Short-time zero-crossing rate

[0121] When two adjacent sample values ​​have opposite signs, it is called zero crossing. The short-time zero crossing rate indicates the number of times a frame of speech signal waveform crosses the zero level. After the three preprocessing steps of speech signal pre-emphasis, framing, and Hamming windowing, we can get , Indicates The first frame of the speech signal after frame preprocessing Sample values. Based on this, the short-time zero-crossing rate is defined as:

[0122] (19)

[0123] in, Indicates Frame short-time zero-crossing rate, Indicates The first frame of the speech signal after frame preprocessing Sample values, represents the symbolic function, N Indicates the length of a frame of data. Indicates The first frame of the speech signal after frame preprocessing A sample value.

[0124] 6) Short-term energy

[0125] Short-time energy represents the sum of squares of the sampled values ​​of a frame of speech signal. After the three preprocessing steps of pre-emphasis, framing, and Hamming windowing of speech signal, we can get , Indicates The first frame of the speech signal after frame preprocessing Sample values. Based on this, short-time energy is defined as:

[0126] (20)

[0127] in, For short-term energy.

[0128] Finally, the five features of formant, MFCC coefficient, fundamental frequency, short-time zero-crossing rate, and short-time energy are concatenated to form the speech modal feature. .

[0129] In one embodiment, windowing is performed on each frame of speech signal to obtain each frame of preprocessed speech signal, including: windowing is performed on each frame of speech signal using a Hamming window to obtain each frame of preprocessed speech signal.

[0130] In one embodiment, if Figure 5 As shown in the figure, the lightweight deep learning neural network includes 1 standard convolution layer, 1 weight sparsification layer, 3 depth-separable convolution layers, 1 global average pooling layer and 1 fully connected layer.

[0131] Specifically, Depthwise Separable Convolution consists of two parts: Depthwise Convolution and Pointwise Convolution. Depthwise convolution uses a convolution kernel for each channel of the input feature map, and then concatenates the outputs of all convolution kernels to get the final output. In depthwise convolution, only one convolution kernel is used for each channel, so the number of output channels of a single channel after the convolution operation is also 1. Pointwise convolution is a convolution with a convolution kernel of 1×1. It can freely change the number of output channels and perform channel fusion on the feature map output by the depthwise convolution to integrate the feature information between different channels. The ratio of the computational complexity of depthwise separable convolution to that of standard convolution is ,in is the size of the convolution kernel, is the number of input feature maps. It can be seen that compared with standard convolution, depthwise separable convolution can greatly reduce the number of parameters and computational complexity, thereby effectively improving the reasoning speed and efficiency of the neural network model.

[0132] When the facial expression images in the training set achieve the optimal classification effect, the parameters of each layer of the lightweight deep learning neural network are determined. Subsequently, the facial expression modality features are extracted through the lightweight deep learning neural network based on these parameters. .

[0133] Preferably, the lightweight deep learning neural network includes 1 3×3 standard convolution, 1 weight sparsification layer, 1 3×3 depth-wise separable convolution layer, 1 5×5 depth-wise separable convolution layer, 1 7×7 depth-wise separable convolution, 1 global average pooling layer and 1 fully connected layer.

[0134] In one embodiment, step 108 includes: according to the speech modality features and the facial expression modality features, a cross attention mechanism is used to obtain the speech modality interaction features and the facial expression modality interaction features as follows:

[0135] (twenty one)

[0136] (twenty two)

[0137] in, , Respectively represent the speech modality interaction features and the facial expression modality interaction features, is the speech modality feature, is the facial expression modality feature, , , , , , are six adaptive coefficient matrices (adaptively updated and iterated by dynamically adjusting the learning rate through the first-order moment estimation and second-order moment estimation of the gradient), represents the dot product, represents the dimension of the key vector, softmax represents the normalization function, , Represent the query vectors of speech modality features and facial expression modality features respectively, , , , Represent the key matrices of speech modality features and facial expression modality features, , , , Represent the value matrices of speech modality features and facial expression modality features, , .

[0138] Specifically, the present application uses a cross-attention mechanism to interact between two modalities. The problem of insufficient emotional information of a single modality is reduced through the cross-attention mechanism between the two modalities. The attention mechanism assigns higher weights to certain parts of the features by focusing on them. The essence of the attention mechanism is to allow the model to learn weights, add weights to the features, and pay more attention to the data you want to pay attention to, so that subsequent data training is affected by this part. The cross-attention mechanism is used to fuse speech features and facial expression features. By calculating the attention weights, the model pays more attention to important features and then performs feature enhancement fusion, thereby enhancing the representation ability of each modality.

[0139] A modality feature as a query ( ), another modal feature as key ( K ) and value ( ), use the attention mechanism to calculate the correlation between each element of the two modal features. The attention weight is combined with the value ( ) are multiplied and summed to obtain the inter-modal interaction feature. The calculation formula of the interaction feature is shown in formula (21) to formula (22).

[0140] Finally, the speech modal interaction features, facial expression modal interaction features, speech modal features, and facial expression modal features are connected to form a multimodal comprehensive feature. .

[0141] By introducing inter-modal interaction features, the limitations of single modal features in sentiment analysis can be effectively reduced, thereby improving the performance of the model.

[0142] In one embodiment, step 110 includes: fusing the speech modality interaction feature, the facial expression modality interaction feature, the speech modality feature, and the facial expression modality feature using a gating mechanism to obtain a multimodal comprehensive feature:

[0143] (twenty three)

[0144] in, is a multimodal comprehensive feature. , Respectively represent the speech modality interaction features and the facial expression modality interaction features, is the speech modality feature, It is the facial expression modality feature. represents the gating function, which is defined as ,The gating function can dynamically adjust the contribution of speech modality features and facial expression modality features according to the video content.

[0145] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0146] In one embodiment, Figure 6 As shown, a drug taking feature detection device based on multimodal information is provided, comprising: a multimodal data acquisition module, a voice modal feature extraction module, a facial expression modal feature extraction module, an interactive feature extraction module, a multimodal comprehensive feature extraction module and a drug taking feature detection module, wherein:

[0147] The multimodal data acquisition module is used to input the video image to be detected; and obtain the facial expression image and voice signal of the target person in the video image to be detected.

[0148] The speech modal feature extraction module is used to extract speech features after preprocessing the speech signal to obtain speech modal features; all speech features are connected in series to obtain speech modal features; the multiple speech features are: formant, MFCC coefficient, fundamental frequency, short-time zero-crossing rate and short-time energy.

[0149] The facial expression modality feature extraction module is used to extract facial expression modality features of facial expression images using a lightweight deep learning neural network.

[0150] The interactive feature extraction module is used to obtain speech modality interaction features and facial expression modality interaction features based on speech modality features and facial expression modality features by adopting a cross-attention mechanism.

[0151] The multimodal comprehensive feature extraction module is used to fuse the speech modal interaction features, the facial expression modal interaction features, the speech modal features and the facial expression modal features using a gating mechanism to obtain multimodal comprehensive features.

[0152] The drug abuse feature detection module is used to detect drug abuse features using multimodal comprehensive features as input and a bidirectional long short-term memory network as a classifier.

[0153] In one of the embodiments, the speech modal feature extraction module is also used to perform pre-emphasis processing on the speech signal and then segment it according to a preset time length to obtain multiple frames of speech signals; perform windowing processing on each frame of the speech signal to obtain each frame of the preprocessed speech signal; perform speech feature extraction based on the preprocessed speech signal to obtain multiple speech features; the multiple speech features are: resonance peak, MFCC coefficient, fundamental frequency, short-time zero-crossing rate and short-time energy; all speech features are connected in series to obtain speech modal features.

[0154] In one of the embodiments, the speech modality feature extraction module is further used to perform windowing processing on each frame of speech signal using a Hamming window to obtain a pre-processed speech signal for each frame.

[0155] In one of the embodiments, the lightweight deep learning neural network in the facial expression modality feature extraction module includes 1 standard convolution layer, 1 weight sparsification layer, 3 depth-separable convolution layers, 1 global average pooling layer and 1 fully connected layer.

[0156] In one of the embodiments, the interaction feature extraction module is also used to obtain the speech modality interaction features and the facial expression modality interaction features as shown in formulas (21) to (22) by using a cross-attention mechanism based on the speech modality features and the facial expression modality features.

[0157] In one of the embodiments, the multimodal comprehensive feature extraction module is also used to fuse the speech modal interaction features, the facial expression modal interaction features, the speech modal features and the facial expression modal features using a gating mechanism to obtain a multimodal comprehensive feature; the multimodal comprehensive feature is as described in formula (23).

[0158] For the specific definition of the drug taking characteristic detection device based on multimodal information, please refer to the definition of the drug taking characteristic detection method based on multimodal information above, which will not be repeated here. Each module in the above-mentioned drug taking characteristic detection device based on multimodal information can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0159] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a drug abuse feature detection method based on multimodal information is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0160] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A drug abuse feature detection method based on multimodal information, characterized in that: The method comprises: Input the video image to be detected; Acquire the facial expression image and voice signal of the target person in the video image to be detected; After preprocessing the speech signal, speech feature extraction is performed to obtain speech modal features; Extracting facial expression modality features of the facial expression image using a lightweight deep learning neural network; According to the speech modality features and the facial expression modality features, a cross attention mechanism is adopted to obtain speech modality interaction features and facial expression modality interaction features; The speech modality interaction feature, the facial expression modality interaction feature, the speech modality feature and the facial expression modality feature are fused using a gating mechanism to obtain a multi-modal comprehensive feature; The multimodal comprehensive features are used as input, and a bidirectional long short-term memory network is used as a classifier to detect drug abuse features.

2. The drug abuse feature detection method based on multimodal information according to claim 1, characterized in that: After preprocessing the speech signal, speech feature extraction is performed to obtain speech modal features, including: After pre-emphasis processing is performed on the speech signal, the speech signal is segmented according to a preset time length to obtain a plurality of frames of speech signals; Performing windowing processing on each frame of speech signal to obtain a pre-processed speech signal for each frame; Speech feature extraction is performed based on the preprocessed speech signal to obtain a variety of speech features; the various speech features are: formant, MFCC coefficient, fundamental frequency, short-time zero-crossing rate and short-time energy; All the speech features are concatenated and spliced ​​to obtain speech modal features.

3. The drug abuse feature detection method based on multimodal information according to claim 2, characterized in that: Perform windowing processing on each frame of speech signal to obtain a pre-processed speech signal for each frame, including: A Hamming window is used to perform windowing processing on each frame of speech signal to obtain a preprocessed speech signal for each frame.

4. The drug abuse feature detection method based on multimodal information according to claim 1, characterized in that: The lightweight deep learning neural network includes 1 standard convolution layer, 1 weight sparsification layer, 3 depth-separable convolution layers, 1 global average pooling layer and 1 fully connected layer.

5. The drug abuse feature detection method based on multimodal information according to claim 1, characterized in that: According to the speech modality features and the facial expression modality features, a cross attention mechanism is adopted to obtain speech modality interaction features and facial expression modality interaction features as follows: ； in, , Respectively represent the speech modality interaction features and the facial expression modality interaction features, is the speech modality feature, is the facial expression modality feature, , , , , , are six adaptive coefficient matrices, represents the dot product, represents the dimension of the key vector, softmax represents the normalization function, , Represent the query vectors of speech modality features and facial expression modality features respectively, , , , Represent the key matrices of speech modality features and facial expression modality features, Represent the value matrices of speech modality features and facial expression modality features, , .

6. The drug abuse feature detection method based on multimodal information according to claim 1, characterized in that: The speech modality interaction feature, the facial expression modality interaction feature, the speech modality feature and the facial expression modality feature are fused using a gating mechanism to obtain a multi-modal comprehensive feature: ； in, is a multimodal comprehensive feature. , Respectively represent the speech modality interaction features and the facial expression modality interaction features, is the speech modality feature, is the facial expression modality feature, represents the gating function, which is defined as , g is the gating parameter.

7. A drug abuse feature detection device based on multimodal information, characterized in that: The device comprises: The multimodal data acquisition module is used to input the video image to be detected; obtain the facial expression image and voice signal of the target person in the video image to be detected; A speech modality feature extraction module, used to extract speech features after preprocessing the speech signal to obtain speech modality features; A facial expression modality feature extraction module, used for extracting facial expression modality features of the facial expression image using a lightweight deep learning neural network; An interactive feature extraction module, used to obtain a speech modality interactive feature and a facial expression modality interactive feature according to the speech modality feature and the facial expression modality feature by adopting a cross attention mechanism; A multimodal comprehensive feature extraction module, used to fuse the speech modality interaction feature, the facial expression modality interaction feature, the speech modality feature and the facial expression modality feature using a gating mechanism to obtain a multimodal comprehensive feature; The drug abuse feature detection module is used to detect drug abuse features using the multimodal comprehensive features as input and a bidirectional long short-term memory network as a classifier.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the drug abuse feature detection method based on multimodal information described in any one of claims 1 to 6 is implemented.

Images