Behavior attribute recognition method and device based on multi-modal fusion and storage medium

Abstract

The embodiment of the application provides a kind of based on multi-modal fusion behavior attribute identification method, device and storage medium, identification method includes obtaining video data to be handled and audio data to be handled;At least one object is included in video data to be handled;According to video data to be handled and audio data to be handled, portrait information of object is determined;According to video data to be handled, action information of object is determined;According to audio data to be handled, emotion information and text information of object are determined;Based on portrait information, action information, emotion information and text information, the behavior attribute of object is determined.The embodiment of the application is based on video, speech and portrait multi-modal fusion and identifies the behavior attribute of object, can reduce the misjudgment rate, improve the identification accuracy of dangerous behavior.

Description

Technical Field

[0001] This invention relates to the field of safe driving technology, and in particular to a method, apparatus and storage medium for behavioral attribute recognition based on multimodal fusion. Background Technology

[0002] With the increasing number of cars on the road and the maturity of artificial intelligence technology, vehicle driving safety has gradually become a focus of public attention, and advanced driver assistance systems (ADAS) have emerged as a new development direction. During driving, the vast majority of traffic accidents are caused by improper operation by the driver or passengers. Furthermore, with the rise of new travel modes such as ride-hailing and carpooling, recognizing dangerous actions by drivers and passengers is particularly important for improving the safety of occupants.

[0003] Most existing methods for identifying dangerous actions rely on single visual or auditory judgments and analyses. Visual judgment analysis includes both machine learning and deep learning algorithms. Machine learning algorithms utilize adaptive augmented face detection algorithms to detect the face range and locate the human action region, and use a Support Vector Machine (SVN) model to train a positive and negative sample database. In the prediction phase, the probability of the dangerous action is output by loading the trained SVN. This machine learning algorithm is fast and requires little computing power, but its prediction accuracy is low and its generalization ability is insufficient, resulting in poor application performance. Deep learning algorithms can extract and learn image features from multiple levels and are widely used in video action recognition. They employ object detection algorithms such as SSD or YOLOv3, extracting features from feature maps and detecting templates for localization and classification in convolutional layers. Since video actions are time-based products of images, to focus on the modeling ability of action recognition network models in the temporal dimension, a 3D CNN neural network model is proposed to convolve multiple consecutive frames of images, connecting multiple feature maps to capture target motion information. Furthermore, Time Transition Model (TSM) and Time Activation and Fusion Model (TEA) have been proposed for judging complex actions. However, these models have large structures and require significant computational power for practical deployment, resulting in poor real-time performance. Currently, computer vision-based algorithms for recognizing dangerous actions are only applicable to simple scenarios such as making phone calls, and are not suitable for complex scenarios. They are prone to misidentifying certain actions of the driver or passengers as dangerous, leading to erroneous operations by the vehicle's infotainment system and causing distress to occupants. Additionally, they are prone to missing detections of fights among occupants and interference from passengers with the driver.

[0004] Visual judgment analysis is primarily applied to voice interaction between drivers and passengers and the vehicle's infotainment system. It employs technologies such as speech recognition, semantic understanding, and emotion detection to convert the collected speech into text or commands. A computer then transforms this text into machine-understandable feature parameters, outputting the recognition results. Related algorithm models mainly include Hidden Markov Models (HMMs), N-gram language models, and deep learning network models. While this can improve judgment accuracy, the models must run sequentially, each requiring the output of the previous model as input, resulting in slow processing speed. Summary of the Invention

[0005] This application provides a method, apparatus, and storage medium for behavioral attribute recognition based on multimodal fusion, which can reduce the false judgment rate and improve the accuracy of identifying dangerous behaviors.

[0006] This application provides a behavior attribute recognition method based on multimodal fusion, comprising:

[0007] Acquire video data and audio data to be processed; the video data to be processed includes at least one object.

[0008] Based on the video and audio data to be processed, determine the profile information of the object;

[0009] Based on the video data to be processed, determine the object's motion information;

[0010] Based on the audio data to be processed, determine the emotional and textual information of the subject;

[0011] Based on profile information, action information, emotion information, and text information, the behavioral attributes of the object are determined.

[0012] Furthermore, based on the video data and audio data to be processed, the object's profile information is determined, including:

[0013] Determine the facial data of the objects from the video data to be processed;

[0014] From the audio data to be processed, determine the speech feature data of the object;

[0015] Based on facial and voice feature data, determine the object's label information, age information, and gender information;

[0016] Based on the tag information, age information, and gender information, determine the profile information of the object.

[0017] Furthermore, based on the video data to be processed, the motion information of the object is determined, including:

[0018] From each frame of video data to be processed, determine the set of key points corresponding to the object;

[0019] Determine the position and confidence data of each key point in the key point set;

[0020] Based on the position data and confidence data of each joint in each frame of video data, the motion information of the object is determined.

[0021] Furthermore, based on the audio data to be processed, the emotional information of the subject is determined, including:

[0022] The audio data to be processed is subjected to feature extraction to obtain a speech feature vector sequence; two adjacent speech feature vectors in the speech feature vector sequence partially overlap.

[0023] Determine the attention vector corresponding to each speech feature vector in the speech feature vector sequence;

[0024] The attention vector corresponding to each speech feature vector is concatenated with each speech feature vector to obtain a speech feature vector sequence with attention characteristic information;

[0025] The emotional information of the subject is determined based on the sequence of speech feature vectors containing attention characteristics.

[0026] Furthermore, based on the audio data to be processed, the emotional information of the subject is determined, including:

[0027] The audio data to be processed is subjected to feature extraction to obtain a first speech feature vector and a second speech feature vector; the first speech feature vector and the second speech feature vector are adjacent and partially overlap;

[0028] The first speech feature vector is subjected to feature extraction processing to obtain the first speech feature subvector and the second speech feature subvector.

[0029] The second speech feature vector and the second speech feature sub-vector are stacked to obtain the stacked speech feature vector.

[0030] Determine the set of attention vectors corresponding to the stacked speech feature vectors;

[0031] Based on the first speech feature sub-vector, the stacked speech feature vector, and the set of attention vectors corresponding to the stacked speech feature vector, determine the first attention speech feature sub-vector and the second attention speech feature sub-vector;

[0032] The emotional information of the object is determined based on the first attention speech feature vector and the second attention speech feature vector.

[0033] Furthermore, based on the audio data to be processed, the text information of the object is determined, including:

[0034] The audio data to be processed is subjected to feature extraction to obtain a speech feature vector sequence; two adjacent speech feature vectors in the speech feature vector sequence partially overlap.

[0035] Speech recognition processing is performed on the speech feature vector sequence to obtain the sentence sequence;

[0036] Identify the target words from each statement in the statement sequence;

[0037] The text information of the object is determined based on the target word statement in each statement of the statement sequence.

[0038] Furthermore, based on the portrait information, action information, emotion information, and text information, the behavioral attributes of the object are determined, including:

[0039] The first parameter is determined based on the portrait information, action information, emotion information, text information, and first attribute information;

[0040] The second parameter is determined based on the portrait information, action information, emotion information, text information, and second attribute information.

[0041] The object's behavioral attributes are determined based on the first and second parameters.

[0042] Accordingly, embodiments of this application provide a behavior attribute recognition device based on multimodal fusion, comprising:

[0043] The acquisition module is used to acquire video data and audio data to be processed; the video data to be processed includes at least one object.

[0044] The first determining module is used to determine the image information of the object based on the video data and audio data to be processed;

[0045] The second determining module is used to determine the motion information of the object based on the video data to be processed;

[0046] The third determination module is used to determine the emotional information and text information of the object based on the audio data to be processed;

[0047] The fourth determination module is used to determine the behavioral attributes of an object based on portrait information, action information, emotion information, and text information.

[0048] Furthermore, the first determining module is also used to determine the facial data of the object from the video data to be processed; to determine the voice feature data of the object from the audio data to be processed; to determine the label information, age information and gender information of the object based on the facial data and voice feature data; and to determine the portrait information of the object based on the label information, age information and gender information.

[0049] Furthermore, the second determining module is also used to determine the set of joints corresponding to the object from each frame of video data of the video data to be processed; determine the position data and confidence data of each joint in the set of joints; and determine the motion information of the object based on the position data and confidence data of each joint in each frame of video data.

[0050] Furthermore, the third determining module is also used to perform feature extraction processing on the audio data to be processed to obtain a speech feature vector sequence; two adjacent speech feature vectors in the speech feature vector sequence partially overlap; determine the attention vector corresponding to each speech feature vector in the speech feature vector sequence; concatenate the attention vector corresponding to each speech feature vector with each speech feature vector to obtain a speech feature vector sequence with attention feature information; and determine the emotional information of the object based on the speech feature vector sequence with attention feature information.

[0051] Furthermore, the third determining module is also used to perform feature extraction processing on the audio data to be processed, to obtain a first speech feature vector and a second speech feature vector; the first speech feature vector and the second speech feature vector are adjacent and partially overlap; feature extraction processing is performed on the first speech feature vector to obtain a first speech feature sub-vector and a second speech feature sub-vector; the second speech feature vector and the second speech feature sub-vector are stacked to obtain a stacked speech feature vector; the attention vector set corresponding to the stacked speech feature vector is determined; based on the first speech feature sub-vector, the stacked speech feature vector, and the attention vector set corresponding to the stacked speech feature vector, a first attention speech feature sub-vector and a second attention speech feature sub-vector are determined; based on the first attention speech feature sub-vector and the second attention speech feature sub-vector, the emotional information of the object is determined.

[0052] Furthermore, the third determining module is also used to perform feature extraction processing on the audio data to be processed to obtain a speech feature vector sequence; two adjacent speech feature vectors in the speech feature vector sequence partially overlap; perform speech recognition processing on the speech feature vector sequence to obtain a sentence sequence; determine the target word from each sentence in the sentence sequence; and determine the text information of the object based on the target word in each sentence of the sentence sequence.

[0053] Furthermore, the fourth determining module is used to determine the first parameter based on the portrait information, action information, emotion information, text information, and first attribute information; determine the second parameter based on the portrait information, action information, emotion information, text information, and second attribute information; and determine the object's behavioral attributes based on the first parameter and the second parameter.

[0054] Accordingly, this application also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the above-described behavior attribute recognition method based on multimodal fusion.

[0055] Accordingly, embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the above-described behavior attribute recognition method based on multimodal fusion.

[0056] The embodiments of this application have the following beneficial effects:

[0057] The behavioral attribute recognition method, apparatus, and storage medium based on multimodal fusion disclosed in this application include: acquiring video data and audio data to be processed; the video data to be processed includes at least one object; determining the object's profile information based on the video data and audio data; determining the object's action information based on the video data; determining the object's emotional and textual information based on the audio data; and determining the object's behavioral attributes based on the profile information, action information, emotional information, and textual information. Based on this application, by determining the object's profile information according to its tags, age, and gender, the judgment of dangerous behaviors inside the vehicle no longer relies solely on instantaneous video and audio data, but comprehensively considers the object's past behavior and profile, reducing misjudgments of habitual safe actions of drivers and passengers and improving the accuracy of identifying dangerous actions inside the vehicle. Furthermore, by designing an open-source system that can directly acquire human skeletal joint position information without complex image processing preprocessing, and using a spatiotemporal graph neural network model to extract the object's pose data for training and prediction, it exhibits stronger expressive power and higher performance in the time dimension, reducing latency, computational power, and enhancing real-time performance and robustness. By employing a bidirectional long short-term memory model based on an attention mechanism and a support vector machine algorithm for emotion and text recognition, auditory judgment analysis is added to the single visual judgment analysis, which can reduce the false positive rate and improve the reliability of behavior recognition. Furthermore, by using the Naive Bayes algorithm to fuse profile, action, emotion, and text information, independent of feature values ​​of different dimensions, computational power can be reduced while increasing the accuracy of identifying dangerous behaviors. Attached Figure Description

[0058] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of an application environment provided by an embodiment of this application;

[0060] Figure 2 This is a flowchart illustrating a behavior attribute recognition method based on multimodal fusion provided in an embodiment of this application;

[0061] Figure 3 This is a flowchart illustrating another behavior attribute recognition method based on multimodal fusion provided in an embodiment of this application;

[0062] Figure 4 This is an example diagram of an action information recognition method provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram illustrating the preprocessing of audio data to be processed, provided in an embodiment of this application.

[0064] Figure 6 This is a schematic diagram of an emotion recognition method provided in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of the structure of a single neuron in a long short-term memory model provided in an embodiment of this application;

[0066] Figure 8 This is a schematic diagram of a long short-term memory model provided in an embodiment of this application.

[0067] Figure 9 This is a schematic diagram of the structure of a behavior attribute recognition device based on multimodal fusion provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely one embodiment of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0069] The term "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this application. In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Furthermore, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprising," "having," and "being," and any variations thereof, are intended to cover non-exclusive inclusion.

[0070] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application, including a server 10, an in-vehicle camera device 20, and an in-vehicle recording device 30. The server 10 can acquire video data to be processed from the in-vehicle camera device 20 and audio data to be processed from the in-vehicle recording device 30. The video data to be processed includes at least one object. Based on the video data and audio data, the server 10 can determine the object's profile information, the object's action information, and the object's emotional and textual information. Then, based on the profile information, action information, emotional information, and textual information, the object's behavioral attributes can be determined.

[0071] In some possible implementations, due to the limited computing power of the onboard chip, the collected video and audio data can be used for local real-time forward inference and model training on a cloud server, respectively. The updated model parameters after each training iteration are then transmitted back to the local forward inference to improve the accuracy and generalization of the judgment results, while also saving onboard computing power and real-time performance. Therefore, server 10 can be a standalone physical server, a service cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory, etc. Server 10 can provide behavioral attribute recognition services.

[0072] The following describes a specific embodiment of a behavior attribute recognition method based on multimodal fusion according to this application. Figure 2 This is a flowchart illustrating a behavior attribute recognition method based on multimodal fusion provided in an embodiment of this application. Figure 3 This is a flowchart illustrating another behavioral attribute recognition method based on multimodal fusion provided in this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one of many execution orders and does not represent the only execution order. In actual execution, the methods can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0073] Specific examples Figure 2 and Figure 3 As shown, the identification method may include:

[0074] S201: Obtain video data and audio data to be processed; the video data to be processed includes at least one object.

[0075] In this embodiment, the server can acquire video data to be processed from a camera inside the vehicle and audio data to be processed from an in-vehicle recording device. The video data acquired by the camera inside the vehicle includes at least one object, i.e., at least one occupant. For ease of understanding, the following explanation uses an example where the video data includes two objects.

[0076] S203: Determine the image information of the object based on the video data and audio data to be processed.

[0077] In this embodiment of the application, the server can determine the facial data of the object from the video data to be processed, determine the voice feature data of the object from the audio data to be processed, and determine the object's tag information, age information and gender information based on the facial data and voice feature data, and then determine the object's portrait information based on the tag information, age information and gender information.

[0078] In some possible implementations, the video and audio data to be processed can be input into a profile recognition model, outputting the profile information of the object. Optionally, when extracting the video data, each object can be labeled and matched using a facial recognition system to obtain the historical behavior data of each object. For example, if the historical behavior data of passengers in the vehicle shows a tendency to frequently perform dangerous actions, the object can be labeled as high-risk. Simultaneously, the age and gender of the object can be analyzed using a facial recognition system. The age and gender of the object can also be determined using a speech recognition system. Alternatively, the age and gender of the object can be determined by combining facial recognition and speech recognition. Then, the object's profile information can be determined by combining the object's label information, age information, and gender information. By determining the object's profile information based on the object's label, age, and gender, the judgment of dangerous behavior in the vehicle no longer relies solely on instantaneous video and audio data, but comprehensively considers the object's past behavior and profile, reducing misjudgments of habitual safe actions of drivers and passengers and improving the accuracy of identifying dangerous actions in the vehicle.

[0079] S205: Determine the motion information of the object based on the video data to be processed.

[0080] In this embodiment of the application, the server can determine the set of key points corresponding to the object from each frame of the video data to be processed, and determine the position data and confidence data of each key point in the set of key points. Then, the server can determine the motion information of the object based on the position data and confidence data of each key point in each frame of video data.

[0081] In some possible implementations, the video data to be processed can be input into an action recognition model, which outputs the action information of the object. Optionally, the action recognition model can be a trained spatiotemporal graph neural network model. Figure 4This is an example diagram of an action information recognition method provided in an embodiment of this application. During the training process of the spatiotemporal graph neural network model, joints of the human body in the training set, such as the neck, shoulder, and elbow joints, can be labeled using OpenPose to connect them into a skeleton, thereby estimating the human posture. A batch of training data input to the spatiotemporal graph neural network model can be represented by a 5-dimensional matrix (N, C, T, V, M). Here, N represents the number of video frames; for example, a batch can have 128 videos. C represents the extracted joint features, including position dimensions x, y and confidence level acc. T represents the number of keyframes; for example, a video can include 150 frames. V represents the number of joints; for example, each object can have 18 joints labeled. M represents the number of occupants in each frame; the two occupants with the highest average confidence level can be selected based on the in-vehicle scene. Thus, the video feature dimension information (128, 3, 150, 18, 2) can be extracted. Then, based on the labeled joints, the action information of the object can be determined. In practical applications, action information can include categories such as typing on a mobile phone, opening a door, yawning, and fighting with people inside a car. The action with the highest probability value after classification is output as the prediction result. The spatiotemporal graph neural network model includes nine convolutional layers, grouped into three layers (front, middle, and back) of three, outputting 64, 128, and 256 channels respectively. Residual connections are used between each convolutional layer, and dropout is used to randomly remove half of the neurons in the middle layers to prevent overfitting. Pooling layers are added in the fourth and seventh convolutional layers, optimized using stochastic gradient descent, and trained using backpropagation to generate weight parameters that express the features. By designing an open-source system that can directly obtain the joint position information of the human skeleton without complex image processing preprocessing, and using the spatiotemporal graph neural network model to extract the object's pose data for training and prediction, it exhibits stronger expressive power and higher performance in the temporal dimension, reducing latency, computational power, and enhancing real-time performance and robustness.

[0082] S207: Determine the emotional information and text information of the object based on the audio data to be processed.

[0083] In this embodiment, the server can perform feature extraction processing on the audio data to be processed to obtain a speech feature vector sequence, wherein two adjacent speech feature vectors in the speech feature vector sequence partially overlap. Then, the server can determine the attention vector corresponding to each speech feature vector in the speech feature vector sequence, and concatenate the attention vector corresponding to each speech feature vector with each speech feature vector to obtain a speech feature vector sequence with attention characteristic information. Based on the speech feature vector sequence with attention characteristic information, the server can determine the emotional information of the object.

[0084] In this embodiment, the server can perform feature extraction processing on the audio data to be processed to obtain a first speech feature vector and a second speech feature vector, wherein the first speech feature vector and the second speech feature vector are adjacent and partially overlap. Then, feature extraction processing can be performed on the first speech feature vector to obtain a first speech feature sub-vector and a second speech feature sub-vector, and the second speech feature vector and the second speech feature sub-vector can be stacked to obtain a stacked speech feature vector. Next, the attention vector set corresponding to the stacked speech feature vector can be determined, and based on the first speech feature sub-vector, the stacked speech feature vector, and the attention vector set corresponding to the stacked speech feature vector, a first attention speech feature sub-vector and a second attention speech feature sub-vector can be determined. Finally, based on the first attention speech feature sub-vector and the second attention speech feature sub-vector, the emotional information of the object can be determined.

[0085] In some possible implementations, after acquiring the audio data to be processed, physical methods can be used to preprocess the audio data to improve its volume and quality. Furthermore, feature extraction can be performed using Mel-frequency cepstral coefficients to convert the audio data into a sequence of speech feature vectors that the server can process. Figure 5 This is a schematic diagram illustrating preprocessing of audio data to be processed, provided in an embodiment of this application. First, the audio data to be processed can be pre-emphasized to eliminate low-frequency interference and enhance the spectrum of the high-frequency components, making the signal flatter. Then, the pre-emphasized audio data can be segmented into frames, dividing it into small segments called "frames." Simultaneously, windowing can be applied, multiplying each frame of audio data by a Hamming window to increase the continuity between previous and subsequent frames. Finally, a Fast Fourier Transform can be performed on each frame of windowed audio data to obtain the spectrum of each frame. Then, the Mel filter bank can be used to smooth the spectrum of each frame of audio data to be processed, eliminate harmonics, highlight the formants of the original audio data to be processed, and then perform logarithmic operations and discrete cosine transforms on it to obtain 13 first-order differences and 13 second-order differences, so as to further concentrate the signal energy of the audio data to be processed and improve the effect of feature extraction. Figure 6This is a schematic diagram of an emotion recognition method provided in an embodiment of this application. After preprocessing the audio data to be processed, the preprocessed audio data can be input into a deep neural network model for variational autoencoding and gender identification based on the pitch of the audio. This reduces data dimensionality, training parameters, and ensures that the hidden layer can effectively extract features from the preprocessed audio data. Then, feature optimization and prediction can be performed using a bidirectional long short-term memory (BiLSTM) model based on an attention mechanism. Figure 7 This is a schematic diagram of the structure of a single neuron in a long short-term memory model provided in an embodiment of this application. Figure 8 This is a schematic diagram of a long short-term memory model provided in an embodiment of this application. The information input after passing through the previous neuron includes C. t-1 and a t-1 a t-1 With input information X t The stacked parameters are copied into four copies, each multiplied by a weight parameter and then activated using α. The first copy of the parameters is multiplied by the weight coefficient W of the forget gate. f Then use α = 1 / (1 + e) －t Activation is performed by mapping values ​​to (0,1), and most values ​​are very close to 0 and 1, thus retaining the information C carried by the previous neuron. t-1 After multiplication, selective forgetting (multiplying by numbers close to 0) and passing (multiplying by numbers close to 1) occurs. The input gate is for a. t-1 Multiplied by weight W i The result of the forward operation on the information activated by α = tanh is restricted, and the processed information is added one by one to the information processed by the forget gate to obtain C. t This allows for the simultaneous input of both old and new information from the preceding context. The output gate restricts the content to be output by the current neuron; the preceding information, after activation by α = tanh, is then combined with the output weight W. o Element-wise multiplication yields the output of the current neuron and the input information 'a' passed to the next neuron. t Thus, C can be used. t It can carry more information from the preceding text, has long-term memory, a tIt can contain more current information and has short-term memory. To simultaneously extract common information from preceding and following segments of the audio data to be processed, a reverse operation can be added to the LSTM, inverting the input information and recalculating the output. The final result can be a stack of forward and reversed values, thus enabling the LSTM to perform joint analysis of preceding and following time segments. Optionally, the LSTM can be stacked with four layers, and then the obtained data can be classified using the Support Vector Machine algorithm to obtain the probability value of each emotion and output the prediction result. In practical applications, emotion categories can be divided into categories such as happy, angry, neutral, and sad.

[0086] Since dangerous behaviors inside a vehicle are often accompanied by the sounds of a mobile phone or other objects, emotion recognition can be performed using a bidirectional long short-term memory model based on an attention mechanism and a support vector machine algorithm. This adds auditory judgment analysis to the single visual judgment analysis, which can reduce the false judgment rate and improve the reliability of behavior recognition.

[0087] In this embodiment, the server can perform feature extraction processing on the audio data to be processed to obtain a speech feature vector sequence, wherein two adjacent speech feature vectors in the speech feature vector sequence partially overlap. Then, speech recognition processing can be performed on the speech feature vector sequence to obtain a sentence sequence, and target words can be determined from each sentence in the sentence sequence. Based on the target words in each sentence of the sentence sequence, the text information of the object can be determined.

[0088] In some possible implementations, after acquiring the audio data to be processed, physical methods can be used to preprocess the audio data to improve its volume and quality. Mel-frequency cepstral coefficients can be used for feature extraction, converting the audio data into a sequence of speech feature vectors that the server can process. The specific preprocessing method is the same as the method used to preprocess the audio data in determining the emotion information of the object described above, and will not be repeated here. After obtaining the speech feature vector sequence, it can be converted into text information. The features extracted by Mel-frequency cepstral coefficients can be input into a three-layer one-dimensional or two-dimensional convolution, then passed through a recurrent neural network (Vanilla RNN), a gated neural network (GRU), or a bidirectional recurrent neural network (Bi directional RNN), followed by a lookahead-convolution, and finally output as text information through a fully connected layer. Then, Connectionist Temporal Classification (CTC) is used to address the problem of the input and output sequences not being able to correspond one-to-one, eliminating the need for repeated iterative speech alignment. Finally, a regression function can be used to classify the text, obtaining the text information of the object. In practical applications, dangerous words such as "alarm", "hit", and "kill" can be predefined. The words can be compared with the obtained text information using regular expression matching. The weight ratio of each statement containing dangerous words can be output, and the statement with the highest probability value can be selected as the prediction result.

[0089] Since dangerous behaviors inside a vehicle are often accompanied by the sounds of a mobile phone or other objects, text recognition using a bidirectional long short-term memory model based on an attention mechanism and a support vector machine algorithm, by adding auditory judgment and analysis to the single visual judgment and analysis, can reduce the false judgment rate and improve the reliability of behavior recognition.

[0090] S209: Determine the behavioral attributes of an object based on profile information, action information, emotion information, and text information.

[0091] In this embodiment, the server can determine a first parameter based on portrait information, action information, emotion information, text information, and a first attribute information, and can determine a second parameter based on portrait information, action information, emotion information, text information, and a second attribute information. Then, the object's behavioral attributes can be determined based on the first and second parameters.

[0092] In some possible implementations, the Naive Bayes algorithm can be used for backend evaluation. This is a backend decision-making method based on prediction results from different modalities. It can comprehensively score the different pattern classifications obtained previously, form a decision recognition matrix, and determine whether the object's behavior is dangerous. In practical applications, the object's behavioral attributes can include dangerous and non-dangerous behaviors, which can be represented by ω, respectively. k (k = 1, 2) represents the sample space. The sample space includes the object's action information, emotion information, text information, and profile information. The total dimension can be the sum of multiple types of information, specifically represented as: X = [x1, ..., x...]. N ], where N can be the sum of information, and then the formula F(ω) = maxP(ω) can be obtained. k |x1,...,x N The algorithm calculates whether the behavioral information under different modalities constitutes dangerous behavior and selects the value with the highest probability for judgment. According to Naive Bayes theory, F(ω) = maxP(ω) k│ x1,...,x N ) can be represented as max(x1,...,x N │ω k )P(ω k ) / P(x1,...,x N In practical applications, this can be simplified to P(x1│ω). k )×...×P(x N │ω k )×P(ω k Based on this formula, the probabilities of label information, action information, emotion information, and text information being dangerous actions and non-dangerous actions can be determined. Furthermore, the behavioral attribute corresponding to the maximum of the two probability values ​​can be identified as the object's behavioral attribute, and appropriate feedback, such as an alarm, can be generated. By using the Naive Bayes algorithm to fuse profile information, action information, emotion information, and text information, regardless of feature values ​​of different dimensions, computational power can be reduced, and the accuracy of dangerous behavior identification can be increased.

[0093] The behavior attribute recognition method based on multimodal fusion provided in this application determines the profile information of the object based on its tags, age, and gender. This allows for the judgment of dangerous behaviors inside the vehicle to no longer rely solely on instantaneous video and audio data, but to comprehensively consider the object's past behavior and profile, reducing misjudgments of habitual safe actions of drivers and passengers and improving the accuracy of identifying dangerous behaviors inside the vehicle. An open-source system that can directly acquire the joint position information of the human skeleton is designed, eliminating the need for complex image processing preprocessing. Furthermore, a spatiotemporal graph neural network model is used to extract the object's posture data for training and prediction, resulting in stronger expressive power and higher performance in the time dimension. This reduces latency, computational effort, and enhances real-time performance and robustness. Emotion recognition and text recognition are performed using a bidirectional long short-term memory model based on an attention mechanism and a support vector machine algorithm. Adding auditory judgment analysis to the single visual judgment analysis reduces the false judgment rate and improves the reliability of behavior recognition. By using a Naive Bayes algorithm to fuse profile information, action information, emotion information, and text information, independent of feature values ​​in different dimensions, computational effort is reduced, and the accuracy of identifying dangerous behaviors is increased.

[0094] This application also provides a behavior attribute recognition device based on multimodal fusion. Figure 9 This is a schematic diagram of the structure of a behavior attribute recognition device based on multimodal fusion provided in an embodiment of this application, as shown below. Figure 9 As shown, the identification device may include:

[0095] The acquisition module 901 is used to acquire video data to be processed and audio data to be processed; the video data to be processed includes at least one object.

[0096] The first determining module 903 is used to determine the portrait information of the object based on the video data to be processed and the audio data to be processed;

[0097] The second determining module 905 is used to determine the motion information of the object based on the video data to be processed;

[0098] The third determining module 907 is used to determine the emotional information and text information of the object based on the audio data to be processed;

[0099] The fourth determination module 909 is used to determine the behavioral attributes of an object based on portrait information, action information, emotion information, and text information.

[0100] In this embodiment of the application, the first determining module 903 is further configured to determine the facial data of the object from the video data to be processed; determine the voice feature data of the object from the audio data to be processed; determine the label information, age information and gender information of the object based on the facial data and voice feature data; and determine the portrait information of the object based on the label information, age information and gender information.

[0101] In this embodiment of the application, the second determining module 905 is further configured to determine the set of joints corresponding to the object from each frame of video data of the video data to be processed; determine the position data and confidence data of each joint in the set of joints; and determine the action information of the object based on the position data and confidence data of each joint in each frame of video data.

[0102] In this embodiment of the application, the third determining module 907 is further configured to perform feature extraction processing on the audio data to be processed to obtain a speech feature vector sequence; two adjacent speech feature vectors in the speech feature vector sequence partially overlap; determine the attention vector corresponding to each speech feature vector in the speech feature vector sequence; concatenate the attention vector corresponding to each speech feature vector with each speech feature vector to obtain a speech feature vector sequence with attention feature information; and determine the emotional information of the object based on the speech feature vector sequence with attention feature information.

[0103] In this embodiment, the third determining module 907 is further configured to perform feature extraction processing on the audio data to be processed, to obtain a first speech feature vector and a second speech feature vector; the first speech feature vector and the second speech feature vector are adjacent and partially overlap; feature extraction processing is performed on the first speech feature vector to obtain a first speech feature sub-vector and a second speech feature sub-vector; the second speech feature vector and the second speech feature sub-vector are stacked to obtain a stacked speech feature vector; the attention vector set corresponding to the stacked speech feature vector is determined; based on the first speech feature sub-vector, the stacked speech feature vector, and the attention vector set corresponding to the stacked speech feature vector, a first attention speech feature sub-vector and a second attention speech feature sub-vector are determined; based on the first attention speech feature sub-vector and the second attention speech feature sub-vector, the emotional information of the object is determined.

[0104] In this embodiment of the application, the third determining module 907 is further configured to perform feature extraction processing on the audio data to be processed to obtain a speech feature vector sequence; two adjacent speech feature vectors in the speech feature vector sequence partially overlap; perform speech recognition processing on the speech feature vector sequence to obtain a sentence sequence; determine the target word from each sentence in the sentence sequence; and determine the text information of the object based on the target word sentence in each sentence of the sentence sequence.

[0105] In this embodiment of the application, the fourth determining module 909 is used to determine a first parameter based on the portrait information, action information, emotion information, text information and first attribute information; determine a second parameter based on the portrait information, action information, emotion information, text information and second attribute information; and determine the behavioral attributes of the object based on the first parameter and the second parameter.

[0106] The apparatus and method embodiments in this application are based on the same application concept.

[0107] The behavior attribute recognition device based on multimodal fusion provided in this application determines the profile information of the object based on its tags, age, and gender. This allows for the judgment of dangerous behaviors inside the vehicle to no longer rely solely on instantaneous video and audio data, but to comprehensively consider the object's past behavior and profile, reducing misjudgments of habitual safe actions of drivers and passengers and improving the accuracy of identifying dangerous behaviors inside the vehicle. By designing an open-source system that can directly acquire the position information of human skeletal joints, without requiring complex image processing preprocessing, and using a spatiotemporal graph neural network model to extract the object's posture data for training and prediction, it has stronger expressive power and higher performance in the time dimension, reducing latency, computational power, and enhancing real-time performance and robustness. Through a bidirectional long short-term memory model based on an attention mechanism and a support vector machine algorithm for emotion recognition and text recognition, auditory judgment analysis is added to the single visual judgment analysis, reducing the false judgment rate and improving the reliability of behavior recognition. By using a Naive Bayes algorithm to fuse profile information, action information, emotion information, and text information, independent of feature values ​​of different dimensions, computational power can be reduced and the accuracy of identifying dangerous behaviors can be increased.

[0108] This application also provides an electronic device that can be installed in a server to store at least one instruction, at least one program, code set, or instruction set related to implementing a behavior attribute recognition method based on multimodal fusion in the method embodiment. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the memory to implement the aforementioned behavior attribute recognition method based on multimodal fusion.

[0109] This application embodiment also provides a storage medium, which can be set in a server to store at least one instruction, at least one program, code set or instruction set related to implementing a behavior attribute recognition method based on multimodal fusion in the method embodiment. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the above-mentioned behavior attribute recognition method based on multimodal fusion.

[0110] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Alternatively, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), portable hard drives, magnetic disks, or optical disks.

[0111] As can be seen from the embodiments of the behavior attribute recognition method, apparatus, electronic device, or storage medium based on multimodal fusion provided in this application, the recognition method in this application includes acquiring video data to be processed and audio data to be processed; the video data to be processed includes at least one object; determining the object's profile information based on the video data to be processed and the audio data to be processed; determining the object's action information based on the video data to be processed; determining the object's emotional information and text information based on the audio data to be processed; and determining the object's behavior attributes based on the profile information, action information, emotional information, and text information. Based on the embodiments of this application, by determining the object's profile information based on the object's tags, age, and gender, the judgment of dangerous behaviors inside the vehicle no longer relies solely on instantaneous video and audio data, but comprehensively considers the object's past behaviors and profile, which can reduce misjudgments of habitual safe actions of drivers and passengers and improve the accuracy of identifying dangerous actions inside the vehicle. By designing an open-source system that can directly acquire the joint position information of the human skeleton, without the need for complex image processing preprocessing, and using a spatiotemporal graph neural network model to extract the object's pose data for training and prediction, it has stronger expressive power and higher performance in the time dimension, which can reduce latency, reduce computing power, and enhance real-time performance and robustness. By employing a bidirectional long short-term memory model based on an attention mechanism and a support vector machine algorithm for emotion and text recognition, auditory judgment analysis is added to the single visual judgment analysis, which can reduce the false positive rate and improve the reliability of behavior recognition. Furthermore, by using the Naive Bayes algorithm to fuse profile, action, emotion, and text information, independent of feature values ​​of different dimensions, computational power can be reduced while increasing the accuracy of identifying dangerous behaviors.

[0112] In this invention, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the connection within two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0113] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, while this specification describes specific embodiments, other embodiments are also within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in the order shown in different embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific order or sequence of connections to achieve the desired results; in some implementations, parallel processing of multiple tasks is possible or may be advantageous.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the embodiments of the apparatus are described simply because they are based on similarities to the method embodiments; relevant parts can be referred to the descriptions of the method embodiments.

[0115] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for recognizing behavior attributes based on multi-modal fusion, characterized in that, The method comprises: obtaining video data to be processed from a vehicle interior camera and audio data to be processed from a vehicle-mounted recording device; the video data to be processed comprises at least one object; from the video data to be processed, face data of the object is determined; the face data of each object is labeled and matched according to a face recognition system, historical behavior data of each object is obtained, and label information of the object is determined according to the historical behavior data; the label information represents the object's tendency to make dangerous actions in the vehicle; according to the label information and the audio data to be processed, portrait information of the object is determined; according to the video data to be processed, action information of the object is determined; the action information includes mobile phone typing, door opening, yawning, and fighting with people in the vehicle; according to the audio data to be processed, emotion information and text information of the object are determined; the emotion information includes happy, angry, neutral, and sad; the text information is the sentence with the maximum weight proportion of dangerous words obtained through voice recognition processing; from a sample space, the probability of the portrait information appearing under the condition of dangerous actions, the probability of the action information appearing, the probability of the emotion information appearing, the probability of the text information appearing, and the probability of dangerous actions in the sample space are determined, and multiplied to obtain a first parameter; the first parameter represents the probability of dangerous actions in the vehicle under the condition that the portrait information, the action information, the emotion information, and the text information appear simultaneously; from a sample space, the probability of the portrait information appearing under the condition of non-dangerous actions, the probability of the action information appearing, the probability of the emotion information appearing, the probability of the text information appearing, and the probability of non-dangerous actions in the sample space are determined, and multiplied to obtain a second parameter; the second parameter represents the probability of non-dangerous actions in the vehicle under the condition that the portrait information, the action information, the emotion information, and the text information appear simultaneously; according to the first parameter and the second parameter, behavior attributes of the object are determined; the behavior attributes include dangerous behavior and non-dangerous behavior.

2. The method of claim 1, wherein, The method further comprises: from the audio data to be processed, voice feature data of the object is determined; according to the face data and the voice feature data, age information and gender information of the object are determined; according to the label information, the age information, and the gender information, portrait information of the object is determined.

3. The method of claim 1, wherein, The method further comprises: from each frame of video data of the video data to be processed, a set of joint nodes corresponding to the object is determined; position data and confidence data of each joint node in the set of joint nodes are determined; according to the position data and confidence data of each joint node in each frame of video data, action information of the object is determined.

4. The method of claim 1, wherein, The method further comprises: feature extraction processing is performed on the audio data to be processed to obtain a voice feature vector sequence; adjacent two voice feature vectors in the voice feature vector sequence partially overlap; determine an attention vector corresponding to each speech feature vector in the sequence of speech feature vectors; concatenate the attention vector corresponding to each speech feature vector with the speech feature vector to obtain a sequence of speech feature vectors with attention characteristic information; determine the emotional information of the object according to the sequence of speech feature vectors with attention characteristic information.

5. The method of claim 1, wherein, The method for determining the emotional information of the object according to the to-be-processed audio data comprises: performing feature extraction processing on the to-be-processed audio data to obtain a first speech feature vector and a second speech feature vector; the first speech feature vector and the second speech feature vector are adjacent and partially overlap; performing feature extraction processing on the first speech feature vector to obtain a first speech feature sub-vector and a second speech feature sub-vector; stacking the second speech feature vector and the second speech feature sub-vector to obtain a stacked speech feature vector; determining a set of attention vectors corresponding to the stacked speech feature vector; determining a first attention speech feature sub-vector and a second attention speech feature sub-vector according to the first speech feature sub-vector, the stacked speech feature vector, and the set of attention vectors corresponding to the stacked speech feature vector; determining the emotional information of the object according to the first attention speech feature sub-vector and the second attention speech feature sub-vector.

6. The method of claim 1, wherein, The method for determining the text information of the object according to the to-be-processed audio data comprises: performing feature extraction processing on the to-be-processed audio data to obtain a sequence of speech feature vectors; two adjacent speech feature vectors in the sequence of speech feature vectors partially overlap; performing speech recognition processing on the sequence of speech feature vectors to obtain a sequence of sentences; determining a target word from each sentence in the sequence of sentences; determining the text information of the object according to the target word in each sentence in the sequence of sentences.

7. A behavior attribute recognition device based on multi-modal fusion, characterized by, The method comprises: an acquisition module configured to acquire to-be-processed video data and to-be-processed audio data; the to-be-processed video data comprises at least one object; a first determination module configured to determine face data of the object from the to-be-processed video data; annotating and matching the face data of each object by using a face recognition system to obtain historical behavior data of each object, and determining label information of the object according to the historical behavior data; the label information represents a tendency of the object to make a dangerous action in a vehicle; determining portrait information of the object according to the label information and the to-be-processed audio data; a second determination module configured to determine action information of the object according to the to-be-processed video data; a third determination module configured to determine emotional information and text information of the object according to the to-be-processed audio data; a fourth determining module, configured to determine, from the sample space, a probability of occurrence of the image information, a probability of occurrence of the action information, a probability of occurrence of the emotion information, a probability of occurrence of the text information, and a probability of occurrence of dangerous action in the sample space, and multiply the probabilities to obtain a first parameter, the first parameter representing a probability of occurrence of dangerous action in the vehicle under a condition that the image information, the action information, the emotion information, and the text information occur simultaneously; determine, from the sample space, a probability of occurrence of the image information, a probability of occurrence of the action information, a probability of occurrence of the emotion information, a probability of occurrence of the text information, and a probability of occurrence of non-dangerous action in the sample space, and multiply the probabilities to obtain a second parameter, the second parameter representing a probability of occurrence of non-dangerous action in the vehicle under a condition that the image information, the action information, the emotion information, and the text information occur simultaneously; determine, according to the first parameter and the second parameter, a behavior attribute of the object, the behavior attribute including dangerous behavior and non-dangerous behavior.

8. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the behavior attribute recognition method based on multi-modal fusion according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the behavior attribute recognition method based on multi-modal fusion according to any one of claims 1-6.

Images