Method and device for recognizing facial emotion of child in cabin

Abstract

The invention relates to the technical field of emotion recognition, in particular to a method and device for recognizing the facial emotion of a child in a cabin, and the method comprises the steps: collecting and preprocessing an initial data set of a child facial image, and obtaining a child facial image data set meeting the requirements of a preset model architecture; inputting the child face image data set into a preset ViT model, and extracting visual features; and on the basis of the visual features, the text description and the randomly initialized query vector, performing fine tuning on the Q-Former model until the model meets a preset stability requirement in the child facial image recognition task, obtaining a child facial emotion recognition model, and recognizing a child facial emotion result in the vehicle cabin by using the child facial emotion recognition model. Therefore, the problems that the recognition accuracy of the child emotion is low and the interpretability is poor due to the fact that related technologies lack cognitive and emotion perception ability and semantic understanding ability for child groups and model training depends on large-scale annotation data and is sensitive to environmental changes are solved.

Description

Technical Field

[0001] This application relates to the field of emotion recognition technology, and in particular to a method and device for recognizing children's facial emotions in a cockpit. Background Technology

[0002] In recent years, with the continuous development of computer technology, intelligent cockpit systems have gradually come into the public eye. As an important component of modern automobiles that enhances the driving experience and creates personalized automotive services, intelligent cockpit systems have significant research value. Current technologies have enabled intelligent cockpit systems to achieve multimodal interaction, including voice and vision, and are beginning to shift from passively responding to commands to proactively perceiving needs. Among these technologies, emotion recognition, as an indispensable part of intelligent cockpit services, has also achieved significant progress.

[0003] Among related technologies, emotion recognition technology based on Convolutional Neural Network (CNN) has dominated the market and is an important milestone in the field of affective computing. Its achievements are reflected in improved accuracy, technology universality and multi-field applications, which have promoted emotion recognition from the laboratory to real-world scenarios.

[0004] However, despite significant progress in existing emotion recognition technologies, they still have obvious shortcomings in meeting the needs of customized services: First, existing systems lack the ability to recognize and perceive the emotions of children. Children often exhibit stronger and more direct emotional fluctuations during rides. Compared to adults, children's emotional information is more directly reflected in their facial expressions. Furthermore, children's facial features, such as facial proportions and muscle movement amplitude, differ significantly from those of adults. Existing models are mostly trained based on adult data, resulting in a low accuracy rate in recognizing children's emotions.

[0005] Secondly, traditional emotion recognition methods are mainly based on convolutional neural networks or shallow machine learning models. Their training relies on large-scale labeled data and is sensitive to changes in the environment such as lighting and angle. However, low light conditions often occur inside the cabin while the vehicle is in motion. Such complex cabin conditions bring unique challenges to emotion recognition.

[0006] Third, existing emotion recognition systems lack semantic understanding capabilities, which poses a challenge to the interpretability of emotion recognition tasks. Traditional convolutional neural network-based emotion recognition techniques are highly effective for outputting discrete emotion labels, but they cannot delve into the reasons for classification, resulting in poor interpretability.

[0007] Therefore, there is an urgent need for a new emotion recognition method to overcome one or more of the aforementioned technical problems. Summary of the Invention

[0008] This application provides a method and device for recognizing children's facial emotions in a cockpit, in order to solve the problems of low accuracy and poor interpretability in recognizing children's emotions due to the lack of cognitive and emotional perception and semantic understanding capabilities of related technologies, and the fact that model training relies on large-scale labeled data and is sensitive to environmental changes.

[0009] The first aspect of this application provides a method for recognizing children's facial emotions in a vehicle cabin, comprising the following steps: acquiring an initial dataset of children's facial images and preprocessing the initial dataset to obtain a children's facial image dataset that meets the requirements of a preset model architecture; inputting the children's facial image dataset into a preset ViT model to extract visual features; and fine-tuning the Q-Former model based on the visual features, text descriptions, and randomly initialized query vectors until the model reaches a preset stability requirement in the children's facial image recognition task, thereby obtaining a children's facial emotion recognition model, which is then used to recognize the facial emotions of children in the vehicle cabin.

[0010] Through the aforementioned technical means, this application embodiment can collect and preprocess a dataset of children's facial images, uniformly converting the original images into a fixed size and pixel distribution required by the VIT model, ensuring that the data can be correctly input into the network architecture; then, the VIT model is used to extract the visual features of the images to overcome the limitations of the local receptive field of traditional CNNs and improve the model's adaptability to the complex environment inside the vehicle; furthermore, the Q-Former model is fine-tuned based on a dual supervision mechanism to optimize the model's cross-modal understanding ability, thereby using the fine-tuned Q-Former model as a bridge to align and fuse visual features with text descriptions, converting visual features into semantic representations, and effectively advancing the recognition of emotions on children's faces.

[0011] Optionally, in one embodiment of this application, the preprocessing of the initial dataset to obtain a children's facial image dataset that meets the requirements of a preset model architecture includes: integrating the dataset information of the initial dataset into a metadata file in a target format, such that each data record is stored in a key-value pair structure to obtain a complete dataset; and dividing the complete dataset according to a preset ratio to obtain a training set and a test set, which are used to train model parameters, test model performance, and evaluate model generalization ability, respectively.

[0012] Through the above-mentioned technical means, the embodiments of this application can encapsulate each data record of the initial children's facial image dataset into a target format metadata file by integrating dataset information and standardizing metadata format. This enables unified organization and complete storage of data information, and finally outputs a children's facial image dataset that meets the input requirements of the preset model architecture, has standardized data organization, and can be directly used for model training. This can effectively reduce the data preprocessing cost of subsequent model adaptation and improve the convenience and compatibility of data use.

[0013] Optionally, in one embodiment of this application, the preprocessing of the initial dataset to obtain a children's facial image dataset that meets the requirements of a preset model architecture further includes: performing scale standardization processing on all images in the training set and the test set to obtain images of a target size, and performing normalization processing, wherein the normalization formula for the images is: , in, This represents the mean vector for each channel. This represents the standard deviation vector of each channel. Represents a tensor.

[0014] Through the above-mentioned technical means, the embodiments of this application can perform scale normalization processing on all images in the training set and the test set, uniformly adjust the images to the target size, and complete the pixel value normalization operation. This can eliminate the scale difference and numerical range difference between different images, and output image data with consistent size and standardized values. This can provide the model with a unified input format and stable data distribution, effectively improve the convergence speed and generalization ability of model training, and ensure that the model can focus on core feature learning rather than data format adaptation.

[0015] Optionally, in one embodiment of this application, the preprocessing of the initial dataset to obtain a children's facial image dataset that meets the requirements of a preset model architecture further includes: generating multiple text descriptions based on the annotations of each image in the training set, wherein the multiple text descriptions include emotion tags and corresponding facial action units and contextual semantic information.

[0016] Through the above-mentioned technical means, the embodiments of this application can generate multiple text descriptions containing emotion labels, corresponding facial action units and contextual semantic information based on the image annotations of the training set. This achieves a deep association between image visual features and multi-dimensional text information, which can not only retain the key features of the original annotations, but also expand the representation dimension of the data through rich semantic descriptions, providing the model with a more comprehensive cross-modal learning basis, and helping to improve the model's ability to understand and reason about children's facial emotions and related features.

[0017] Optionally, in one embodiment of this application, the fine-tuning of the Q-Former model includes: feeding the query vector with image information output by the Q-Former model into a fully connected layer for linear projection to obtain the projected vector, so as to align the Q-Former output dimension with the embedding dimension of the large language model; using the projected vector sequence as pseudo-word embeddings and concatenating them to the word embedding front end of the task prompt to form a complete multimodal input sequence; and inputting the multimodal input sequence into the large language model to generate complete emotion description text and emotion classification labels.

[0018] Through the above-mentioned technical means, the embodiments of this application can align the visual feature vector output by Q-Former with the embedding space of the large language model through linear projection, and can further embed and concatenate the projected vector as a visual prefix with text prompt words to construct a unified multimodal input sequence. This fusion mechanism enables the subsequent large language model to understand visual semantics and text instructions at the same time, and finally realize the end-to-end generation of structured emotion description text and classification labels, completing the leap from image features to multimodal emotion understanding.

[0019] Optionally, in one embodiment of this application, the fine-tuning of the Q-Former model further includes: comparing and learning the query vector with the pre-processed text feature representation.

[0020] Through the aforementioned technical means, this embodiment of the application can learn by comparing query vectors with text features, thereby enabling the model to approximate matching image-text representations and distancing irrelevant representations in a shared semantic space, thus effectively aligning visual and linguistic modalities. This process allows query vectors to learn to capture discriminative visual features highly correlated with linguistic descriptions, enhancing their semantic representation capabilities, providing high-quality visual context for subsequent multimodal tasks, and ultimately improving the performance of large language models in understanding image content and generating accurate descriptions.

[0021] A second aspect of this application provides a child facial emotion recognition device for use in a vehicle cabin, comprising: a data acquisition module for acquiring an initial dataset of child facial images and preprocessing the initial dataset to obtain a child facial image dataset that meets the requirements of a preset model architecture; an extraction module for inputting the child facial image dataset into a preset ViT model to extract visual features; and a recognition module for fine-tuning a Q-Former model based on the visual features, text description, and randomly initialized query vectors until the model reaches a preset stability requirement in the child facial image recognition task, thereby obtaining a child facial emotion recognition model, which is used to recognize the facial emotions of children in the vehicle cabin.

[0022] Through the aforementioned technical means, this application embodiment can collect and preprocess a dataset of children's facial images, uniformly converting the original images into a fixed size and pixel distribution required by the VIT model, ensuring that the data can be correctly input into the network architecture; then, the VIT model is used to extract the visual features of the images to overcome the limitations of the local receptive field of traditional CNNs and improve the model's adaptability to the complex environment inside the vehicle; furthermore, the Q-Former model is fine-tuned based on a dual supervision mechanism to optimize the model's cross-modal understanding ability, thereby using the fine-tuned Q-Former model as a bridge to align and fuse visual features with text descriptions, converting visual features into semantic representations, and effectively advancing the recognition of emotions on children's faces.

[0023] Optionally, in one embodiment of this application, the acquisition module includes: an integration unit, used to integrate the dataset information of the initial dataset into a metadata file in a target format, so that each data record is stored in a key-value pair structure to obtain a complete dataset; and a partitioning unit, used to partition the complete dataset according to a preset ratio to obtain a training set and a test set, which are used to train model parameters, test model performance, and evaluate model generalization ability, respectively.

[0024] Through the above-mentioned technical means, the embodiments of this application can encapsulate each data record of the initial children's facial image dataset into a target format metadata file by integrating dataset information and standardizing metadata format. This enables unified organization and complete storage of data information, and finally outputs a children's facial image dataset that meets the input requirements of the preset model architecture, has standardized data organization, and can be directly used for model training. This can effectively reduce the data preprocessing cost of subsequent model adaptation and improve the convenience and compatibility of data use.

[0025] Optionally, in one embodiment of this application, the acquisition module further includes: a processing unit, configured to perform scale normalization processing on all images in the training set and the test set to obtain images of a target size, and to perform normalization processing, wherein the normalization formula for the images is: , in, This represents the mean vector for each channel. This represents the standard deviation vector of each channel. Represents a tensor.

[0026] Through the above-mentioned technical means, the embodiments of this application can perform scale normalization processing on all images in the training set and the test set, uniformly adjust the images to the target size, and complete the pixel value normalization operation. This can eliminate the scale difference and numerical range difference between different images, and output image data with consistent size and standardized values. This can provide the model with a unified input format and stable data distribution, effectively improve the convergence speed and generalization ability of model training, and ensure that the model can focus on core feature learning rather than data format adaptation.

[0027] Optionally, in one embodiment of this application, the acquisition module further includes: a first generation unit, configured to generate multiple text descriptions based on the annotations of each image in the training set, wherein the multiple text descriptions include emotion tags and corresponding facial action units and contextual semantic information.

[0028] Through the above-mentioned technical means, the embodiments of this application can generate multiple text descriptions containing emotion labels, corresponding facial action units and contextual semantic information based on the image annotations of the training set. This achieves a deep association between image visual features and multi-dimensional text information, which can not only retain the key features of the original annotations, but also expand the representation dimension of the data through rich semantic descriptions, providing the model with a more comprehensive cross-modal learning basis, and helping to improve the model's ability to understand and reason about children's facial emotions and related features.

[0029] Optionally, in one embodiment of this application, the recognition module includes: an alignment unit, used to feed the query vector with image information output by the Q-Former model into a fully connected layer for linear projection to obtain the projected vector, so as to align the Q-Former output dimension with the embedding dimension of the large language model; a concatenation unit, used to embed the projected vector sequence as pseudo-words and concatenate it to the word embedding front end of the task prompt to form a complete multimodal input sequence; and a second generation unit, used to input the multimodal input sequence into the large language model to generate complete emotion description text and emotion classification labels.

[0030] Through the above-mentioned technical means, the embodiments of this application can align the visual feature vector output by Q-Former with the embedding space of the large language model through linear projection, and can further embed and concatenate the projected vector as a visual prefix with text prompt words to construct a unified multimodal input sequence. This fusion mechanism enables the subsequent large language model to understand visual semantics and text instructions at the same time, and finally realize the end-to-end generation of structured emotion description text and classification labels, completing the leap from image features to multimodal emotion understanding.

[0031] Optionally, in one embodiment of this application, the identification module further includes: a learning unit, used to compare and learn the query vector with the pre-processed text feature representation.

[0032] Through the aforementioned technical means, this embodiment of the application can learn by comparing query vectors with text features, thereby enabling the model to approximate matching image-text representations and distancing irrelevant representations in a shared semantic space, thus effectively aligning visual and linguistic modalities. This process allows query vectors to learn to capture discriminative visual features highly correlated with linguistic descriptions, enhancing their semantic representation capabilities, providing high-quality visual context for subsequent multimodal tasks, and ultimately improving the performance of large language models in understanding image content and generating accurate descriptions.

[0033] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the in-cabin child facial emotion recognition method as described in the above embodiments.

[0034] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described in-cabin child facial emotion recognition method.

[0035] A fifth aspect of this application provides a computer program product that stores a computer program that, when executed by a processor, implements the above-described in-cabin child facial emotion recognition method.

[0036] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0037] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a method for recognizing a child's facial emotions in a cockpit according to an embodiment of this application. Figure 2 This is a flowchart illustrating the fine-tuning of a Q-Former model according to a specific embodiment of this application; Figure 3 This is a basic flowchart of a cockpit child facial emotion recognition method according to a specific embodiment of this application; Figure 4 This is a schematic diagram of the structure of an in-cabin child facial emotion recognition device according to an embodiment of this application; Figure 5This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0039] The following describes a method and apparatus for recognizing children's facial emotions in a cockpit, based on embodiments of this application, with reference to the accompanying drawings. Addressing the issues mentioned in the background section regarding the lack of cognitive and emotional perception and semantic understanding capabilities for children, and the reliance on large-scale labeled data for model training and sensitivity to environmental changes, resulting in low accuracy and poor interpretability in recognizing children's emotions, this application provides a method for recognizing children's facial emotions in a cockpit. This method involves collecting a dataset of children's facial images, preprocessing them, and then using a VIT (Vision Transformer) model to extract visual features. A finely tuned Q-Former (Query Transformer) model is then used to align and fuse the visual features with textual descriptions. The Q-Former acts as a bridge, converting the visual features into semantic representations, which are then input into a large language model to generate emotion labels and textual descriptions, thus completing the recognition of children's facial emotions. This method is particularly suitable for monitoring and responding to children's emotional states in smart cockpit scenarios. This solves the problems of low accuracy and poor interpretability in recognizing children's emotions due to the lack of cognitive and emotional perception and semantic understanding capabilities of related technologies, the reliance on large-scale labeled data for model training and sensitivity to environmental changes.

[0040] Specifically, Figure 1 This is a flowchart illustrating a method for recognizing children's facial emotions in a cockpit, as provided in an embodiment of this application.

[0041] like Figure 1 As shown, the method for recognizing children's facial emotions in the cockpit includes the following steps: In step S101, an initial dataset of children's facial images is collected, and the initial dataset is preprocessed to obtain a children's facial image dataset that meets the requirements of the preset model architecture.

[0042] In actual implementation, this application embodiment can construct a multimodal children's facial emotion dataset to meet the needs of children's facial emotion recognition and classification; the dataset can cover four core facial emotion states: Neutral, Happy, Sad, and Angry.

[0043] The preset model architecture can adopt the ViT model architecture, and the specific model can be selected as the EVA-CLIP model. Based on the input requirements of the ViT model (EVA-CLIP), a systematic preprocessing process is implemented on the original image data.

[0044] ViT is an image feature extraction model based on the Transformer architecture. Its core is to capture global features by converting images into sequential data and then using a self-attention mechanism, which completely breaks the limitation of traditional CNNs that rely on local receptive fields.

[0045] The EVA-CLIP model is a contrastive learning-based visual-language multimodal model optimized from the ViT architecture. With ViT as the core visual backbone, it solves the problems of high training cost and poor stability of the traditional CLIP model through adaptive modifications and innovative training strategies.

[0046] Specifically, this application embodiment can collect a considerable number of real-world children's facial images through compliant channels to ensure data diversity in terms of gender, lighting conditions, and background environment. Furthermore, data augmentation processing can be performed on the data to enhance the generalization ability of subsequent models and avoid excessive data in extreme cases or of the same category, thus preventing severe distortion of decision boundaries and a decline in generalization ability.

[0047] Optionally, in one embodiment of this application, the initial dataset is preprocessed to obtain a children's facial image dataset that meets the requirements of a preset model architecture. This includes: integrating the dataset information of the initial dataset into a metadata file in a target format, so that each data record is stored in a key-value pair structure to obtain a complete dataset; and dividing the complete dataset according to a preset ratio to obtain a training set and a test set, which are used to train model parameters, test model performance, and evaluate model generalization ability, respectively.

[0048] As a feasible approach, to meet the model's requirement for consistency in image data and annotation information, and to facilitate data reading and indexing during subsequent model training, this embodiment integrates dataset information into a JSON-formatted metadata file. This file stores each data record in a key-value pair structure, with core fields including: a unique image identifier (image_id), an image storage path (image_path), the corresponding emotion category label (emotion_label), and multiple text descriptions generated in subsequent steps. This approach greatly improves the flexibility and scalability of data management.

[0049] Furthermore, to conduct effective model training and performance evaluation, the complete dataset after the above preprocessing is randomly divided into a training set and a test set according to a predetermined ratio. The preferred division ratio for the dataset is 8:2, that is, 80% of the data is used to train the model parameters and 20% of the data is used to test the model performance and evaluate the model's generalization ability. This division strategy can effectively ensure the reliability and statistical significance of the evaluation results while ensuring that the model learns fully.

[0050] Through the above-mentioned technical means, the embodiments of this application can encapsulate each data record of the initial children's facial image dataset into a target format metadata file by integrating dataset information and standardizing metadata format. This enables unified organization and complete storage of data information, and finally outputs a children's facial image dataset that meets the input requirements of the preset model architecture, has standardized data organization, and can be directly used for model training. This can effectively reduce the data preprocessing cost of subsequent model adaptation and improve the convenience and compatibility of data use.

[0051] Optionally, in one embodiment of this application, preprocessing the initial dataset to obtain a children's facial image dataset that meets the requirements of a preset model architecture further includes: performing scale standardization processing on all images in the training and test sets to obtain images of a target size, and performing normalization processing, wherein the normalization formula for the images is: , in, Represents the mean vector of each channel. Represents the standard deviation vector of each channel. Represents a tensor.

[0052] To meet the input requirements of the ViT model, embodiments of this application can perform scale normalization on all images in the dataset; for example, the image size can be normalized to [specific value]. Pixels can adapt to the input requirements of the model and achieve a good balance between computational efficiency and feature preservation.

[0053] Based on this, in order to conform to the numerical specifications of the input data in the EVA-CLIP model, the image after size standardization needs to be converted into tensor form and normalized. Through tensor transformation operation, the image data is linearly mapped from the pixel value domain [0, 255] to the floating-point domain [0.0, 1.0]. This mapping is achieved by dividing each pixel value by the scalar coefficient 255 and completing the structural reorganization of the data dimension from (height, width, channel) to (channel, height, width), and then the tensor is normalized.

[0054] Furthermore, the standard statistics of the ImageNet dataset are adopted: the mean vector [0.485, 0.456, 0.406] and the standard deviation vector [0.229, 0.224, 0.225], which correspond to the RGB three color channels respectively. Through this normalization operation, the numerical distribution of the input tensor can be adjusted to approximate a standard normal distribution with a mean of zero and a standard deviation of one.

[0055] Through the above-mentioned technical means, the embodiments of this application can perform scale normalization processing on all images in the training set and the test set, uniformly adjust the images to the target size, and complete the pixel value normalization operation. This can eliminate the scale difference and numerical range difference between different images, and output image data with consistent size and standardized values. This can provide the model with a unified input format and stable data distribution, effectively improve the convergence speed and generalization ability of model training, and ensure that the model can focus on core feature learning rather than data format adaptation.

[0056] Optionally, in one embodiment of this application, the initial dataset is preprocessed to obtain a children's facial image dataset that meets the requirements of a preset model architecture. The preprocessing further includes generating multiple text descriptions based on the annotations of each image in the training set, wherein the multiple text descriptions include emotion labels and corresponding facial action units and contextual semantic information.

[0057] Given that training the Q-Former model in the subsequent processes of this application embodiment requires visual-language alignment learning using image-text pairs, text descriptions are indispensable as crucial supervision signals at this stage. They provide the model with a learning objective: how to extract semantic information from images and convert it into a format understandable by the language model. Therefore, this application embodiment manually annotates each image in the training set with three text descriptions. These descriptions not only include core emotion labels but also cover details of specific facial action units that evoke the emotion and contextual semantic information. For example, for an image of a child representing the emotion of "happiness," the corresponding text descriptions could be: "The child's mouth corners are clearly upturned, showing a smile," "The eyes are squinted, wrinkles appear at the corners of the eyes, revealing a joyful emotion," and "The eyebrows are naturally raised, and the overall face presents a relaxed and happy state." This fine-grained text annotation lays a solid foundation for the model to learn more accurate visual-language correspondences. Subsequently, the text descriptions are extracted by a text compiler to generate text feature representations for fine-tuning the Q-Former model.

[0058] Through the above-mentioned technical means, the embodiments of this application can generate multiple text descriptions containing emotion labels, corresponding facial action units and contextual semantic information based on the image annotations of the training set. This achieves a deep association between image visual features and multi-dimensional text information, which can not only retain the key features of the original annotations, but also expand the representation dimension of the data through rich semantic descriptions, providing the model with a more comprehensive cross-modal learning basis, and helping to improve the model's ability to understand and reason about children's facial emotions and related features.

[0059] In step S102, the children's facial image dataset is input into a preset ViT model to extract visual features.

[0060] In scenarios where children's facial image datasets are input into ViT models (such as EVA-CLIP), visual features can be understood as digital vectors / tensors extracted by the model that represent the core information of children's faces. Specifically, they can cover basic physical attributes such as skin color and facial features, local structural combinations such as "eyes + nose" in the middle layer, and hierarchical quantitative expressions of abstract semantic cues such as expressions and age in the high layer. Finally, they are output in the form of image patch sequence vectors or global vectors of classification labels, providing core data support for subsequent semantic transformation and cross-modal tasks.

[0061] Specifically, in this embodiment, the preprocessed standardized tensor can be input into the ViT model (such as the EVA-CLIP model) for visual feature extraction. Throughout the entire process, the ViT model parameters remain frozen and are used only for forward feature extraction.

[0062] The EVA-CLIP model transforms the input tensor into a sequential feature representation through image patch embedding. This process involves key linear projection and dimensionality transformation steps: each pixel patch is flattened and mapped to a high-dimensional embedding space through a learnable linear projection matrix, achieving a transformation from the original pixel space to a deep feature space. This linear transformation not only enhances the expressive power of the features but also provides rich feature representation capacity for subsequent processing through dimensional expansion.

[0063] The ViT model introduces positional encoding and a Transformer encoder structure, achieving global facial expression understanding through a global self-attention mechanism. This overcomes the limitations of local receptive fields in traditional CNNs, enabling comprehensive analysis of coordinated movements across multiple regions such as the eyes, eyebrows, and mouth, providing rich contextual information for emotion recognition. In particular, the EVA-CLIP model, pre-trained with extensive image-text pairs, naturally possesses multimodal understanding capabilities. Its extracted global visual features reside in a high-quality semantic space, supporting efficient visual-linguistic feature alignment, which is naturally compatible with the subsequent Q-Former module.

[0064] Therefore, the embodiments of this application can adapt to the complex changes in children's facial expressions in the vehicle environment and maintain stable emotion recognition performance under challenging conditions such as lighting and posture.

[0065] In step S103, the Q-Former model is fine-tuned based on visual features, text descriptions, and randomly initialized query vectors until the model reaches a preset stability requirement in the child facial image recognition task, thereby obtaining a child facial emotion recognition model, which is then used to identify the facial emotions of children in the vehicle cabin.

[0066] Q-Former (Query Transformer) is a lightweight Transformer module whose core function is to act as a "bridge" between the frozen image encoder (such as CLIP, EVA-CLIP) and the frozen large language model (LLM), and is used in the embodiments of this application to solve the feature alignment problem between visual and language modalities.

[0067] For the fine-tuning process of the Q-Former model, this embodiment employs a multi-level parameter optimization strategy, focusing on targeted training of the learnable query vector, self-attention and cross-attention parameters, feedforward neural network layer parameters, and normalization layer parameters. This fine-tuning process is based on a specific supervised learning framework and aims to optimize the model's cross-modal understanding capability.

[0068] The input to the Q-Former model consists of three key components: visual feature representations extracted by the ViT model, text descriptions, and randomly initialized query vectors. The final output of the model is a set of query vectors that have undergone deep semantic fusion. These vectors act as a bridge connecting the visual and linguistic modalities, carrying effective information from the image and passing it to the next module.

[0069] The model works based on a hierarchical processing mechanism: First, the internally learnable query vectors interact and are enhanced through a bidirectional self-attention mechanism, establishing semantic associations between query vectors. Then, these enhanced query vectors undergo unidirectional cross-attention calculation with the visual feature representations provided by ViT, enabling the query vectors to actively extract relevant information from the visual features. Furthermore, there is no direct cross-attention between the query vectors and the text features. This operation ensures that the visual representation of the query vectors is not "contaminated" by text information before it comes into contact with the text. After the attention calculation, the model introduces nonlinear transformations through a feedforward neural network layer to further enhance the expressive power of the features and promote the deep integration of visual and text information.

[0070] Optionally, in one embodiment of this application, fine-tuning the Q-Former model includes: feeding the query vector with image information output by the Q-Former model into a fully connected layer for linear projection to obtain the projected vector, so as to align the Q-Former output dimension with the embedding dimension of the large language model; using the projected vector sequence as pseudo-word embeddings and concatenating them to the word embedding front end of the task prompt to form a complete multimodal input sequence; and inputting the multimodal input sequence into the large language model to generate complete emotion description text and emotion classification labels.

[0071] like Figure 2 As shown, in order to achieve effective fine-tuning of the model, the embodiments of this application can adopt a dual supervision mechanism, namely main supervision and auxiliary supervision. In model fine-tuning, a dual supervision mechanism of main supervision signal and auxiliary supervision signal is adopted. The core is to anchor the main signal to the core task objective, and the auxiliary signal to make up for the shortcomings of the main signal (such as avoiding overfitting, supplementing detailed features, etc.). Through the collaboration of the two, the model parameters are optimized and the fine-tuning effect is improved.

[0072] Among them, the main supervision signal is the core guiding signal that directly points to the core task objective of the model, while the auxiliary supervision signal is a supplementary guiding signal that makes up for the shortcomings of the main signal. The two work together to optimize the model and improve the fine-tuning effect.

[0073] Regarding the main monitoring signal: The query vector containing image information, output by the Q-Former, is fed into a fully connected layer for linear projection to align the dimensions of the Q-Former output with the embedding dimensions of the subsequent large language model. This projected vector sequence is then used as pseudo-word embeddings and concatenated to the word embedding front end of the task prompt, forming a complete multimodal input sequence. Finally, this sequence is input into the large language model to generate complete sentiment description text and sentiment classification labels. By calculating the text loss function for this task and performing backpropagation, the attention parameters, feedforward layer parameters, and layer normalization parameters in the model can be fine-tuned.

[0074] In the main supervisory signal, the embodiments of this application can use the cross-entropy function as the loss function of the main supervisory signal, and the Q-Former model obtains a set of compact visual semantic representations. The cross-entropy loss function for generating text can be defined as:

[0075] Where T represents the length of the target description text (i.e., the number of tokens). This represents the t-th real token. This represents the token preceding the t-th word, and prompt represents the prompt text.

[0076] Through the above-mentioned technical means, the embodiments of this application can align the visual feature vector output by Q-Former with the embedding space of the large language model through linear projection, and can further embed and concatenate the projected vector as a visual prefix with text prompt words to construct a unified multimodal input sequence. This fusion mechanism enables the subsequent large language model to understand visual semantics and text instructions at the same time, and finally realize the end-to-end generation of structured emotion description text and classification labels, completing the leap from image features to multimodal emotion understanding.

[0077] Optionally, in one embodiment of this application, fine-tuning the Q-Former model further includes: performing comparative learning between the query vector and a pre-processed text feature representation.

[0078] Based on the aforementioned dual supervision mechanism, this embodiment describes the auxiliary supervision, specifically the auxiliary supervision signal: The query vector is compared and learned with the processed text feature representation. Specifically, a contrastive learning loss function is constructed to shorten the distance between semantically matching query-text pairs in the feature space, while distancing mismatched pairs. This auxiliary loss function, together with the main task loss function, guides the model optimization process, thereby more effectively aligning the semantic spaces of the visual and text modalities and fine-tuning the model parameters, including the learnable query vector.

[0079] In the auxiliary supervision signal, this embodiment can use the InfoNCE function as the contrastive learning loss function. The InfoNCE loss function can be defined as follows:

[0080] in, Image features and text features dot product similarity, is a temperature hyperparameter used to adjust the sensitivity of the loss function, and n represents the number of image features and text features.

[0081] Furthermore, the contrastive learning loss function and the generative loss function together constitute a multi-task learning framework, and the total loss function is obtained by weighted summation:

[0082] in , These are the weight parameters.

[0083] Through the above mechanism, the embodiments of this application can achieve integrated modeling of emotion understanding and generation without explicit classification heads. Under the dual guidance of generative feedback and contrastive learning from the language model, Q-Former learns visual representations that are both consistent with emotional semantics and highly compatible with the language space. At the same time, since the large language model remains frozen, the system significantly reduces training costs while fully inheriting the powerful language priors and reasoning capabilities of LLM (Large Language Model), ultimately achieving a unity of high accuracy, high fluency, and strong interpretability in the emotion description generation task.

[0084] Through the aforementioned technical means, this embodiment of the application can learn by comparing query vectors with text features, thereby enabling the model to approximate matching image-text representations and distancing irrelevant representations in a shared semantic space, thus effectively aligning visual and linguistic modalities. This process allows query vectors to learn to capture discriminative visual features highly correlated with linguistic descriptions, enhancing their semantic representation capabilities, providing high-quality visual context for subsequent multimodal tasks, and ultimately improving the performance of large language models in understanding image content and generating accurate descriptions.

[0085] In summary, the methods for recognizing children's facial emotions in the cockpit mainly involve the following aspects: First, we collected images of children's faces, divided them into training and testing sets, and performed a series of preprocessing operations on the images. For the VIT model used in this experiment, we processed the images to meet the requirements of the model architecture.

[0086] Second, the Q-Former model was fine-tuned using a children's facial image dataset, resulting in more stable performance in children's facial image recognition tasks.

[0087] Third, the output of the ViT model and the text information are input into the fine-tuned Q-Former model, and the resulting visual and text semantic representations are then input into the large language for text generation.

[0088] Furthermore, to quantify model performance, embodiments of this application can parse emotion labels from the final emotion recognition generation results and compare them with the real emotion labels of corresponding samples in the test set to calculate the emotion classification accuracy. The formula for calculating the accuracy can be defined as:

[0089] In the above formula, Represents the overall accuracy of emotion classification. This represents the total number of correctly categorized emotion types out of the four emotion categories. This represents the total number of all images.

[0090] This accuracy rate directly reflects the ability of the embodiments of this application to discern user emotions in real-world scenarios. Meanwhile, the generated descriptive text can be further used for manual evaluation or automatic indicator analysis.

[0091] To enable those skilled in the art to more clearly understand this application, the following specific embodiment illustrates the in-cabin child facial emotion recognition method proposed in this application, such as... Figure 3 As shown, this method can be roughly implemented through the following steps: Step S301: Acquire the captured images of the child's face inside the vehicle and preprocess them to use as input images for subsequent steps; Step S302: Input the image obtained in step S301 into the VIT model to extract visual features from the input image using the VIT model; Step S303: Input the visual feature extraction results, text description and randomly initialized query vector into the fine-tuned Q-Former model, and output a set of query vectors after deep semantic fusion; the output query vectors serve as a bridge connecting the visual and language modalities, integrating a vector sequence of key visual semantics and text alignment information, and carrying the effective information in the image.

[0092] Step S304: Input the output of the Q-Former model into an LLM large language model (e.g., Flan-T5-XL model) for text generation; Step S301: Output the text generation results of the Big Prophecy model. The results are semi-structured data, including sentiment labels and descriptive text.

[0093] In summary, the in-cabin child facial emotion recognition method proposed in this application has the following beneficial effects: 1. This method generates a text description of the image information while outputting the emotion category label, which enhances the interpretability of the classification results compared to the traditional method that relies on a fixed classification head to directly output the emotion label.

[0094] 2. This method introduces a large language model as a general language prior. Because it is pre-trained on massive amounts of data, it has rich language knowledge and reasoning ability, as well as strong zero-shot reasoning ability and generalization.

[0095] 3. This method combines a dual supervision mechanism (language modeling loss and image-text contrastive learning). The Q-Former module is effectively optimized under the generative feedback of the large language model, ensuring that its output visual representation can accurately guide the generation of emotional keywords and is highly aligned with the text semantic space, thereby improving the final accuracy.

[0096] The in-cabin child facial emotion recognition method proposed in this application involves collecting a dataset of children's facial images, preprocessing them, extracting visual features using a VIT model, and then aligning and fusing the visual features with text descriptions using a fine-tuned Q-Former model. The Q-Former acts as a bridge, converting visual features into semantic representations, which are then input into a large language model to generate emotion labels and text descriptions, thus completing the recognition of children's facial emotions. This solves the problems of related technologies lacking cognitive and emotional perception and semantic understanding capabilities towards children, and the model training relying on large-scale labeled data and being sensitive to environmental changes, resulting in low accuracy and poor interpretability in recognizing children's emotions.

[0097] Next, refer to the appendix. Figure 4 This application describes an in-cabin child facial emotion recognition device according to an embodiment of the present application.

[0098] Figure 4 This is a block diagram of a child facial emotion recognition device in the cockpit according to an embodiment of this application.

[0099] like Figure 4 As shown, the child facial emotion recognition device 10 in the cockpit includes: a data acquisition module 100, an extraction module 200, and a recognition module 300.

[0100] The acquisition module 100 is used to acquire an initial dataset of children's facial images and preprocess the initial dataset to obtain a children's facial image dataset that meets the requirements of the preset model architecture.

[0101] The extraction module 200 is used to input the children's facial image dataset into a preset ViT model to extract visual features.

[0102] The recognition module 300 is used to fine-tune the Q-Former model based on visual features, text descriptions, and randomly initialized query vectors until the model reaches the preset stability requirements in the child facial image recognition task, thereby obtaining a child facial emotion recognition model, which is then used to identify the facial emotions of children in the vehicle cabin.

[0103] Optionally, in one embodiment of this application, the acquisition module 100 includes: an integration unit and a partitioning unit; wherein, the integration unit is used to integrate the dataset information of the initial dataset into a metadata file of the target format, so that each data record is stored in a key-value pair structure to obtain a complete dataset; the partitioning unit is used to partition the complete dataset according to a preset ratio to obtain a training set and a test set, which are used to train model parameters, test model performance and evaluate model generalization ability, respectively.

[0104] Optionally, in one embodiment of this application, the acquisition module 100 further includes: a processing unit, which performs scale normalization processing on all images in the training set and the test set to obtain images of the target size, and performs normalization processing, wherein the normalization formula for the images is: , in, This represents the mean vector for each channel. This represents the standard deviation vector of each channel. Represents a tensor.

[0105] Optionally, in one embodiment of this application, the acquisition module 100 further includes: a first generation unit, which is used to generate multiple text descriptions based on the annotations of each image in the training set, wherein the multiple text descriptions include emotion tags and corresponding facial action units and contextual semantic information.

[0106] Optionally, in one embodiment of this application, the recognition module 300 includes: an alignment unit, a splicing unit, and a second generation unit; wherein, the alignment unit is used to feed the query vector with image information output by the Q-Former model into a fully connected layer for linear projection to obtain the projected vector, so as to align the Q-Former output dimension with the embedding dimension of the large language model; the splicing unit is used to use the projected vector sequence as pseudo-word embedding and splice it to the word embedding front end of the task prompt to form a complete multimodal input sequence; the second generation unit is used to input the multimodal input sequence into the large language model to generate complete emotion description text and emotion classification labels.

[0107] Optionally, in one embodiment of this application, the identification module 300 further includes a learning unit, which is used to learn by comparing the query vector with the pre-processed text feature representation.

[0108] The in-vehicle emotion recognition device based on the Q-Former model can accurately capture subtle emotional changes of cabin occupants through cross-modal representation learning. While identifying and distinguishing basic emotion categories, it generates interpretable descriptive output, thereby providing deep semantic understanding support for the proactive services of the smart cockpit.

[0109] It should be noted that the foregoing explanation of the embodiment of the in-cabin child facial emotion recognition method also applies to the in-cabin child facial emotion recognition device of this embodiment, and will not be repeated here.

[0110] The in-cabin child facial emotion recognition device proposed in this application can collect a dataset of children's facial images, preprocess them, and use a VIT model to extract visual features. Then, a finely tuned Q-Former model is used to align and fuse the visual features with textual descriptions. The Q-Former acts as a bridge, converting visual features into semantic representations, which are then input into a large language model to generate emotion labels and textual descriptions, thus completing the recognition of children's facial emotions. This solves the problems of related technologies lacking cognitive and emotional perception and semantic understanding capabilities towards children, and the model training relying on large-scale labeled data and being sensitive to environmental changes, resulting in low accuracy and poor interpretability in recognizing children's emotions.

[0111] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0112] When the processor 502 executes the program, it implements the in-cabin child facial emotion recognition method provided in the above embodiments.

[0113] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.

[0114] The memory 501 is used to store computer programs that can run on the processor 502.

[0115] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0116] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0117] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0118] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0119] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for recognizing facial emotions of children in the cockpit.

[0120] This application also provides a computer program product storing a computer program that, when executed by a processor, implements the above-described method for recognizing children's facial emotions in the cockpit.

[0121] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0122] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0123] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0125] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0128] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for recognizing children's facial emotions in a cockpit, characterized in that, Includes the following steps: An initial dataset of children's facial images is collected, and the initial dataset is preprocessed to obtain a children's facial image dataset that meets the requirements of a preset model architecture. The children's facial image dataset is input into a preset ViT model to extract visual features; Based on the visual features, text descriptions, and randomly initialized query vectors, the Q-Former model is fine-tuned until it reaches a preset stability requirement in the child facial image recognition task, thus obtaining a child facial emotion recognition model. This model is then used to identify the facial emotions of children in the vehicle cabin.

2. The method according to claim 1, characterized in that, The preprocessing of the initial dataset to obtain a children's facial image dataset that meets the requirements of the preset model architecture includes: The dataset information of the initial dataset is integrated into the metadata file of the target format, so that each data record is stored in a key-value pair structure, thus obtaining a complete dataset; The complete dataset is divided into training and testing sets according to a preset ratio, which are used to train model parameters, test model performance, and evaluate model generalization ability, respectively.

3. The method according to claim 2, characterized in that, The preprocessing of the initial dataset to obtain a children's facial image dataset that meets the requirements of the preset model architecture further includes: All images in the training set and the test set are scale-normalized to obtain images of the target size, and then normalized. The normalization formula for the images is as follows: ， in, This represents the mean vector for each channel. This represents the standard deviation vector of each channel. Represents a tensor.

4. The method according to claim 2 or 3, characterized in that, The preprocessing of the initial dataset to obtain a children's facial image dataset that meets the requirements of the preset model architecture further includes: Multiple text descriptions are generated based on the annotations of each image in the training set, wherein the multiple text descriptions include emotion tags and corresponding facial action units and contextual semantic information.

5. The method according to claim 1, characterized in that, The fine-tuning of the Q-Former model includes: The query vector with image information output by the Q-Former model is fed into a fully connected layer for linear projection to obtain the projected vector, so as to align the Q-Former output dimension with the embedding dimension of the large language model. The projected vector sequence is used as a pseudo-word embedding and concatenated to the word embedding front end of the task prompt to form a complete multimodal input sequence; The multimodal input sequence is input into the large language model to generate complete emotion description text and emotion classification labels.

6. The method according to claim 5, characterized in that, The fine-tuning of the Q-Former model also includes: The query vector is compared and learned by comparison with the pre-processed text feature representation.

7. A child facial emotion recognition device for use in a cockpit, characterized in that, include: The acquisition module is used to acquire an initial dataset of children's facial images and preprocess the initial dataset to obtain a children's facial image dataset that meets the requirements of a preset model architecture. The extraction module is used to input the children's facial image dataset into a preset ViT model to extract visual features; The recognition module is used to fine-tune the Q-Former model based on the visual features, text description, and randomly initialized query vector until the model reaches a preset stability requirement in the child facial image recognition task, thereby obtaining a child facial emotion recognition model, which is then used to identify the facial emotions of children in the vehicle cabin.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the in-cabin child facial emotion recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the in-cabin child facial emotion recognition method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the in-cabin child facial emotion recognition method as described in any one of claims 1-6.

Images