Method, device and equipment for generating vehicle-mounted expression dataset and storage medium

Abstract

The application discloses a kind of generation methods, devices and equipment of vehicle expression dataset and storage medium, by obtaining the scene parameter library containing multiple vehicle environment parameters and the anonymization standard expression library without identity as basic material, expression data is decomposed into multiple action unit combinations using action unit analysis module, action features and parameter features are extracted and fused, initial expression data is obtained by inputting generative adversarial network generator, and then privacy enhancement and quality check form standardized dataset.The application avoids the privacy disclosure risk collected by real drivers, reduces the collection and labeling cost, by fusing environmental parameters and action unit features, the dataset covers multiple vehicle conditions, improves scene adaptability, and privacy enhancement and quality check guarantee data security and quality, which can be directly used for vehicle expression recognition algorithm training, meet the demand of multi-scene, high-quality, compliance data.The technical scheme of the application can be widely applied to the field of vehicle technology.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a method, apparatus, device, and storage medium for generating in-vehicle facial expression datasets. Background Technology

[0002] With the rapid development of intelligent cockpits and autonomous driving technologies, the demand for in-vehicle systems to perceive driver status is increasing. Among these, emotion recognition based on facial expressions has become a crucial support for improving driving safety and human-machine interaction. Against this backdrop, building facial expression datasets for the in-vehicle environment to train and deploy efficient facial expression analysis algorithms has gradually become a key focus of the industry.

[0003] In related technologies, mainstream in-vehicle facial expression datasets rely on image acquisition from real driving scenarios. This approach has several drawbacks. On one hand, relying on real drivers carries a high risk of privacy breaches and fails to meet increasingly stringent data security standards. On the other hand, the acquisition process is time-consuming, labor-intensive, and costly, with insufficient coverage, failing to fully reflect the impact of factors such as lighting changes, viewing angle differences, and occlusions in the in-vehicle environment, thus making it difficult to meet the training requirements of in-vehicle facial expression recognition algorithms. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for generating in-vehicle facial expression datasets, which can achieve efficient, compliant, and scenario-adaptable construction of in-vehicle facial expression datasets.

[0005] One aspect of this application provides a method for generating an in-vehicle facial expression dataset, the method comprising: Obtain standardized basic materials; wherein, the basic materials include an in-vehicle scene parameter library and an anonymized standard expression library, the in-vehicle scene parameter library contains various in-vehicle environment parameters, and the anonymized standard expression library contains standard expression data without identity identifiers; The standard facial expression data is deconstructed by the action unit parsing module, and the standard facial expression data is decomposed into a combination of multiple action units; Extract the action features corresponding to the action unit, and extract the parameter features of the vehicle environment parameters; The action features and parameter features are fused, and the fused features are input into the generator of the generative adversarial network for processing to obtain initial facial expression data; The initial facial expression data is subjected to privacy enhancement processing and quality verification to obtain a standardized facial expression dataset.

[0006] For example, in some embodiments, obtaining standardized basic materials includes: Collect in-vehicle environmental parameters under various different scenarios or conditions to construct the in-vehicle scenario parameter library; Obtain publicly released authorized emoji data, and perform de-identification processing on the authorized emoji data to obtain the standard emoji data; An anonymous standard emoji library was built based on the aforementioned standard emoji data.

[0007] For example, in some embodiments, the step of deconstructing the standard facial expression data through the action unit parsing module, decomposing the standard facial expression data into a combination of multiple action units, includes: Obtain a sequence of facial images composed of the standard facial expression data; The facial image sequence is analyzed to identify the locations of key points reflecting facial deformation and the changing trends of these key points. Based on the key point locations and the changing trends, the standard facial expression data is decomposed into a combination of multiple action units.

[0008] For example, in some embodiments, analyzing the facial image sequence to identify key point locations reflecting facial deformation includes: Construct a facial shape model, and initialize and align the facial image sequence based on the facial shape model; Based on the initial alignment results, determine the initial coordinates of key points reflecting facial deformation; According to the facial motion coding system specification, the initial coordinates are smoothed and denoised to obtain the target coordinates corresponding to the key points.

[0009] For example, in some embodiments, the fusion of the action features and the parameter features, and the input of the fused features into the generator of the generative adversarial network for processing to obtain initial facial expression data, includes: The action features and parameter features are spatiotemporally aligned and then fused using weighted superposition or feature concatenation to obtain the fused features; The fused features are input into the generator of the generative adversarial network, and the generator generates dynamic facial expression video clips that match the in-vehicle environment based on the spatial distribution and temporal changes of the fused features. The resolution of the dynamic facial expression video clip is modulated, and the initial facial expression data is extracted from the modulated dynamic facial expression video clip.

[0010] For example, in some embodiments, the privacy enhancement processing and quality verification of the initial facial expression data includes: The initial facial expression data is masked for identity features to obtain intermediate facial expression data; wherein, the identity feature masking is used to remove or blur facial information that can uniquely identify an individual; The quality indicators of the intermediate facial expression data are detected; wherein the quality indicators include at least one of clarity, dynamic continuity and adaptability to the vehicle scene; If the quality indicators meet the preset requirements, the intermediate facial expression data is determined to have passed the quality verification, and the intermediate facial expression data is added to the facial expression dataset. If the quality indicators do not meet the preset requirements, it is determined that the intermediate facial expression data has failed the quality verification, and the intermediate facial expression data is discarded.

[0011] Exemplarily, in some embodiments, the method further includes: For each expression data in the expression dataset, corresponding label information is generated; wherein, the label information includes scene number, expression category, combination of action units, intensity and duration of each action unit.

[0012] On the other hand, embodiments of this application provide an apparatus for generating an in-vehicle facial expression dataset, the apparatus comprising: The acquisition unit is used to acquire standardized basic materials; wherein, the basic materials include an in-vehicle scene parameter library and an anonymized standard expression library, the in-vehicle scene parameter library contains various in-vehicle environment parameters, and the anonymized standard expression library contains standard expression data without identity identifiers; The deconstruction unit is used to deconstruct the standard facial expression data through the action unit parsing module, and decompose the standard facial expression data into a combination of multiple action units. An extraction unit is used to extract the action features corresponding to the action unit, and to extract the parameter features of the vehicle environment parameters; The processing unit is used to fuse the action features and the parameter features, and input the fused features into the generator of the generative adversarial network for processing to obtain initial expression data; The post-processing unit is used to perform privacy enhancement processing and quality verification on the initial facial expression data to obtain a standardized facial expression dataset.

[0013] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory; The memory is used to store computer programs; The processor executes the computer program to implement the aforementioned method for generating the in-vehicle facial expression dataset.

[0014] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned method for generating an in-vehicle facial expression dataset.

[0015] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned method for generating an in-vehicle facial expression dataset.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides a method, apparatus, device, and storage medium for generating an in-vehicle facial expression dataset. It obtains an in-vehicle scene parameter library containing various in-vehicle environmental parameters and an anonymized standard facial expression library containing unidentified standard facial expression data as standardized basic materials. An action unit parsing module decomposes the standard facial expression data into a combination of multiple action units, extracts the action features corresponding to the action units and the parameter features of the in-vehicle environmental parameters, and fuses them. The fused features are input into a generator of a generative adversarial network to obtain initial facial expression data, which is then processed through privacy enhancement and quality verification to form a standardized facial expression dataset. This application avoids the privacy leakage risk associated with collecting real driver data, reduces collection and annotation costs, and by fusing in-vehicle environmental parameters and action unit features, enables the generated dataset to cover various in-vehicle environmental conditions, improving the data's adaptability in in-vehicle scenarios. Furthermore, privacy enhancement and quality verification ensure data security and quality, allowing it to be directly used for training in-vehicle facial expression recognition algorithms, effectively meeting the algorithm's requirements for multi-scenario, high-quality, and compliant data. Attached Figure Description

[0017] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0018] Figure 1 This is a system architecture diagram of the method for generating an in-vehicle facial expression dataset provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a method for generating an in-vehicle facial expression dataset provided in an embodiment of this application. Figure 3 This is a structural block diagram of a device for generating an in-vehicle facial expression dataset provided in an embodiment of this application; Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0021] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0022] With the rapid development of intelligent cockpits and autonomous driving technologies, the demand for in-vehicle systems to perceive driver status is increasing. Among these, emotion recognition based on facial expressions has become a crucial support for improving driving safety and human-machine interaction. Against this backdrop, building facial expression datasets for the in-vehicle environment to train and deploy efficient facial expression analysis algorithms has gradually become a key focus of the industry.

[0023] In related technologies, mainstream in-vehicle facial expression datasets rely on image acquisition from real driving scenarios. This approach has several drawbacks. On one hand, relying on real drivers carries a high risk of privacy breaches and fails to meet increasingly stringent data security standards. On the other hand, the acquisition process is time-consuming, labor-intensive, and costly, with insufficient coverage, failing to fully reflect the impact of factors such as lighting changes, viewing angle differences, and occlusions in the in-vehicle environment, thus making it difficult to meet the training requirements of in-vehicle facial expression recognition algorithms.

[0024] In view of this, this application provides a method, apparatus, device, and storage medium for generating an in-vehicle facial expression dataset. It uses an in-vehicle scene parameter library containing various in-vehicle environmental parameters and an anonymized standard facial expression library containing unidentified standard facial expression data as standardized basic materials. An action unit parsing module decomposes the standard facial expression data into combinations of multiple action units, extracts the action features corresponding to the action units and the parameter features of the in-vehicle environmental parameters, and fuses them. The fused features are input into a generator of a generative adversarial network to obtain initial facial expression data, which is then processed through privacy enhancement and quality verification to form a standardized facial expression dataset. This application avoids the privacy leakage risk associated with collecting real driver data, reduces collection and annotation costs, and by fusing in-vehicle environmental parameters and action unit features, enables the generated dataset to cover various in-vehicle environmental conditions, improving the data's adaptability in in-vehicle scenarios. Furthermore, privacy enhancement and quality verification ensure data security and quality, allowing it to be directly used for training in-vehicle facial expression recognition algorithms, effectively meeting the algorithm's requirements for multi-scenario, high-quality, and compliant data.

[0025] System architecture and scenario description used in the embodiments of this application Please refer to Figure 1 , Figure 1 The diagram shows the system architecture of a method for generating an in-vehicle facial expression dataset according to an embodiment of this application. It includes a terminal device 140, an Internet 130, a gateway 120, a backend server 110, etc.

[0026] In this embodiment, the terminal device 140 can be a vehicle-mounted terminal, a mobile terminal, or a fixed data acquisition device. The backend server 110 refers to a computer system capable of providing certain services to the terminal device 140. Compared to a regular terminal device 140, the backend server 110 has higher requirements in terms of stability, security, and performance. The backend server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0027] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal device 140 to backend server 110 are forwarded to the corresponding backend server 110 via gateway 120. Messages sent from backend server 110 to terminal device 140 are also forwarded to the corresponding terminal device 140 via gateway 120.

[0028] The backend server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0029] The method for generating vehicle-mounted facial expression datasets provided in this application embodiment can be executed independently on the terminal device 140, or based on data interaction between the terminal device 140 and the backend server 110.

[0030] Of course, it is understood that the implementation environment corresponding to the method in the embodiments of this application is not limited to that of the implementation environment. Figure 1 As shown, those skilled in the art can flexibly select the specific implementation environment according to actual needs, and this application does not impose any restrictions on this.

[0031] General Description of Embodiments in this Application Please refer to Figure 2 , Figure 2 A flowchart illustrating a method for generating an in-vehicle facial expression dataset according to an embodiment of this application is shown. Figure 2 As shown, a method for generating an in-vehicle facial expression dataset according to an embodiment of this application includes, but is not limited to, the following steps: Step 210: Obtain standardized basic materials; wherein, the basic materials include an in-vehicle scene parameter library and an anonymized standard expression library, the in-vehicle scene parameter library contains various in-vehicle environment parameters, and the anonymized standard expression library contains standard expression data without identity identifiers. Step 220: Deconstruct the standard facial expression data using the action unit parsing module, breaking down the standard facial expression data into a combination of multiple action units; Step 230: Extract the action features corresponding to the action unit, and extract the parameter features of the vehicle environment parameters; Step 240: Fuse the action features and the parameter features, and input the fused features into the generator of the generative adversarial network for processing to obtain initial facial expression data; Step 250: Perform privacy enhancement processing and quality verification on the initial facial expression data to obtain a standardized facial expression dataset.

[0032] This application provides a method for generating an in-vehicle facial expression dataset. This method aims to improve the problems of high privacy leakage risk, high collection and annotation costs, and difficulty in covering the complex in-vehicle environment caused by relying on real driver data collection in related technologies. By introducing standardized basic materials and action unit parsing, and combining generative adversarial networks and privacy enhancement mechanisms, an efficient, compliant and scenario-adaptable in-vehicle facial expression dataset can be constructed.

[0033] Specifically, this method first acquires an in-vehicle scene parameter library containing various in-vehicle environment parameters and an anonymous standard expression library without identity verification as basic materials. Then, through an action unit parsing module, the standard expression data is decomposed into combinations of multiple action units, and action features and parameter features are extracted and fused. The fused features are input into a generative adversarial network generator to generate initial expression data, which is then subjected to privacy enhancement processing such as identity feature masking and quality verification to form a standardized expression dataset that meets the needs of in-vehicle scenarios. This method fuses anonymized standard data with in-vehicle environment features, ensuring privacy and data quality while generating dynamic expression data covering various lighting, viewing angles, and occlusion conditions. This directly serves the training of in-vehicle expression recognition algorithms, effectively improving the recognition accuracy and reliability of the algorithm in real driving environments.

[0034] Below, in conjunction with Figure 2 This paper introduces and explains the various process steps of the method for generating the vehicle-mounted facial expression dataset in the embodiments of this application.

[0035] In step 210, the first step is to obtain standardized basic materials. Here, the basic materials include a vehicle scene parameter library and an anonymized standard emoji library.

[0036] The in-vehicle scene parameter library can contain various in-vehicle environmental parameters, such as different light intensities, viewing angle changes, seat angles, and occlusion conditions. This data can be derived from real-vehicle road test statistics or user-defined parameters, enhancing the realism and diversity of the generated data. The anonymized standard emoji library contains unidentified standard emoji data, which can originate from publicly released or authorized emoji datasets. This data undergoes de-identification processing to remove personally identifiable features, ensuring privacy and security from the source. The combination of these two types of materials provides a compliant and rich input source for subsequent emoji deconstruction and feature fusion.

[0037] In step 220, the standard facial expression data is deconstructed by the action unit parsing module, and decomposed into a combination of multiple action units.

[0038] Specifically, the action unit parsing module is a functional unit in this application used to perform structured decomposition of standard facial expression data. Based on the principles of facial morphology, it breaks down continuous facial expression performance into a combination of several independently describable basic action units.

[0039] In this embodiment, the overall facial expression can be decomposed into a combination of multiple action units using an action unit parsing module. Each action unit represents a basic facial muscle activity pattern, and their combination and variation jointly determine the final expression category and intensity. This deconstruction transforms continuous and complex facial expressions into quantifiable and reusable action unit representations, providing an operable basic expression for subsequent extraction of action features and fusion with in-vehicle environmental parameters. This allows the generation process to maintain the naturalness of the expression while achieving precise control over its details.

[0040] For example, in the embodiments of this application, when implementing the action unit parsing module, an existing facial motion coding system can be relied upon to treat facial expressions as a combination of several basic action units. For example, when parsing a "smiling" expression, the module analyzes the positional changes of key facial points (such as the corners of the mouth, around the eyes, and cheeks): detecting patterns such as the outer corners of the mouth moving upwards, localized cheek bulging, and slight tightening of the muscles around the eyes, and mapping these patterns to preset action units, such as "corners of the mouth raised," "cheeks lifted," and "eyelids tightened." By analyzing the order of occurrence, duration, and intensity of these action units, the overall smiling expression is decomposed into a combination of these action units, and the degree of participation of each is recorded.

[0041] In step 230, the action features corresponding to the action unit are extracted, as well as the parameter features of the vehicle environment parameters are extracted.

[0042] In this embodiment, for each deconstructed action unit, the amplitude of movement, rate of change and duration of change in different areas of the face are analyzed to form action features describing the dynamic attributes of the action unit; environmental parameters associated with the current generated task are read from the vehicle scene parameter library, and their numerical distribution and change patterns are analyzed to form parameter features reflecting environmental conditions; the action features and parameter features are stored separately for subsequent fusion processing.

[0043] For example, taking the action unit of "upturned corners of the mouth" as an example, in this embodiment of the application, the key points corresponding to the corners of the mouth can be located in the deconstructed facial key point sequence, and their coordinate changes can be calculated in consecutive frames to obtain relevant data. Then, the relevant data is organized into action features according to the time series. In some embodiments, feature extraction can also be achieved through relevant feature models. This application does not limit the specific model type.

[0044] In step 240, the action features and parameter features are fused, and the fused features are input into the generator of the generative adversarial network for processing to obtain the initial expression data.

[0045] For example, action features and parameter features can first be spatiotemporally aligned and then fused using weighted superposition or feature concatenation to obtain fused features. These fused features are then input into the generator of a generative adversarial network (GAN). The generator, based on the spatial distribution and temporal variations of the fused features, generates dynamic facial expression video clips that match the in-vehicle environment. Finally, the dynamic facial expression video clips are resolution modulated, and initial facial expression data is extracted from the modulated video clips. In this way, structured features can be transformed into realistic dynamic facial expression data through a generative adversarial network, achieving efficient expansion of data scale.

[0046] Specifically, in this embodiment, the action features and parameter features are first matched and mapped in the temporal and spatial dimensions. It is easy to understand that action features reflect the temporal changes in muscle activity in different facial regions, while parameter features describe the numerical state of the in-vehicle environment. The two may have different time bases or spatial coordinate systems when collected or recorded. Therefore, before fusion, the two types of features need to be spatiotemporally aligned based on a unified timestamp or scene number, so that facial expressions at the same moment can correspond one-to-one with the corresponding environmental state.

[0047] After completing spatiotemporal alignment, action features and parameter features are combined. This can be achieved through either a weighted summation at the feature vector level, assigning different weights to each feature based on its importance in the generation task, and then adding them together to form a fused feature; or by concatenating features, linking action and parameter feature vectors dimensionally to create a comprehensive feature that includes both facial expression dynamics and environmental information. The fused feature carries both fine-grained dynamic information about the expression and constraints of the generation environment, providing comprehensive input for subsequent generators.

[0048] After the fused features are input into the generator of the generative adversarial network (GAN), the generator performs spatial modeling and temporal extrapolation internally based on the spatial distribution information (such as the changing patterns of different facial regions) and temporal change information (such as the order and speed of actions) in the features. Combining the mapping relationship between the in-vehicle scene and facial expressions learned by the generator, it gradually synthesizes dynamic facial expression video clips that match the specified environmental conditions. The video clips should visually present the facial expression changes that naturally occur under the environmental conditions.

[0049] Next, the generated dynamic facial expression video clips are resolution modulated to meet preset output specifications, such as uniformly adjusting to high-definition resolution. Complete frame sequences or segments are then extracted from the modulated video as initial facial expression data for subsequent privacy enhancement and quality verification steps. This fusion and generation process ensures the realism of the facial expressions while making the generated data naturally adaptable to the target in-vehicle scenario.

[0050] In step 250, the initial facial expression data undergoes privacy enhancement processing and quality verification to obtain a standardized facial expression dataset.

[0051] Specifically, the initial facial expression data can first be masked by removing or blurring facial information that uniquely identifies an individual, resulting in intermediate facial expression data. Then, the quality metrics of the intermediate facial expression data are checked, including at least one of clarity, dynamic continuity, and adaptability to the in-vehicle environment. If the quality metrics meet preset requirements, the intermediate facial expression data passes quality verification and is added to the facial expression dataset; otherwise, it is discarded or regenerated. In this way, while ensuring privacy and security, the clarity, realism, and scene adaptability of the generated data can be guaranteed, forming a standardized facial expression dataset that can be directly used for training in-vehicle facial expression recognition algorithms.

[0052] It is understood that, in this embodiment of the application, a framework for generating facial expression datasets that does not rely on real driver data collection and can cover various in-vehicle environments is constructed. By combining the structured action unit parsing of anonymized standard facial expressions, the fusion modeling of in-vehicle environment parameters, and the realistic data synthesis capabilities of generative adversarial networks, and applying privacy enhancement and quality verification after generation, the problems of privacy leakage and high collection costs are solved, while ensuring the adaptability and annotation quality of data in in-vehicle scenarios. This effectively improves the training effect and application reliability of in-vehicle facial expression recognition algorithms in real driving environments.

[0053] Specifically, in some embodiments, obtaining standardized basic materials includes: Collect in-vehicle environmental parameters under various different scenarios or conditions to construct the in-vehicle scenario parameter library; Obtain publicly released authorized emoji data, and perform de-identification processing on the authorized emoji data to obtain the standard emoji data; An anonymous standard emoji library was built based on the aforementioned standard emoji data.

[0054] In this embodiment, the process of obtaining standardized basic materials can begin by collecting in-vehicle environmental parameters under various scenarios or conditions to construct the in-vehicle scenario parameter library. These scenarios or conditions include light intensity at different times, different weather conditions, various camera installation positions and viewing angles, different seat adjustment angles, and possible occlusion situations (such as wearing glasses, masks, or direct sunlight). These parameters are obtained through real vehicle road tests, simulator tests, or by organizing existing data to ensure that the parameter library can cover various environmental changes that in-vehicle facial expression recognition may encounter.

[0055] At the same time, publicly released authorized emoji data can be obtained. This data usually comes from emoji databases that have undergone ethical review and legal authorization, and includes various basic emoji categories. These authorized emoji data undergo de-identification processing, that is, removing or blurring information that can uniquely identify an individual, such as prominent facial contours, skin texture details, and feature markers that can be associated with a specific individual, retaining only the key information related to the emoji itself, thereby obtaining standard emoji data that does not contain interference from identity information.

[0056] Then, an anonymized standard expression library is built based on the standard expression data. The de-identified expression data is organized, labeled, and stored in a unified format, making it directly usable for subsequent action unit parsing and feature extraction steps. This anonymized standard expression library, together with the aforementioned in-vehicle scene parameter library, constitutes standardized basic materials, providing compliant and diverse input sources for the subsequent generation of expression datasets adapted to the in-vehicle environment.

[0057] Specifically, in some embodiments, the step of deconstructing the standard facial expression data through the action unit parsing module, breaking down the standard facial expression data into a combination of multiple action units, includes: Obtain a sequence of facial images composed of the standard facial expression data; The facial image sequence is analyzed to identify the locations of key points reflecting facial deformation and the changing trends of these key points. Based on the key point locations and the changing trends, the standard facial expression data is decomposed into a combination of multiple action units.

[0058] In this embodiment of the application, when decomposing standard facial expression data into a combination of multiple action units, a facial image sequence composed of the standard facial expression data can be obtained first. These facial image sequences are typically composed of continuously captured facial expression video frames, covering the complete dynamic process from the start to the end of the expression, providing continuous information in the time dimension for subsequent analysis.

[0059] By analyzing facial image sequences, key points reflecting facial deformation and their changing trends can be identified. Here, key points refer to characteristic locations that represent facial structure and muscle movement, such as the corners of the eyes, mouth, brow, and sides of the nose. By analyzing the positional changes of these points in consecutive frames, the morphological features and dynamic trends of the face when expressing different emotions can be captured, such as upturned corners of the mouth, furrowed brows, and open eyelids.

[0060] Based on the locations of these key points and their changing trends, standard facial expression data can be decomposed into combinations of multiple action units. For example, if the control point at the corner of the mouth is detected to move upward and outward, while the key points in the cheek area bulge, it is classified as the action unit corresponding to "upturned corners of the mouth"; if the key point at the center of the brow is clearly gathered and the eyebrows are lowered, it is classified as the action unit corresponding to "frowning". Each action unit represents a typical facial muscle activity pattern, and its determination is based on whether the displacement direction, amplitude, and rate of change of the relevant key points meet the characteristic conditions of that action unit.

[0061] It is understood that, in this embodiment of the application, this decomposition method transforms the originally continuous facial expression video into a quantifiable description composed of multiple action units. Each action unit can be analyzed independently or reproduced in different combinations to reproduce various expressions, thereby providing a structured basic expression form for subsequent extraction of action features, fusion with vehicle environment parameters, and precise control of expression synthesis in generative adversarial networks.

[0062] Specifically, in some embodiments, the analysis of the facial image sequence to identify the locations of key points reflecting facial deformation includes: Construct a facial shape model, and initialize and align the facial image sequence based on the facial shape model; Based on the initial alignment results, determine the initial coordinates of key points reflecting facial deformation; According to the facial motion coding system specification, the initial coordinates are smoothed and denoised to obtain the target coordinates corresponding to the key points.

[0063] In this embodiment, when analyzing the facial image sequence and identifying the key point locations reflecting facial deformation, the following method can be used: First, a facial shape model is constructed. This model is trained from a large number of facial images annotated with key points and can describe the geometric structure of the facial contour and the positions of facial features. After obtaining the facial image sequence, the model is used to initialize and align each frame in the sequence, so that the position, scale, and pose of the face in the image are unified to the reference coordinate system defined by the model, thereby eliminating deviations caused by differences in shooting angle or distance.

[0064] Based on the initial alignment results, key points corresponding to the facial shape model are located in each frame of the image to determine the initial coordinates reflecting facial deformation. These key points cover areas such as eyebrows, eyes, nose, mouth, and cheeks, and their initial coordinates reflect the morphological features of the face in static or dynamic states. Then, according to the specifications of the facial motion coding system, the initial coordinates are smoothed and denoised. This process analyzes the motion trajectory of key points in consecutive frames, filters out abnormal jumps caused by image noise, lighting changes, or temporary occlusion, and retains trajectory changes that conform to the natural facial muscle movement patterns, thereby obtaining stable key point target coordinates. Through the above steps, the positions of key points reflecting facial deformation can be reliably extracted from the facial image sequence, providing accurate morphological basis for subsequent deconstruction of expressions into action unit combinations.

[0065] Specifically, in some embodiments, the method further includes: For each expression data in the expression dataset, corresponding label information is generated; wherein, the label information includes scene number, expression category, combination of action units, intensity and duration of each action unit.

[0066] In this embodiment of the application, after generating the expression dataset, corresponding label information can be generated for each expression data to provide a structured reference for subsequent model training and scene adaptation.

[0067] Specifically, the in-vehicle scene parameters associated with the expression data are read, and a unique scene number is assigned to it according to the classification rules in the parameter library to identify the in-vehicle environment in which the expression was generated, such as lighting, viewing angle, and occlusion. At the same time, based on the semantic attributes of the expression itself, its expression category is labeled, such as "neutral," "happy," "surprised," and "tired," which facilitates subsequent category-based filtering or classification training.

[0068] In some embodiments, the motion unit combinations obtained during the motion unit parsing phase of the expression can also be recorded, including the types of motion units that appear and their order. For each motion unit, its intensity value (which can be calculated from the displacement amplitude or velocity of key points) and duration (determined by the time difference between the start frame and the end frame) during the expression process are also recorded.

[0069] In this way, each piece of facial expression data is accompanied by a complete set of label information, including scene number, expression category, action unit combination, and the intensity and duration of each unit. This information is stored in a structured form bound to the facial expression data, so that the generated facial expression dataset not only has diverse environment and expression coverage, but also has fine-grained annotations that can be directly used for supervised training, significantly improving the training effect and recognition accuracy of the vehicle-mounted facial expression recognition algorithm in real-world scenarios.

[0070] Reference Figure 3 In this embodiment of the application, an apparatus for generating an in-vehicle facial expression dataset is also provided. The apparatus for generating the in-vehicle facial expression dataset includes: The acquisition unit 310 is used to acquire standardized basic materials; wherein, the basic materials include an in-vehicle scene parameter library and an anonymized standard expression library, the in-vehicle scene parameter library contains a variety of in-vehicle environment parameters, and the anonymized standard expression library contains standard expression data without identity identifiers. The deconstruction unit 320 is used to deconstruct the standard facial expression data through the action unit parsing module, and decompose the standard facial expression data into a combination of multiple action units. Extraction unit 330 is used to extract the action features corresponding to the action unit and the parameter features of the vehicle environment parameters; The processing unit 340 is used to fuse the action features and the parameter features, and input the fused features into the generator of the generative adversarial network for processing to obtain initial expression data. The post-processing unit 350 is used to perform privacy enhancement processing and quality verification on the initial facial expression data to obtain a standardized facial expression dataset.

[0071] It is understandable that, such as Figure 2 The content of the vehicle-mounted expression dataset generation method embodiment shown is applicable to the vehicle-mounted expression dataset generation device embodiment. The specific functions implemented by the vehicle-mounted expression dataset generation device embodiment are the same as those shown in the example. Figure 2 The method for generating the in-vehicle facial expression dataset shown is the same as the embodiment described above, and the beneficial effects achieved are the same as those described above. Figure 2 The beneficial effects achieved by the embodiment of the method for generating the in-vehicle facial expression dataset shown are also the same.

[0072] Reference Figure 4 This application also discloses an electronic device, including: At least one processor 410; At least one memory 420 is used to store at least one program; When at least one program is executed by at least one processor 410, such that at least one processor 410 performs as follows: Figure 2 An example of a method for generating an in-vehicle facial expression dataset is shown.

[0073] The electronic device in the embodiments of this application may be a terminal device, a computer device, or a server device.

[0074] Understandable, Figure 2 The content of the method embodiment for generating the in-vehicle facial expression dataset shown is applicable to the embodiment of this electronic device. The specific functions implemented in the embodiment of this electronic device are the same as those in the embodiment of this electronic device. Figure 2 The method for generating the in-vehicle facial expression dataset shown is the same as the embodiment, and the beneficial effects achieved are the same. Figure 2 The beneficial effects achieved by the embodiment of the method for generating the in-vehicle facial expression dataset shown are also the same.

[0075] This application also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement, for example... Figure 2 An example of a method for generating an in-vehicle facial expression dataset is shown.

[0076] Understandable, Figure 2 The content of the method embodiment for generating the in-vehicle facial expression dataset shown is applicable to the embodiment of this computer-readable storage medium. The specific functions implemented in the embodiment of this computer-readable storage medium are the same as those in the embodiment of this computer-readable storage medium. Figure 2 The method for generating the in-vehicle facial expression dataset shown is the same as the embodiment, and the beneficial effects achieved are the same. Figure 2 The beneficial effects achieved by the embodiment of the method for generating the in-vehicle facial expression dataset shown are also the same.

[0077] This application also discloses a computer program product or computer program, which includes computer instructions stored in the aforementioned computer-readable storage medium. Figure 4 The processor of the illustrated electronic device can read the computer instructions from the aforementioned computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 2 An example of a method for generating an in-vehicle facial expression dataset is shown.

[0078] Understandable, Figure 2 The content of the method embodiment for generating the in-vehicle facial expression dataset shown is applicable to this computer program product or computer program embodiment. The specific functions implemented by this computer program product or computer program embodiment are the same as those in the embodiment. Figure 2 The method for generating the in-vehicle facial expression dataset shown is the same as the embodiment, and the beneficial effects achieved are the same. Figure 2 The beneficial effects achieved by the embodiment of the method for generating the in-vehicle facial expression dataset shown are also the same.

[0079] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0080] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0081] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0083] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0084] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0085] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0086] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0087] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for generating an in-vehicle facial expression dataset, characterized in that, The method includes: Obtain standardized basic materials; wherein, the basic materials include an in-vehicle scene parameter library and an anonymized standard expression library, the in-vehicle scene parameter library contains various in-vehicle environment parameters, and the anonymized standard expression library contains standard expression data without identity identifiers; The standard facial expression data is deconstructed by the action unit parsing module, and the standard facial expression data is decomposed into a combination of multiple action units; Extract the action features corresponding to the action unit, and extract the parameter features of the vehicle environment parameters; The action features and parameter features are fused, and the fused features are input into the generator of the generative adversarial network for processing to obtain initial facial expression data; The initial facial expression data is subjected to privacy enhancement processing and quality verification to obtain a standardized facial expression dataset.

2. The method for generating an in-vehicle facial expression dataset according to claim 1, characterized in that, The acquisition of standardized basic materials includes: Collect in-vehicle environmental parameters under various different scenarios or conditions to construct the in-vehicle scenario parameter library; Obtain publicly released authorized emoji data, and perform de-identification processing on the authorized emoji data to obtain the standard emoji data; An anonymous standard emoji library was built based on the aforementioned standard emoji data.

3. The method for generating an in-vehicle facial expression dataset according to claim 1, characterized in that, The step of deconstructing the standard facial expression data through the action unit parsing module, breaking down the standard facial expression data into a combination of multiple action units, includes: Obtain a sequence of facial images composed of the standard facial expression data; The facial image sequence is analyzed to identify the locations of key points reflecting facial deformation and the changing trends of these key points. Based on the key point locations and the changing trends, the standard facial expression data is decomposed into a combination of multiple action units.

4. The method for generating an in-vehicle facial expression dataset according to claim 3, characterized in that, The analysis of the facial image sequence to identify the locations of key points reflecting facial deformation includes: Construct a facial shape model, and initialize and align the facial image sequence based on the facial shape model; Based on the initial alignment results, determine the initial coordinates of key points reflecting facial deformation; According to the facial motion coding system specification, the initial coordinates are smoothed and denoised to obtain the target coordinates corresponding to the key points.

5. The method for generating an in-vehicle facial expression dataset according to claim 1, characterized in that, The process involves fusing the action features and the parameter features, then inputting the fused features into the generator of a generative adversarial network for processing to obtain initial facial expression data, including: The action features and parameter features are spatiotemporally aligned and then fused using weighted superposition or feature concatenation to obtain the fused features; The fused features are input into the generator of the generative adversarial network, and the generator generates dynamic facial expression video clips that match the in-vehicle environment based on the spatial distribution and temporal changes of the fused features. The resolution of the dynamic facial expression video clip is modulated, and the initial facial expression data is extracted from the modulated dynamic facial expression video clip.

6. The method for generating an in-vehicle facial expression dataset according to claim 5, characterized in that, The privacy enhancement and quality verification of the initial facial expression data includes: The initial facial expression data is masked for identity features to obtain intermediate facial expression data; wherein, the identity feature masking is used to remove or blur facial information that can uniquely identify an individual; The quality indicators of the intermediate facial expression data are detected; wherein the quality indicators include at least one of clarity, dynamic continuity and adaptability to the vehicle scene; If the quality indicators meet the preset requirements, the intermediate facial expression data is determined to have passed the quality verification, and the intermediate facial expression data is added to the facial expression dataset. If the quality indicators do not meet the preset requirements, it is determined that the intermediate facial expression data has failed the quality verification, and the intermediate facial expression data is discarded.

7. The method for generating an in-vehicle facial expression dataset according to any one of claims 1 to 6, characterized in that, The method further includes: For each expression data in the expression dataset, corresponding label information is generated; wherein, the label information includes scene number, expression category, combination of action units, intensity and duration of each action unit.

8. A device for generating an in-vehicle facial expression dataset, characterized in that, The device includes: The acquisition unit is used to acquire standardized basic materials; wherein, the basic materials include an in-vehicle scene parameter library and an anonymized standard expression library, the in-vehicle scene parameter library contains various in-vehicle environment parameters, and the anonymized standard expression library contains standard expression data without identity identifiers; The deconstruction unit is used to deconstruct the standard facial expression data through the action unit parsing module, and decompose the standard facial expression data into a combination of multiple action units. An extraction unit is used to extract the action features corresponding to the action unit, and to extract the parameter features of the vehicle environment parameters; The processing unit is used to fuse the action features and the parameter features, and input the fused features into the generator of the generative adversarial network for processing to obtain initial expression data; The post-processing unit is used to perform privacy enhancement processing and quality verification on the initial facial expression data to obtain a standardized facial expression dataset.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method for generating the vehicle-mounted facial expression dataset as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for generating the in-vehicle facial expression dataset as described in any one of claims 1 to 7.

Images