Vehicle-mounted emotion monitoring system and method based on multi-modal large model

Abstract

This invention provides an in-vehicle emotion monitoring system and method based on a multimodal large model, relating to the field of emotion monitoring. The in-vehicle emotion monitoring system based on a multimodal large model includes: a first acquisition module, a second acquisition module, a control processing unit, and a cloud server; both the first and second acquisition modules are connected to the control processing unit, which is connected to the cloud server; the system implements a two-layer emotion monitoring mechanism from preliminary emotion recognition to deep emotion analysis. Through initial text data analysis, it quickly determines whether the user's emotion exceeds a safe range. If the emotion is too intense, video acquisition is initiated for further analysis and judgment, ensuring that the system can accurately monitor the user's emotions in complex driving environments, thereby providing effective support for safe driving.

Description

Technical Field

[0001] This invention relates to the field of emotion monitoring, specifically to an in-vehicle emotion monitoring system and method based on a multimodal large model. Background Technology

[0002] With the rapid development of the automotive industry, in-vehicle intelligent systems have become an important means of improving driving safety and user experience. During driving, the driver's emotional state directly affects their decision-making ability and reaction speed; negative emotions such as anxiety, anger, and fatigue increase the probability of traffic accidents. Therefore, accurately monitoring the driver's emotional state to take timely intervention measures has become an important research direction for improving driving safety. Summary of the Invention

[0003] According to embodiments of the present invention, an in-vehicle emotion monitoring system and method based on a multimodal large model are provided to address the problems mentioned in the background section.

[0004] In a first aspect of the invention, an in-vehicle emotion monitoring system based on a multimodal large model is provided.

[0005] The in-vehicle emotion monitoring system based on a multimodal large model includes: a first acquisition module, a second acquisition module, a control processing unit, and a cloud server; the first acquisition module and the second acquisition module are both connected to the control processing unit, and the control processing unit is connected to the cloud server; The first acquisition module is configured to acquire user facial expression data, form first acquisition data, and transmit it to the control processing unit. The control processing unit transmits the received first acquisition data to the cloud server in text form. The second acquisition module is configured to acquire user facial expression data, form second acquisition data, and transmit it to the control processing unit. The control processing unit performs video encoding processing on the received second acquisition data, generates video data, and transmits it to the cloud server. The cloud server is configured to perform a first emotion analysis based on the first collected data, generate a first emotion analysis result, and send a control command to the control processing unit according to the result to start or stop the second collection module; at the same time, the cloud server performs a second emotion analysis based on the second collected data to obtain a second analysis result.

[0006] Preferably, both the first and second collected data include: eye features, head posture, eyebrow features, mouth features, forehead features, and facial muscle features.

[0007] Preferably, the cloud server includes a text extraction module, a video extraction module, an emotion classification unit, and at least two emotion analysis units; The text extraction module is used to receive the first collected data and transmit the first collected data to the emotion classification unit; The emotion classification unit classifies emotions based on the first collected data, outputs the emotion category, and transmits it to the corresponding emotion analysis unit. The emotion analysis unit analyzes emotions and outputs a first analysis result. When the first analysis result exceeds a predetermined first threshold, the data is fed back to the control processing unit, which controls the second acquisition module to start in order to acquire second acquisition data. The video extraction module is used to extract video information data from the second collected data and transmit the video information data to the corresponding emotion analysis unit. The emotion analysis unit analyzes the video information data of the second collected data, outputs a second analysis result, and determines whether the second analysis result exceeds a predetermined second threshold.

[0008] Preferably, the in-vehicle emotion monitoring system based on a multimodal large model further includes a prompting device, which includes an optical projector, a speaker, and a central controller. The optical projector and the speaker are both connected to the central controller, which is also connected to the emotion analysis unit. When the second analysis result exceeds a predetermined second threshold, the emotion analysis unit sends a control signal to the central controller. After receiving the control signal, the central controller controls the optical projector to generate an image for display and projects it onto the vehicle's windshield. The central controller then controls the speaker to emit a prompting sound.

[0009] Preferably, the second acquisition module includes a patch, a first direction adjustment component, a second direction adjustment component, and a video acquisition device; the patch is used to attach to the inside of the vehicle's windshield, the first direction adjustment component is connected to the patch, the second direction adjustment component is connected to the first direction adjustment component, and the video acquisition device is connected to the second direction adjustment component.

[0010] Preferably, the first direction adjustment assembly includes a first mounting base, a slide groove, a slider, an adjusting rod, a connecting rod, a mounting plate, and a locking rod; the first mounting base and the slide groove are both fixedly connected to the patch, the mounting plate is hinged to the first mounting base, the slider is slidably connected to the slide groove, the adjusting rod is rotatably connected to the slider, the axis of the adjusting rod is parallel to the movement direction of the slider, the locking rod is fixedly connected to the adjusting rod, the axis of the locking rod is perpendicular to the axis of the adjusting rod, and a plurality of limiting rods are equidistantly arranged on the lower wall of the slide groove, forming a limiting space between adjacent limiting rods to accommodate the locking rod; one end of the connecting rod is hinged to the slider, and the end of the connecting rod away from the slider is hinged to the mounting plate; The second direction adjustment assembly includes a cavity, a mounting shaft, a second mounting base, an arc-shaped elastic sheet, a limiting part, and a locking tooth. The cavity is fixedly connected to the mounting plate, the second mounting base is connected to the video capture device, the mounting shaft is fixedly connected to the second mounting base, the mounting shaft passes through the limiting part and is fixedly connected to the limiting part, the limiting part is cylindrical, the mounting shaft is rotatably connected to the cavity, the cavity has a recessed part for accommodating the limiting part, the limiting part has strip-shaped protrusions arranged equidistantly along the outer periphery of the limiting part, the recessed part of the cavity has a strip-shaped hole, the arc-shaped elastic sheet is fixedly installed in the cavity, the locking tooth is connected to the arc-shaped elastic sheet, the locking tooth extends out from the strip-shaped hole and extends between two adjacent strip-shaped protrusions to limit the limiting part.

[0011] Preferably, the video capture device includes a base, a first drive assembly, a mounting housing, a second drive assembly, and a high-definition camera; the base is fixedly connected to the second mounting base, the mounting housing is rotatably connected to the base, and an installation space for the first drive assembly is formed between the mounting housing and the base; the first drive assembly includes a gear ring, a drive gear, a first motor, and two driven gears; the first motor is fixedly connected to the base, the drive gear is connected to the output end of the first motor, the gear ring is fixedly connected to the mounting housing, and the drive gear meshes with the gear ring; the two driven gears are rotatably connected to the base via shafts, and the driven gears mesh with the gear ring; the mounting housing includes two connecting ears, the high-definition camera is rotatably mounted between the two connecting ears via two rotating shafts, the second drive assembly is installed in one of the connecting ears, and the second drive assembly includes a second motor, a worm gear, and a worm wheel; the output end of the second motor is connected to the worm gear, the worm wheel is connected to the rotating shaft, and the worm gear and worm wheel mesh with each other.

[0012] Preferably, the in-vehicle emotion monitoring system based on a multimodal large model also includes an external camera, which is used to collect image information of road sections during vehicle travel.

[0013] The in-vehicle emotion monitoring method based on a multimodal large model includes the following steps: S1 collects the user's facial expression data through the first acquisition module, forms the first acquisition data, and transmits the first acquisition data to the control processing unit; The S2 control processing unit transmits the first collected data to the cloud server in text format. The S3 cloud server performs a first sentiment analysis based on the first collected data and generates a first sentiment analysis result. S4 If the first emotion analysis result does not exceed the predetermined first threshold, the processing ends; if the first emotion analysis result exceeds the predetermined first threshold, the cloud server sends a control command to the control processing unit to start the second acquisition module for further emotion monitoring. After the S5 second acquisition module is started, it acquires the user's facial expression video data and generates the second acquisition data. The S6 control processing unit receives the second acquired data and performs video encoding processing to generate video data, which is then transmitted to the cloud server. The S7 cloud server performs a second emotion analysis based on the second collected data, obtains the second emotion analysis results, and determines the user's final emotional state.

[0014] Preferably, multiple emotion analysis units are set up in the cloud server, each corresponding to a specific emotion category; the cloud server performs preliminary classification of the user's emotional state based on the first collected data, and transmits the user's emotion category to the corresponding emotion analysis unit for in-depth emotion analysis; when the analysis result of a certain emotion category exceeds a predetermined emotion threshold, the cloud server triggers the control processing unit to start the second acquisition module to collect video data to further verify the accuracy of the emotional state; based on the analysis result of the second collected data, if the user's emotion still exceeds the threshold, a prompting device is triggered to display prompt information on the windshield through an optical projector, and at the same time, a voice warning is issued through a speaker to remind the user to pay attention to emotion control.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages: By setting up a first acquisition module and a second acquisition module, facial expression data of the user is acquired first, and text data is transmitted first, followed by video data. Since the volume of text data is much smaller than that of video data, prioritizing text transmission can reduce system bandwidth consumption and network load, improve data transmission speed, and thus accelerate the initial processing time of emotion analysis. This ensures that the system can complete the first round of emotion analysis in a short time, meeting the real-time requirements of in-vehicle emotion monitoring. Text data can be quickly transmitted to the cloud server for the first emotion analysis. By analyzing simple but effective facial expression features, the system can quickly determine whether the user is in an emotional state that requires further monitoring. If the text analysis result exceeds a preset emotion threshold, the system then initiates video transmission, thereby reducing unnecessary video processing and improving the overall system's response efficiency.

[0016] The second acquisition module captures the user's facial expression data, generating second acquisition data. This second acquisition data is then transmitted to the control processing unit. Upon receiving the second acquisition data, the control processing unit performs video encoding processing to generate visualized video data, which is then transmitted to the cloud server.

[0017] After receiving the video data, the cloud server performs a more in-depth secondary emotion analysis. This analysis utilizes the processing power of a multimodal large model to generate more accurate secondary analysis results. These results represent the user's current specific emotional state, such as anxiety, drowsiness, or anger. By judging the results of this secondary analysis, the system can further determine whether the user's emotions require immediate intervention, or use these results for intelligent control of the in-vehicle safety system.

[0018] This system implements a two-layer emotion monitoring mechanism, from initial emotion recognition to deep emotion analysis. Through initial text data analysis, it quickly determines whether the user's emotions exceed safe limits. If the emotions are too intense, video capture is initiated for further analysis and judgment, ensuring the system can accurately monitor user emotions in complex driving environments, thereby providing effective support for driving safety.

[0019] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A system block diagram of in-vehicle emotion monitoring based on a multimodal large model according to the present invention is shown; Figure 2 A system block diagram showing the connection between a cloud server, a prompting device, and a control processing unit for in-vehicle emotion monitoring based on a multimodal large model according to the present invention is shown. Figure 3 A three-dimensional structural schematic diagram of the second acquisition module for in-vehicle emotion monitoring based on a multimodal large model according to the present invention is shown; Figure 4 An exploded view of the second acquisition module for in-vehicle emotion monitoring based on a multimodal large model according to the present invention is shown; Figure 5 A schematic diagram of the connection structure of the patch and groove based on the multimodal large model according to the present invention is shown; Figure 6 A three-dimensional structural schematic diagram of a chute based on a multimodal large model according to the present invention is shown; Figure 7 A three-dimensional structural diagram of the cavity and mounting plate connection based on a multimodal large model according to the present invention is shown; Figure 8 A three-dimensional cross-sectional view of the cavity, mounting plate, and limiting part based on a multimodal large model according to the present invention is shown. Figure 9 A cross-sectional structural schematic diagram of the cavity, mounting plate, and limiting part based on the multimodal large model according to the present invention is shown; Figure 10 A three-dimensional cross-sectional view of the toothed and arc-shaped elastic sheet based on a multimodal large model according to the present invention is shown. Figure 11 A partial three-dimensional cross-sectional view of the mounting housing based on a multimodal large model according to the present invention is shown; Figure 12 A three-dimensional structural schematic diagram of the second driving component and the high-definition camera based on a multimodal large model according to the present invention is shown; Figure 13 A three-dimensional structural schematic diagram of the prompting device based on a multimodal large model according to the present invention is shown; Figure 14 A three-dimensional structural schematic diagram of an external camera based on a multimodal large model according to the present invention is shown.

[0021] Explanation of reference numerals in the attached figures 1-First acquisition module, 2-Second acquisition module, 21-Pattern, 22-First direction adjustment component, 221-First mounting base, 222-Slide groove, 2221-Limiting rod, 223-Slider, 224-Adjusting rod, 225-Connecting rod, 226-Mounting plate, 227-Locking rod, 23-Second direction adjustment component, 231-Cavity, 2311-Recess, 2312-Strip hole, 232-Mounting shaft, 233-Second mounting base, 234-Arc-shaped elastic sheet, 235-Limiting part, 2351-Protrusion, 236-Clamping tooth, 24-Video acquisition unit, 241-Base, 242-First drive Components, 2421-Ring gear, 2422-Drive gear, 2423-First motor, 2424-Driven gear, 243-Mounting housing, 2431-Connecting ear, 244-Second drive assembly, 2441-Second motor, 2442-Worm gear, 2443-Worm wheel, 245-High-definition camera, 2451-Rotating shaft, 3-Control processing unit, 4-Cloud server, 41-Text extraction module, 42-Video extraction module, 43-Emotion classification unit, 44-Emotion analysis unit, 5-Prompt device, 51-Optical projector, 52-Speaker, 53-Central controller, 6-External camera. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] like Figure 1 As shown, the in-vehicle emotion monitoring system based on a multimodal large model includes: a first acquisition module 1, a second acquisition module 2, a control processing unit 3, and a cloud server 4. Both the first acquisition module 1 and the second acquisition module 2 are connected to the control processing unit 3, and the control processing unit 3 is connected to the cloud server 4 via a network.

[0025] First, the first acquisition module 1 is used to collect the user's facial expression data, forming the first acquisition data. The first acquisition module 1 can capture the user's facial expression features through a camera or other sensors and transmit this data to the control processing unit 3. In this embodiment, the first acquisition module 1 is an existing camera, and its specific usage and working principle will not be described in detail here. After receiving the first acquisition data, the control processing unit 3 performs preliminary processing on the first acquisition data, converting it into text, and transmits the processed first acquisition data to the cloud server 4 in text form. The text data may include, but is not limited to, feature values ​​and emotion tags extracted from facial expressions.

[0026] Next, after receiving the first collected data, the cloud server 4 reads the text content and performs a first emotion analysis on it. This emotion analysis can be based on a large model, generating the user's first emotion analysis result through the classification of facial expression features and emotion modeling. The cloud server 4 stores a pre-set first threshold for the user's emotions, which is used to determine whether the user is in a specific emotional state (such as anger). If the first emotion analysis result exceeds the first threshold, it indicates that the user's emotional state may affect driving safety. At this time, the cloud server 4 will send a control command to the control processing unit 3 to activate the second acquisition module 2.

[0027] After the second acquisition module 2 is activated, it will again acquire the user's facial expression data. The second acquisition module 2 captures the user's facial expression data and generates second acquisition data. The second acquisition data is then transmitted to the control processing unit 3. After receiving the second acquisition data, the control processing unit 3 performs video encoding processing on the second acquisition data to generate visualized video data, and then transmits the video data to the cloud server 4.

[0028] After receiving the video data, cloud server 4 performs a more in-depth secondary emotion analysis. This analysis utilizes the processing power of a multimodal large model to generate more accurate secondary analysis results. These results indicate the user's current specific emotional state, such as anxiety, drowsiness, or anger. By judging the results of the secondary analysis, the system can further determine whether the user's emotions require immediate intervention, or use these results for the intelligent control of the vehicle safety system.

[0029] The system collects driver facial expression data and video data through a first acquisition module 1 and a second acquisition module 2, achieving multimodal data acquisition. The system implements a two-layer emotion monitoring mechanism, from initial emotion recognition to deep emotion analysis. Through initial text data analysis, it quickly determines whether the user's emotions exceed safe limits. If the emotions are too intense, video acquisition is initiated for further analysis and judgment, ensuring the system can accurately monitor user emotions in complex driving environments, thereby providing effective support for driving safety.

[0030] In a specific embodiment, the system can first capture the user's facial expressions through the first acquisition module 1. The acquired data is processed by the control processing unit 3 and then transmitted to the cloud server 4 in a simplified text format. If the cloud server determines that the user's emotion exceeds a predetermined threshold (e.g., anger exceeds 70%) based on the first emotion analysis result, it sends a control command to the control processing unit (3) to activate the second acquisition module 2.

[0031] The more detailed facial expression video data collected by the second acquisition module 2 will be encoded and transmitted to the cloud server 4 in real time. The cloud server 4 will analyze the facial expression dynamics in the video using a multimodal emotion analysis algorithm to obtain accurate second analysis results. For example, if the user is showing anxiety at this time, the result can be used to automatically adjust the in-car environment (such as lowering the volume, changing the seat posture, etc.) to alleviate the user's negative emotional state.

[0032] By setting up a first acquisition module 1 and a second acquisition module 2, the system first collects the user's facial expression data, prioritizing the transmission of text data before transmitting video data. Since text data is much smaller than video data, prioritizing text transmission reduces system bandwidth consumption and network load, increases data transmission speed, and thus accelerates the initial processing time for emotion analysis. This ensures the system can complete the first round of emotion analysis in a shorter time, meeting the real-time requirements of in-vehicle emotion monitoring. The text data can be quickly transmitted to the cloud server 4 for initial emotion analysis. By analyzing simple yet effective facial expression features, the system can quickly determine whether the user is in an emotional state requiring further monitoring. If the text analysis result exceeds a preset emotion threshold, the system then initiates video transmission, thereby reducing unnecessary video processing and improving the overall system's response efficiency.

[0033] In this embodiment, both the first and second collected data contain multiple feature information related to the user's facial expressions. These feature information specifically include, but are not limited to, the following: Eye characteristics, such as the opening and closing of the eyelids, pupil movement, and blinking frequency, are used to judge emotional states such as fatigue and inattention.

[0034] Head posture: This includes information such as the user's head rotation angle, tilt, and forward and backward movement. By monitoring head posture, it is possible to determine whether the user has a tendency to lose attention or become distracted, especially during long-distance driving, where abnormal changes in head posture may indicate driver fatigue or drowsiness.

[0035] Eyebrow features: such as the raising or tightening of eyebrows. Eyebrow movements are an important signal of a user's emotional changes, reflecting negative emotions such as anger and anxiety.

[0036] Mouth features: These include the opening and closing of the user's mouth, and the upward or downward tilt of the corners of the mouth. Changes in mouth features, such as smiling, tightly closing the lips, or clenching the teeth, can be used to determine the user's emotions, such as pleasure, tension, or anger.

[0037] Forehead features: Changes in the muscles of the forehead area can also reflect a user's emotional state. For example, an increase in forehead wrinkles may indicate that the user is in a tense, focused, or anxious state.

[0038] Facial muscle features: including muscle activity in the cheek and chin areas. Facial muscle features, when used in conjunction with other facial features, can more comprehensively reflect a user's emotional state.

[0039] Furthermore, the first data collection is primarily generated by the first collection module 1, which focuses on static facial expression features, such as capturing the user's eye, eyebrow, and mouth features at a given moment. After processing, this feature data is simplified into text data and transmitted to the cloud server 4 for preliminary emotion analysis.

[0040] Simultaneously, the second set of data is collected and generated by the second acquisition module 2. This module focuses on the dynamic changes in the user's facial expressions, especially capturing subtle changes in head posture, facial muscle features, and forehead features during extended acquisition periods. After processing through video encoding, video data is generated and transmitted to the cloud server 4 for more in-depth emotion analysis. Based on the results of the emotion analysis, the system can determine whether the user's emotional state poses a threat to driving safety, thereby taking timely preventative measures or interventions.

[0041] like Figure 2 As shown, in this embodiment, the cloud server 4 includes a text extraction module 41, a video extraction module 42, an emotion classification unit 43, and at least two emotion analysis units 44.

[0042] The text extraction module 41 receives the first acquired data from the first acquisition module 1. The text extraction module 41 processes the received first acquired data, extracting textual feature data of facial expressions (such as numerical descriptions of features like eyes, mouth, and eyebrows), and then transmits this processed text data to the emotion classification unit 43. The function of the text extraction module 41 is to convert the text data into feature vectors, facilitating subsequent emotion analysis and judgment. The text extraction module 41 mainly consists of a text encoder and a text mapping layer. The first step of the text encoder is to tokenize the input sentence. Tokenization refers to decomposing the sentence into multiple tokens, that is, dividing the text elements into the smallest units for subsequent processing. Specifically, the input sentence is decomposed into individual tokens, such as words or characters. After tokenization, special tokens are added to help the model understand the sentence structure, typically including the following two tokens: Start markers are used to identify the beginning of a text sequence, helping the model capture overall contextual information.

[0043] End markers are used to indicate the end of a text sequence and to clearly define the boundaries of the text.

[0044] After tokenization, the text encoder uses the Open Pre-trained Transformer (OPT) large language model to process the tokenized text sequences, converting them into corresponding feature vectors. OPT is a pre-trained natural language processing model that uses its deep self-attention mechanism to capture semantic information and contextual relationships in text sequences, thereby generating highly expressive feature vectors. Each token corresponds to a feature vector after processing, and these vectors accurately represent the semantic meaning of the input text.

[0045] After feature vector generation, these vectors are passed to the text mapping layer. The main function of the text mapping layer is to further process and map the high-dimensional feature vectors generated by OPT to ensure they meet the input requirements of subsequent sentiment analysis models. Through this process, text features are transformed into feature representations more suitable for multimodal sentiment analysis. The text mapping layer can be formally represented as: (1) in Represents text content, For OPT encoder mapping functions, This is for encoding feature vectors.

[0046] The emotion classification unit 43 performs preliminary emotion classification based on the first collected data transmitted by the text extraction module 41. In a specific embodiment, the emotion classification unit 43 divides the user's emotions into three categories: sadness, anger, and tension. After classification, the emotion classification unit 43 transmits the classification results to the corresponding emotion analysis unit 44 for more in-depth emotion analysis.

[0047] Each emotion analysis unit 44 corresponds to an emotion category. In this embodiment, at least three emotion analysis units are included, corresponding to sadness, anger, and tension, respectively. The emotion analysis unit 44 is responsible for analyzing the emotion category transmitted by the emotion classification unit 43. For example, if the first collected data is classified as sadness, the sadness emotion analysis unit will further analyze the data and generate a first analysis result.

[0048] When the first analysis result exceeds the system's preset first threshold, it indicates that the emotional state may pose a threat to driving safety. At this time, the emotion analysis unit 44 feeds back the analysis result to the control processing unit 3. The control processing unit 3 controls the second acquisition module 2 to start based on the feedback result, and collects the user's dynamic facial expression data, i.e., the second acquisition data.

[0049] After the second acquisition module 2 is activated, it collects the user's dynamic facial expression data to form the second acquisition data, which is in video format. The video extraction module 42 is used to extract video information from the second acquisition data, mainly including changes in the user's facial expressions and other dynamic features (such as head posture, facial muscle changes, etc.) during long-term driving. The extracted video information will be directly transmitted to the corresponding emotion analysis unit 44 without further emotion classification.

[0050] The video extraction module 42 is used to extract image features from the image set of the video. This module mainly consists of three parts: an image encoder, a query transformer, and a linear layer. It can effectively extract image region features from the video for subsequent sentiment analysis and judgment. The image encoder uses a Vision Transformer (VIT) to extract image features from the video. Specifically, image frames in the video are divided into image blocks of the same size, and these image blocks are then passed as input sequences to the Vision Transformer for feature encoding. The Vision Transformer processes images by dividing them into fixed-size blocks, using start markers to label head features, and treating them as global representation features of the entire image. This processing method can extract region features in images more accurately and is widely used in various computer vision tasks.

[0051] The video extraction module employs a query converter based on the BLIP-2 language model to further extract image features. The BLIP-2-based query converter structure has been designed and optimized, removing the text converter part from the original model and retaining only the key parts for image feature extraction. The query converter consists of multiple stacked self-attention modules, each containing three core layers: Self-attention layer: used to capture global dependencies between image feature blocks, improving the effect of feature extraction.

[0052] Cross-attention layer: used to fuse different features, thereby enhancing the feature representation capability of the image.

[0053] Feedforward layer: Used to further process features and improve the overall expressive power of the model.

[0054] This multi-layered stacked structure design enables deep feature extraction from image patches in videos, generating more accurate sentiment analysis data.

[0055] Finally, the features processed by the query transformer are mapped through a linear layer, transforming them into high-dimensional feature data required for sentiment analysis. These features are then transmitted to a cloud server for further sentiment judgment and analysis. The core calculation formula in the query transformer is the attention mechanism, which can be formally expressed as: (2) (3) (4) in This represents the query value, key value, and numerical value converted from image features. , , express Transition parameters in attention mechanisms The feature dimensions for querying values. This represents the normalized exponential function. This represents the dimension concatenation function. This is a transpose operation. Indicates the number of heads of attention.

[0056] The emotion analysis unit 44 analyzes the video information in the received second collected data to generate a second analysis result. This analysis is based on changes in dynamic facial expressions and can more accurately determine the user's real-time emotional state. For example, if the first collected data is classified as sadness and the first analysis result exceeds a first threshold, the second collected data will be directly transmitted to the emotion analysis unit 44 responsible for sadness analysis. The emotion analysis unit 44 further analyzes the video data to determine whether the user's fatigue state continues or worsens.

[0057] If the second analysis result exceeds the preset second threshold, the system can send a warning signal to the vehicle safety system or take other safety measures through the control processing unit 3, such as reminding the driver to rest or automatically adjusting the in-vehicle environment (such as adjusting the seat or volume).

[0058] For example, when the first collected data is classified as sadness by the emotion classification unit 43, and the first analysis result of this sadness exceeds a predetermined first threshold, the cloud server 4 feeds back the analysis result to the control processing unit 3. The control processing unit 3 then activates the second acquisition module 2 to collect the user's dynamic facial expression video data. The video extraction module 42 extracts the video information and transmits it directly to the emotion analysis unit 44 responsible for sadness analysis. At this time, the cloud server 4 does not need to reclassify the second collected data, but directly performs further sadness analysis on the video data to generate a second analysis result. If this result exceeds the system's preset second threshold, the system will take further preventive or intervention measures to ensure the driver's safety.

[0059] When the system detects that a user may be experiencing sadness, the emotion analysis unit 44 prioritizes analyzing the following facial features: Eye features include the degree of eye opening, drooping corners of the eyes, and the direction of gaze. Sadness is often accompanied by slightly drooping eyelids, downturned corners of the eyes, a gaze that is off-center, or a blank, listless expression. By observing these subtle changes in these features, the system can identify a user's sadness.

[0060] Head posture: The system detects slight head drooping and head stability. Sadness is often manifested by a slightly drooping head or a persistent downward posture, indicating low mood. The head lacks frequent movement or nodding, presenting a static or slow posture; the system uses these postural features to determine the user's emotional state.

[0061] When the emotion analysis unit 44 detects that a user may be in a state of anger, it will prioritize analyzing the following facial features: Eyebrow features: Frowning eyebrows and deepened frown lines are typical signs of anger. The system monitors these features to determine if a user is in an angry state.

[0062] Eye feature points: A user's wide-open or sharp gaze is associated with anger, and the system can assess the intensity of anger by the degree of eye expansion.

[0063] Mouth features: Anger is often accompanied by actions such as downturned corners of the mouth and clenched teeth. The system can capture these features to further confirm the emotional state.

[0064] When the system detects that a user may be in a state of stress, the following characteristics will be prioritized for analysis by the sentiment analysis unit 44: Forehead features: Deepened forehead wrinkles and the sheen caused by sweating on the forehead are signs of tension. The system identifies the user's level of anxiety by monitoring these changes.

[0065] Mouth features: When nervous, users often have tightly closed lips and trembling chins, which are common facial reactions. The system can analyze these features to assess the user's emotional fluctuations.

[0066] Facial muscle feature points: When facial muscles are tense or the masseter muscle is noticeably prominent, it usually indicates that the user is in a state of tension. The system can further analyze the user's emotional state through these features. The module for further analyzing feature emotions is trained using a multilayer perceptron. During the training process of the multilayer perceptron, contrastive learning and cross-entropy classification strategies are used to deeply mine the emotional representations in the features, enabling the feature emotion analysis module to efficiently detect emotional features in video data, thereby improving the accuracy of the module's emotion analysis.

[0067] The second-collected data is directly transmitted to the corresponding emotion analysis unit 44 without further classification, reducing internal data processing steps. This simplified process significantly shortens data processing time, enabling the system to more quickly judge and react to the user's emotional state, especially in emergency situations, improving real-time monitoring efficiency. Reclassification requires additional computing resources for data analysis, while skipping the classification step effectively reduces the computational load on the cloud server, thereby lowering overall system resource consumption. Especially in large-scale data processing, this approach avoids redundant calculations, saves bandwidth and computing power, and improves system performance and stability. Since the second-collected data is triggered by the emotion classification results of the first-collected data, its emotion category has been preliminarily confirmed. Direct transmission to the corresponding emotion analysis unit for more in-depth analysis allows focus on specific emotional states, avoiding errors or biases that may arise from repeated classification and improving the accuracy of emotion recognition.

[0068] Multiple emotion analysis units (44) are set up in the cloud server 4, each corresponding to a different emotion category, such as fatigue, anger, tension, etc. These units receive and process data of different modalities (such as text features and video features), thereby improving the accuracy of emotion recognition.

[0069] like Figure 2 and Figure 13 As shown, in this embodiment, the in-vehicle emotion monitoring system based on a multimodal large model further includes a prompting device 5, which includes an optical projector 51, a speaker 52, and a central controller 53. The optical projector 51 and the speaker 52 are both connected to the central controller 53, which is also connected to the emotion analysis unit 44. When the second analysis result exceeds a predetermined second threshold, the emotion analysis unit 44 sends a control signal to the central controller 53. After receiving the control signal, the central controller 53 controls the optical projector 51 to generate an image for display and projects it onto the vehicle's windshield. The central controller 53 then controls the speaker 52 to emit a prompting sound.

[0070] The optical projector 51 displays relevant prompts on the vehicle's windshield. Its function is to visually convey warnings or prompts to the driver to draw their attention. When the driver's emotional state reaches a dangerous threshold, the projector will display specific image prompts, such as "Please be aware of fatigue driving" or "You are too emotional, please relax," to help the driver adjust their state in time and ensure driving safety. The speaker 52 is used to warn the driver through sound. The speaker 52 can emit prompts or play voice messages to provide a more intuitive and direct reminder. For example, when the driver's emotion analysis indicates severe anger or sadness, the speaker 52 will emit a warning sound and may play voice prompts such as "Please slow down" or "It is recommended to take a break," to enhance the warning effect. The central controller 53, as the core control unit of the prompting device 5, is connected to the optical projector 51, speaker 52, and emotion analysis unit 44. It is responsible for receiving control signals from the emotion analysis unit 44 and controlling the operation of the optical projector and speaker.

[0071] When the emotion analysis unit 44 determines, based on the emotion analysis results of the second collected data, that the result exceeds a predetermined second threshold, it indicates that the driver's emotional state may have a serious impact on driving safety. At this time, the emotion analysis unit 44 will generate a control signal and transmit it to the central controller 53.

[0072] An optical projector 51 is mounted near the dashboard in front of the driver's field of vision and angled towards the windshield. After processing image or text signals through its internal optical elements, it projects them as a beam onto a specific location on the windshield. The windshield in this area is specially treated to ensure that the projected image is clearly displayed without obstructing the driver's view. The optical projector 51 and its applications in vehicles, particularly head-up display (HUD) technology, are mature existing technologies widely used in modern automobiles and will not be described in detail here.

[0073] like Figures 3 to 12 As shown, in this embodiment, the second acquisition module 2 includes a patch 21, a first direction adjustment component 22, a second direction adjustment component 23, and a video acquisition device 24; the patch 21 is used to attach to the inside of the vehicle's windshield, the first direction adjustment component 22 is connected to the patch 21, the second direction adjustment component 23 is connected to the first direction adjustment component 22, and the video acquisition device 24 is connected to the second direction adjustment component 23.

[0074] The patch 21 is an adhesive component used to fix the second acquisition module to the inside of the vehicle's windshield. The patch 21 uses a strong adhesive material to fix the entire video acquisition device to the glass surface, ensuring that the acquisition module will not shift or vibrate during vehicle movement, thus guaranteeing the stability of video data acquisition. The first direction adjustment component 22 is used to adjust the angle of the second direction adjustment component 23. Since the second direction adjustment component 23 is connected to the video acquisition unit 24, the first direction adjustment component 22 can indirectly adjust the angle of the video acquisition unit 24 in the first direction, and the second direction adjustment component 23 is used to adjust the angle of the video acquisition unit 24 in the second direction.

[0075] During actual installation, the patch 21 is first firmly attached to the inside of the vehicle's windshield. Then, the installer can manually adjust the first-direction adjustment component 22 to change the second-direction angle of the video capture unit 24, ensuring the camera is aimed at the driver's face. Furthermore, the second-direction adjustment component 23 can be used to adjust the first-direction angle of the video capture unit 24 to accommodate different driver seat positions or driver heights. Through this two-stage adjustment system, the video capture unit 24 can achieve precise bidirectional adjustment, ensuring the camera is correctly aimed at the driver's face under various environments and conditions, capturing complete facial expressions and dynamic data.

[0076] In this embodiment, the first direction adjustment assembly 22 includes a first mounting base 221, a slide groove 222, a slider 223, an adjusting rod 224, a connecting rod 225, a mounting plate 226, and a locking rod 227. The first mounting base 221 and the slide groove 222 are both fixedly connected to the patch 21. The mounting plate 226 is hinged to the first mounting base 221. The slider 223 is slidably connected to the slide groove 222. The adjusting rod 224 is rotatably connected to the slider 223. The axis of the adjusting rod 224 is aligned with the axis of the slider 223. The moving direction is parallel, the locking rod 227 is fixedly connected to the adjusting rod 224, the axis of the locking rod 227 is perpendicular to the axis of the adjusting rod 224, and multiple limiting rods 2221 are equidistantly arranged on the lower wall of the slide groove 222. The adjacent limiting rods 2221 form a limiting space to accommodate the locking rod 227, which is used to limit the sliding of the slider 223 and ensure that the adjusted position is stable and does not move. One end of the connecting rod 225 is hinged to the slider 223, and the end of the connecting rod 225 away from the slider 223 is hinged to the mounting plate 226. During use, the operator rotates the adjusting rod 224, which in turn rotates the locking rod 227, causing it to disengage from the limiting space. The adjusting rod 224 then pushes or pulls the slider 223 to slide within the groove 222. The slider 223 drives the connecting rod 225, which in turn rotates the mounting plate 226 relative to the first mounting base 221. The mounting plate 226 is connected to the video capture device 24 via the second adjusting component 23, thereby adjusting the first direction angle of the video capture device 24. After adjustment, the locking rod 227 is rotated back into the limiting space. Because the limiting space restricts the locking rod 227, it is limited by the limiting rod 2221. In this state, the slider 223 cannot slide within the groove 222. Due to vehicle vibrations, the locking rod 227 may disengage from the limiting space. Therefore, in this embodiment, a rubber sleeve (not shown in the figure) is provided at the end of the locking rod 227 away from the adjusting rod 224, and the distance between two adjacent limiting rods 2221 is smaller than the rubber sleeve. By rotating the adjusting rod 224, the locking rod 227 can drive the rubber sleeve into the limiting space. At this time, the rubber sleeve is compressed by the two adjacent limiting rods 2221. Through the friction between the rubber sleeve and the limiting rods 2221, the locking rod 227 is less likely to disengage from the limiting space due to vehicle vibrations.

[0077] The second direction adjustment assembly 23 includes a cavity 231, a mounting shaft 232, a second mounting base 233, an arc-shaped elastic sheet 234, a limiting part 235, and a locking tooth 236. The cavity 231 is fixedly connected to the mounting plate 226, the second mounting base 233 is connected to the video capture unit 24, the mounting shaft 232 is fixedly connected to the second mounting base 233, the mounting shaft 232 passes through the limiting part 235 and is fixedly connected to the limiting part 235, the limiting part 235 is cylindrical, the mounting shaft 232 is rotatably connected to the cavity 231, and the cavity 231 is provided with a recess 231. 1. The recessed portion 2311 is used to accommodate the limiting portion 235. The limiting portion 235 is provided with strip-shaped protrusions 2351 arranged at equal intervals along the outer periphery of the limiting portion 235. The recessed portion 2311 of the cavity 231 is provided with a strip-shaped hole 2312. The arc-shaped elastic sheet 234 is fixedly installed in the cavity 231. The locking tooth 236 is connected to the arc-shaped elastic sheet 234. The locking tooth 236 extends out from the strip-shaped hole 2312 and extends between two adjacent strip-shaped protrusions 2351 to limit the limiting portion 235.

[0078] Among them, such as Figure 9 As shown, the cross-sections of both the tooth 236 and the strip protrusion 2351 are wedge-shaped.

[0079] When the video capture device 24 needs to be adjusted in the second direction, simply move the video capture device 24 to rotate the second mounting base 233. The second mounting base 233 will then rotate the mounting shaft 232 and the limiting part 235. When the limiting part 235 rotates, the strip-shaped protrusion 2351 retracts the drive tooth 236 into the cavity 231, and at the same time, the arc-shaped elastic sheet 234 deforms. When the video capture device 24 is rotated to the appropriate position, the tooth 236, under the restoring force of the arc-shaped elastic sheet 234, extends back into the space between the adjacent strip-shaped protrusions 2351. The tooth 236 limits the limiting part 235. To improve the stability of the limiting, such as Figure 8 and Figure 9 As shown, there are three arc-shaped elastic sheets 234 and three locking teeth 236.

[0080] In this embodiment, the video capture device 24 includes a base 241, a first drive assembly 242, a mounting housing 243, a second drive assembly 244, and a high-definition camera 245. The base 241 is fixedly connected to the second mounting base 233, and the mounting housing 243 is rotatably connected to the base 241. An installation space for the first drive assembly 242 is formed between the mounting housing 243 and the base 241. The first drive assembly 242 includes a gear ring 2421 and a drive gear 2422. The system comprises a first motor 2423 and two driven gears 2424. The first motor 2423 is fixedly connected to the base 241. The drive gear 2422 is connected to the output end of the first motor 2423. The gear ring 2421 is fixedly connected to the mounting housing 243, and the drive gear 2422 meshes with the gear ring 2421. The two driven gears 2424 are rotatably connected to the base 241 via shafts, and mesh with the gear ring 2421. When the first motor 2423 is started, it drives the drive gear 2422 to rotate, which in turn drives the gear ring 2421 to rotate. The gear ring 2421 then drives the mounting housing 243 to rotate, which in turn causes the high-definition camera 245 to rotate in a first direction, thus achieving angle adjustment of the high-definition camera 245 in the first direction. During the rotation of the gear ring 2421, the driven gear 2424 follows the rotation of the gear ring 2421 to ensure that the rotation of the gear ring 2421 and the mounting housing 243 is more stable and smooth.

[0081] The mounting housing 243 includes two connecting ears 2431. The high-definition camera 245 is rotatably mounted between the two connecting ears 2431 via two rotating shafts 2451. The second drive assembly 244 is installed in one of the connecting ears 2431. The second drive assembly 244 includes a second motor 2441, a worm gear 2442, and a worm wheel 2443. The output end of the second motor 2441 is connected to the worm gear 2442, and the worm wheel 2443 is connected to the rotating shaft 2451. The worm gear 2442 and the worm wheel 2443 are meshed together. When the second motor 2441 is started, it drives the worm gear 2442 to rotate, which in turn drives the worm wheel 2443 to rotate. The worm wheel 2443 then drives the high-definition camera 245 to rotate, thereby achieving angle adjustment of the high-definition camera 245 in a second direction.

[0082] like Figure 14As shown, in this embodiment, the in-vehicle emotion monitoring system based on a multimodal large model also includes an external camera 6, which is used to collect image information of road segments during vehicle travel. The external camera 6 can be installed at the front of the vehicle, such as the outside of the windshield or the front of the vehicle, to ensure clear capture of the road conditions ahead. The main function of the external camera 6 is to collect real-time image information of the road during vehicle travel. This image information can include road signs, traffic conditions, surrounding environment, and other factors, providing reference data on the external environment for the emotion monitoring system. For example, in congested traffic or inclement weather conditions, drivers may experience emotional fluctuations due to external stress or sudden events, and the image information from the external camera 6 can serve as auxiliary data to help the system more accurately assess the reasons for the driver's emotional changes. In practical applications, the external camera 6 can not only capture ordinary road conditions but also identify sudden events, such as sudden traffic obstacles or emergency situations on the road. This type of information is crucial for the system to determine whether the driver's emotional fluctuations are directly related to driving safety. For example, when the external camera 6 detects emergency braking or a vehicle suddenly changing lanes ahead, the system can further optimize driver emotion monitoring and warnings based on such external events, prompting the driver to remain calm or take appropriate countermeasures.

[0083] The in-vehicle emotion monitoring method based on a multimodal large model includes the following steps: Step S1. Collect the user's facial expression data through the first acquisition module 1 to generate first acquisition data. This data typically includes the user's facial features, such as expression information of the eyes, eyebrows, and mouth. The first acquisition data is then transmitted to the control processing unit 3.

[0084] Step S2. After receiving the first collected data, the control processing unit 3 converts it into text. This text data contains key features extracted from facial expressions and is then transmitted to the cloud server 4 for further emotion analysis.

[0085] Step S3. Based on the received first collected data, the cloud server 4 performs the first emotion analysis. This analysis process uses large model technology to perform a preliminary assessment of the user's facial expression features and generates the first emotion analysis result.

[0086] Step S4. Based on the results of the first sentiment analysis, the system proceeds with different processing paths: If the first sentiment analysis result does not exceed the predetermined first threshold, that is, the user's emotional state is within the normal range, the system ends the processing.

[0087] If the first emotion analysis result exceeds the predetermined first threshold, it indicates that the user's emotional state may affect driving safety. The cloud server 4 will send a control command to the control processing unit 3 to start the second acquisition module 2 for further emotion monitoring.

[0088] Step S5. After the second acquisition module 2 is started, it acquires video data of the user's facial expressions and generates second acquisition data. This video data contains the user's dynamic facial expression information, providing support for more accurate emotion analysis.

[0089] Step S6. After receiving the second acquired data, the control processing unit 3 performs video encoding processing on it to generate video data and transmits it to the cloud server 4.

[0090] Step S7. Cloud server 4 performs a second sentiment analysis based on the second collected data. Through a multimodal large model, the cloud server can perform in-depth analysis of the user's emotional state and generate a second sentiment analysis result. The system ultimately determines the user's emotional state based on this result and makes an appropriate response.

[0091] To go further: To improve the accuracy and efficiency of emotion analysis, multiple emotion analysis units 44 are set up in the cloud server 4, each corresponding to a specific emotion category, such as fatigue, anger, or tension. The system first performs a preliminary classification of the user's emotional state based on the initial collected data, and then transmits the user's emotion category to the corresponding emotion analysis unit 44 for further analysis.

[0092] When the analysis result of a certain emotion category exceeds a predetermined emotion threshold, the system will activate the second acquisition module 2 to collect more detailed facial expression video data and further verify the accuracy of the user's emotional state. Based on the analysis results of the second acquisition data, if the user's emotional state still exceeds the threshold, the system will trigger the prompt device 5 to display a prompt message on the windshield via the optical projector 51, and simultaneously issue a voice warning via the speaker 52 to remind the user to pay attention to emotional control.

[0093] For example, when the system detects that a user is exhibiting obvious signs of sadness (such as frequent eye closure or abnormal head posture), the cloud server 4 will trigger the control processing unit 3 to activate the second acquisition module 2 to collect more detailed video data. After further analysis of the video data, if the user's emotion analysis results still exceed a predetermined threshold, the system will display prompts such as "Please take a break" or "Significant emotional fluctuations, please remain calm" on the windshield via an optical projector and emit a warning sound through a speaker to remind the driver to pay attention to emotional control.

[0094] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An in-vehicle emotion monitoring system based on a multimodal large model, characterized in that, It includes a first acquisition module (1), a second acquisition module (2), a control processing unit (3), and a cloud server (4); the first acquisition module (1) and the second acquisition module (2) are both connected to the control processing unit (3), and the control processing unit (3) is connected to the cloud server (4); The first acquisition module (1) is configured to acquire user facial expression data, form first acquisition data, and transmit it to the control processing unit (3). The control processing unit (3) transmits the received first acquisition data to the cloud server (4) in text form. The second acquisition module (2) is configured to acquire user facial expression data, form second acquisition data, and transmit it to the control processing unit (3). The control processing unit (3) performs video encoding processing on the received second acquisition data, generates video data, and transmits it to the cloud server (4). The cloud server (4) is configured to perform a first emotion analysis based on the first collected data, generate a first emotion analysis result, and send a control command to the control processing unit (3) according to the result to start or stop the second collection module (2); at the same time, the cloud server (4) performs a second emotion analysis based on the second collected data to obtain a second analysis result.

2. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 1, characterized in that, Both the first and second collected data include: eye features, head posture, eyebrow features, mouth features, forehead features, and facial muscle features.

3. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 1, characterized in that, The cloud server (4) includes a text extraction module (41), a video extraction module (42), an emotion classification unit (43), and at least two emotion analysis units (44). The text extraction module (41) is used to receive the first collected data and transmit the first collected data to the emotion classification unit (43). The emotion classification unit (43) classifies emotions based on the first collected data, outputs the emotion category, and transmits it to the corresponding emotion analysis unit (44). The emotion analysis unit (44) analyzes the emotion and outputs the first analysis result. When the first analysis result exceeds the predetermined first threshold, the data is fed back to the control processing unit (3). The control processing unit (3) controls the second acquisition module (2) to start in order to acquire the second acquisition data. The video extraction module (42) is used to extract the video information data of the second collected data and transmit the video information data to the corresponding emotion analysis unit (44). The emotion analysis unit (44) analyzes the video information data of the second collected data, outputs the second analysis result, and determines whether the second analysis result exceeds the predetermined second threshold.

4. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 3, characterized in that, It also includes a prompting device (5), which includes an optical projector (51), a speaker (52) and a central controller (53); the optical projector (51) and the speaker (52) are both connected to the central controller (53), and the central controller (53) is also connected to the emotion analysis unit (44). When the second analysis result exceeds a predetermined second threshold, the emotion analysis unit (44) sends a control signal to the central controller (53). After receiving the control signal, the central controller (53) controls the optical projector (51) to generate an image for display and project it onto the windshield of the vehicle. The central controller (53) controls the speaker (52) to emit a prompting sound.

5. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 1, characterized in that, The second acquisition module (2) includes a patch (21), a first direction adjustment component (22), a second direction adjustment component (23), and a video acquisition device (24); the patch (21) is used to attach to the inside of the windshield of a vehicle, the first direction adjustment component (22) is connected to the patch (21), the second direction adjustment component (23) is connected to the first direction adjustment component (22), and the video acquisition device (24) is connected to the second direction adjustment component (23).

6. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 5, characterized in that, The first direction adjustment assembly (22) includes a first mounting base (221), a slide groove (222), a slider (223), an adjusting rod (224), a connecting rod (225), a mounting plate (226), and a locking rod (227); the first mounting base (221) and the slide groove (222) are both fixedly connected to the patch (21), the mounting plate (226) is hinged to the first mounting base (221), the slider (223) is slidably connected to the slide groove (222), and the adjusting rod (224) is rotatably connected to the slider (223). The axis of the sliding groove (222) is parallel to the direction of movement of the slider (223). The locking rod (227) is fixedly connected to the adjusting rod (224). The axis of the locking rod (227) is perpendicular to the axis of the adjusting rod (224). Multiple limiting rods (2221) are equidistantly arranged on the lower wall of the sliding groove (222). A limiting space for accommodating the locking rod (227) is formed between adjacent limiting rods (2221). One end of the connecting rod (225) is hinged to the slider (223), and the end of the connecting rod (225) away from the slider (223) is hinged to the mounting plate (226). The second direction adjustment assembly (23) includes a cavity (231), a mounting shaft (232), a second mounting base (233), an arc-shaped elastic sheet (234), a limiting part (235), and a locking tooth (236); the cavity (231) is fixedly connected to the mounting plate (226), the second mounting base (233) is connected to the video capture device (24), the mounting shaft (232) is fixedly connected to the second mounting base (233), the mounting shaft (232) passes through the limiting part (235) and is fixedly connected to the limiting part (235), the limiting part (235) is cylindrical, the mounting shaft (232) is rotatably connected to the cavity (231), and the cavity (231) is provided with a recessed part ( 2311), the recess (2311) is used to accommodate the limiting part (235), the limiting part (235) is provided with strip-shaped protrusions (2351) arranged at equal intervals along the outer periphery of the limiting part (235), the recess (2311) of the cavity (231) is provided with a strip-shaped hole (2312), the arc-shaped elastic sheet (234) is fixedly installed in the cavity (231), the arc-shaped elastic sheet (234) is fixedly installed in the cavity (231), the locking tooth (236) is connected to the arc-shaped elastic sheet (234), the locking tooth (236) extends out from the strip-shaped hole (2312) and extends into the space between two adjacent strip-shaped protrusions (2351) to limit the limiting part (235).

7. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 6, characterized in that, The video capture device (24) includes a base (241), a first drive assembly (242), a mounting housing (243), a second drive assembly (244), and a high-definition camera (245). The base (241) is fixedly connected to the second mounting base (233), and the mounting housing (243) is rotatably connected to the base (241). The mounting housing (243) and the base (241) form an installation space for the first drive assembly (242). The first drive assembly (242) includes a gear ring (2421), a drive gear (2422), a first motor (2423), and two driven gears (2424). The first motor (2423) is fixedly connected to the base (241), the drive gear (2422) is connected to the output end of the first motor (2423), the gear ring (2421) is fixedly connected to the mounting housing (243), and the drive gear (2424) is rotatably connected to the base (241). The wheel (2422) meshes with the gear ring (2421), and the two driven gears (2424) are rotatably connected to the base (241) via shafts. The driven gears (2424) mesh with the gear ring (2421). The mounting housing (243) includes two connecting ears (2431), and the high-definition camera (245) is rotatably mounted between the two connecting ears (2431) via two rotating shafts (2451). The second drive assembly (244) is installed in one of the connecting lugs (2431). The second drive assembly (244) includes a second motor (2441), a worm (2442), and a worm wheel (2443). The output end of the second motor (2441) is connected to the worm (2442), and the worm wheel (2443) is connected to the rotating shaft (2451). The worm (2442) and the worm wheel (2443) are meshed together.

8. The in-vehicle emotion monitoring system based on a multimodal large model according to claim 1, characterized in that, It also includes an external camera (6), which is used to collect image information of the road section during vehicle travel.

9. A vehicle-mounted emotion monitoring method based on a multimodal large model, applied to the multimodal large model vehicle-mounted emotion monitoring system according to any one of claims 1 to 8, characterized in that, Includes the following steps: S1 collects the user's facial expression data through the first acquisition module (1) to form the first acquisition data, and transmits the first acquisition data to the control processing unit (3). The S2 control processing unit (3) transmits the received first collected data to the cloud server (4) in text form. The S3 cloud server (4) performs a first sentiment analysis based on the first collected data and generates a first sentiment analysis result; S4 If the first emotion analysis result does not exceed the predetermined first threshold, the processing ends; if the first emotion analysis result exceeds the predetermined first threshold, the cloud server (4) sends a control command to the control processing unit (3) to start the second acquisition module (2) for further emotion monitoring. After the S5 second acquisition module (2) is started, it acquires the user's facial expression video data and generates the second acquisition data; S6 control processing unit (3) receives the second acquired data and performs video encoding processing, generates video data and transmits it to cloud server (4). The S7 cloud server (4) performs a second emotion analysis based on the second collected data, obtains the second emotion analysis results, and determines the user's final emotional state.

10. The in-vehicle emotion monitoring method based on a multimodal large model according to claim 9, characterized in that, Also includes: Multiple emotion analysis units (44) are set in the cloud server (4), and each emotion analysis unit corresponds to a specific emotion category; The cloud server (4) performs a preliminary classification of the user's emotional state based on the first collected data, and transmits the user's emotional category to the corresponding emotion analysis unit (44) for in-depth emotion analysis. When the analysis result of a certain emotion category exceeds the predetermined emotion threshold, the cloud server (4) triggers the control processing unit (3) to start the second collection module (2) to collect video data to further verify the accuracy of the emotional state. Based on the analysis result of the second collected data, if the user's emotion still exceeds the threshold, the prompting device (5) is triggered to display the prompt information on the windshield through the optical projector (51), and at the same time, a voice warning is issued through the speaker (52) to remind the user to pay attention to emotion control.

Images