In-vehicle environment intelligent monitoring method and system, and storage medium

Abstract

The application is suitable for the field of automobile intelligent control technology, and particularly relates to an in-vehicle environment intelligent monitoring method, system and storage medium. The in-vehicle environment intelligent monitoring method comprises: acquiring in-vehicle environment time series data, and preprocessing the in-vehicle environment time series data to obtain multi-modal data; performing feature extraction on the multi-modal data; and based on a multi-head cross attention mechanism, performing interactive fusion on visual features, audio features and environmental features, and based on a space-time attention network, capturing time series dynamic features in the cross-modal fusion features; based on Shannon entropy, dynamically distributing the weights corresponding to the enhanced visual features, audio features and environmental features; according to the final fusion features, predicting a risk probability to obtain a prediction result; and when the prediction result is an in-vehicle environment anomaly, issuing a warning notification. Compared with the prior art, the application has higher accuracy, real-time performance and robustness in in-vehicle environment risk identification.

Description

Technical Field

[0001] This invention relates to the field of automotive intelligent control technology, and in particular to a method, system and storage medium for intelligent monitoring of the in-vehicle environment. Background Technology

[0002] With the rapid development of intelligent transportation and automotive electronics technologies, in-vehicle environmental safety is receiving increasing attention. The in-vehicle environment includes multi-dimensional information such as temperature, humidity, PM2.5, harmful gases (such as CO and formaldehyde), smoke, and abnormal noises. Accurate monitoring of this environment is of great significance for protecting the health of drivers and passengers and preventing safety accidents (such as fires and poisoning).

[0003] However, the existing methods for monitoring the in-vehicle environment have the following shortcomings:

[0004] (1) Lack of semantic interaction in multimodal fusion: Existing fusion methods have not established semantic associations between cross-modal features, and cannot effectively utilize the complementary information between visual, audio, and environmental sensor data, resulting in feature redundancy and loss of key information. For example, there is a strong correlation between smoke images in visual data and abnormal popping sounds in audio data and temperature rise signals from environmental sensors, but existing methods are unable to effectively capture this cross-modal dependency, resulting in insufficient feature representation ability after fusion.

[0005] (2) Insufficient extraction of spatiotemporal features: Existing methods independently model the temporal and spatial distribution features of environmental data, failing to form a joint spatiotemporal representation and thus unable to capture the spatiotemporal coupling patterns of environmental changes. In-vehicle environmental data exhibits significant spatiotemporal dynamic characteristics, such as the spatial distribution and temporal evolution of smoke diffusion, and the changing trend of odor concentration over time. Existing methods often employ fixed-structure neural networks (such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to process spatial or temporal features separately, lacking the ability to jointly model spatiotemporal features and making it difficult to capture the dynamic patterns of environmental changes.

[0006] (3) Insufficient attention to key information: Existing methods do not dynamically evaluate the importance of features of each modality and dimension during feature processing, resulting in low model learning efficiency and weak anti-interference ability. In the in-vehicle environment, some features (such as the color and texture of smoke and the concentration peak of abnormal odors) play a decisive role in risk identification, but existing methods treat all features equally and lack an adaptive focusing mechanism for key information, which makes the model prone to false alarms or missed detections in complex noisy environments.

[0007] Therefore, there is an urgent need for a new intelligent monitoring method, system, and storage medium for the in-vehicle environment to solve the above-mentioned technical problems. Summary of the Invention

[0008] This invention provides a method, system, and storage medium for intelligent monitoring of the in-vehicle environment. The purpose of this invention is to improve the accuracy, real-time performance, and robustness of in-vehicle environment risk identification.

[0009] In a first aspect, the present invention provides a method for intelligent monitoring of the in-vehicle environment, the method comprising the following steps: S1. Acquire time-series data of the in-vehicle environment and preprocess the time-series data of the in-vehicle environment to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data; S2. Perform feature extraction on the multimodal data to obtain visual features, audio features, and environmental features; S3. Enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and then interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain spatiotemporal temporal features; dynamically allocate the weights corresponding to the enhanced visual features, audio features, and environmental features based on Shannon entropy to obtain Shannon fusion features. S4. The spatiotemporal time series features and the Shannon fusion features are fused to obtain the final fused features; S5. Perform risk probability prediction based on the final fusion features to obtain the prediction result; when the prediction result indicates an abnormal in-vehicle environment, issue an early warning notification.

[0010] Preferably, step S1 includes the following sub-steps: S11. The in-vehicle environment timing data is timestamped using a hardware synchronization clock. S12. Based on the target detection model, the visual data in the time series data of the in-vehicle environment after the timestamp alignment is normalized to obtain the visual modal data. The audio data in the time-series data of the in-vehicle environment after timestamp alignment is converted into a Mel spectrogram through short-time Fourier transform, and the Mel spectrogram is filtered to obtain the audio modal data. The environmental data in the in-vehicle environment time series data after timestamp alignment is standardized and subjected to sliding window processing to obtain the environmental modal data.

[0011] Preferably, step S2 includes the following sub-steps: The visual modality data is feature extracted based on a residual network, and the visual features are obtained by global average pooling compression. The audio features are obtained by extracting features from the audio modal data using a convolutional neural network. The environmental modal data is mapped to a high-dimensional feature space to obtain the environmental features.

[0012] Preferably, step S3 includes the following sub-steps: S31. Using the cross-modal fusion features as input to the spatiotemporal attention network, position encoding is performed on the cross-modal fusion features to obtain position-encoded features; S32. The position encoding features are concatenated based on a multi-head self-attention mechanism to obtain multi-head concatenated features; S33. Perform nonlinear transformation on the multi-head splicing features based on a feedforward network, and calculate the time step weights of the multi-head splicing features using a time attention vector; S34. Calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

[0013] Preferably, step S3 includes the following sub-steps: S35. Normalize the enhanced visual features, audio features, and environmental features respectively, and calculate the probability distributions corresponding to the enhanced visual features, audio features, and environmental features respectively to obtain the distribution results; S36. Calculate the Shannon entropy based on the distribution results to obtain the Shannon entropy value; S37. Adjust and fuse the weights corresponding to the enhanced visual features, audio features and environmental features according to the Shannon entropy value to obtain the Shannon fused features.

[0014] Secondly, the present invention also provides an intelligent in-vehicle environment monitoring system, comprising: The preprocessing module is used to acquire time-series data of the in-vehicle environment and preprocess the time-series data of the in-vehicle environment to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data; The feature extraction module is used to extract features from the multimodal data to obtain visual features, audio features, and environmental features; The capture module is used to enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and to interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; and to capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain spatiotemporal temporal features. And to dynamically allocate the weights corresponding to the enhanced visual features, the audio features and the environmental features based on Shannon entropy, respectively, to obtain Shannon fusion features; The fusion module is used to fuse the spatiotemporal time-series features and the Shannon fusion features to obtain the final fused features; The risk prediction module is used to predict the risk probability based on the final fusion features and obtain the prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, an early warning notification is issued.

[0015] Preferably, the feature extraction module includes the following sub-units: The visual extraction unit is used to extract features from the visual modality data based on the residual network and to obtain the visual features through global average pooling compression processing. An audio extraction unit is used to extract features from the audio modality data based on a convolutional neural network to obtain the audio features; An environment extraction unit is used to map the environmental modal data to a high-dimensional feature space to obtain the environmental features.

[0016] Preferably, the capture module includes the following sub-units: The encoding unit is used to take the cross-modal fusion features as input to the spatiotemporal attention network and perform position encoding on the cross-modal fusion features to obtain position-encoded features; The splicing unit is used to splice the positional encoding features based on a multi-head self-attention mechanism to obtain multi-head spliced ​​features; The transformation unit is used to perform nonlinear transformation on the multi-head splicing features based on the feedforward network, and to calculate the time step weights of the multi-head splicing features through the time attention vector; The calculation unit is used to calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

[0017] Thirdly, the present invention also provides a computer device, including: a memory, a processor, and an in-vehicle environment intelligent monitoring program stored in the memory and executable on the processor, wherein the processor executes the in-vehicle environment intelligent monitoring program to implement the steps of the in-vehicle environment intelligent monitoring method as described in any of the above embodiments.

[0018] Fourthly, the present invention also provides a computer-readable storage medium storing an in-vehicle environment intelligent monitoring program, wherein the in-vehicle environment intelligent monitoring program, when executed by a processor, implements the steps of the in-vehicle environment intelligent monitoring method as described in any of the above embodiments.

[0019] Compared with existing technologies, this invention collects environmental, audio, and visual data from all dimensions within the vehicle and achieves deep semantic interaction between different modal data through a multi-head attention mechanism. This enables visual, audio, and environmental features to guide and enhance each other, effectively solving the problem of insufficient cross-modal feature interaction in existing technologies and constructing a more complete in-vehicle environment representation model.

[0020] Secondly, this invention uses a spatiotemporal attention network to jointly model cross-modal fusion features in both temporal and spatial dimensions. At the temporal level, it captures the temporal patterns of environmental changes through dynamic gating recurrent units, while at the spatial level, it utilizes a multi-scale convolutional attention network to achieve hierarchical feature extraction for complex scenes. This spatiotemporal feature fusion technology forms a refined spatiotemporal dynamic representation, overcoming the limitations of existing models' fragmented spatiotemporal modeling and significantly improving the ability to capture features of abnormal events inside the vehicle.

[0021] This invention dynamically allocates weights to the enhanced visual features, audio features, and environmental features based on Shannon entropy, thereby significantly improving the adaptability to complex working conditions and effectively solving the problem of weak scene adaptability in existing fixed-weight fusion methods. This invention effectively improves the accuracy, real-time performance, and robustness of in-vehicle environment risk identification. Attached Figure Description

[0022] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings: Figure 1 This is a flowchart of the intelligent monitoring method for the in-vehicle environment provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the multi-head cross-attention mechanism structure of the intelligent in-vehicle environment monitoring method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the structure of the intelligent in-vehicle environment monitoring system provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Example 1 Please refer to Figure 1 and Figure 2This invention provides an intelligent monitoring method for the in-vehicle environment, the method comprising the following steps: S1. Acquire time-series data of the in-vehicle environment and preprocess the time-series data of the in-vehicle environment to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data; In this embodiment of the invention, a multimodal sensor array (environmental sensors, vehicle-mounted camera, microphone) is used to collect time-series data of the in-vehicle environment, real-time acquiring multi-dimensional data such as environmental data (temperature, humidity, light intensity), visual data (visual images), and audio data (speech signals). For example, the system deploys six types of environmental sensors, including temperature, humidity, and PM2.5 sensors (outputting time-series data). ), 1920×1080 resolution / 25fps vehicle-mounted camera (capturing visual images) ) and a microphone with a 44.1kHz sampling rate (for acquiring audio signals) ).

[0025] Step S1 includes the following sub-steps: S11. The in-vehicle environment timing data is timestamped using a hardware synchronization clock (error ±5ms). The hardware synchronization clock is a system that synchronizes clock signals between different devices using physical layer signals or dedicated hardware circuits, ensuring consistency of physical layer signals or distributed devices on a time base.

[0026] S12. Based on the object detection model, the visual data in the time-series data of the in-vehicle environment after timestamp alignment is normalized to obtain the visual modality data; wherein, the visual data is normalized to [0,1] after being extracted and cropped from the key areas of the cockpit by the pre-trained object detection model, and the normalization formula is: .

[0027] The audio data in the in-vehicle environment time-series data, after being timestamped, is converted into a Mel spectrogram using a Short-Time Fourier Transform (STFT). The Mel spectrogram is then filtered to obtain the audio modal data. Specifically, the Mel spectrogram is filtered by multiple filters to output audio modal data of dimension T×100×40 (where T is the feature matrix of time-series length).

[0028] The environmental data in the in-vehicle environment time series data after timestamp alignment is standardized and processed using a sliding window to obtain the environmental modal data. Specifically, after Z-score standardization, the environmental data is used to generate environmental modal data (T×6 time series) through a 10-second sliding window.

[0029] S2. Perform feature extraction on the multimodal data to obtain visual features, audio features, and environmental features; In this embodiment of the invention, step S2 includes the following sub-steps: S21. The visual modality data is extracted based on the residual network (ResNet50), and the visual features are obtained by global average pooling compression.

[0030] Specifically, a residual network is used to extract features from the visual modality data, and a first feature matrix is ​​output. Visual features are compressed using global average pooling. ( To unify feature dimensions.

[0031] S22. Extract features from the audio modal data based on a convolutional neural network (CNN) to obtain the audio features.

[0032] Specifically, a second feature matrix is ​​output by extracting features from the audio modal data using a convolutional neural network. After pooling, it is converted into audio features. .

[0033] S23. Map the environmental modal data to a high-dimensional feature space to obtain the environmental features. Specifically, the environmental modal data is mapped to a high-dimensional feature space through a fully connected layer. Mapped to environmental features This enables dimensional alignment between modalities.

[0034] S3. Enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and then interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain spatiotemporal temporal features; dynamically allocate the weights corresponding to the enhanced visual features, audio features, and environmental features based on Shannon entropy to obtain Shannon fusion features. In this embodiment of the invention, a multi-head cross-attention mechanism is constructed to achieve bidirectional information interaction between visual features, audio features, and environmental features. The inter-modal cross-attention calculation is as follows: for any two modalities... (Query modality) and (Key / value modality).

[0035] Cross-attention output for: .

[0036] ; in Let be the projection matrix. The key dimension is defined as follows. Subsequently, in order to perform interactive fusion of the three features, cross-attention operations were performed on the visual features, audio features, and environmental features respectively to obtain the enhanced features.

[0037] Visual Enhancement: Visual Features that Integrate Audio and Environmental Information ; Audio Enhancement: Audio Features that Integrate Visual and Environmental Information ; Environmental Enhancement: Environmental Features that Integrate Visual and Audio Information , Furthermore, the original modal features are preserved through residual connections to avoid information loss. This invention utilizes a multi-head cross-attention mechanism to enable each modality to explicitly focus on key regions of other modalities (such as the location of abnormal audio sources corresponding to visual smoke regions). Compared to traditional fixed fusion strategies, it can dynamically capture cross-modal semantic associations. For example, when the visual branch detects smoke, cross-attention will enhance the attention weight of the audio branch to concurrent abnormal combustion noise, achieving synergistic enhancement of "visual-audio" abnormal features.

[0038] This invention constructs a cross-modal semantic interaction mechanism based on bidirectional cross-attention, enabling three pairs of bidirectional information interactions: visual-audio, visual-environment, and audio-environment. Specifically, for any two modalities m (query modality) and n (key / value modality), a cross-modal attention weight matrix is ​​calculated through cross-attention. This allows the features of the query modality to dynamically adjust their focus based on the content of the key / value modality, thereby enhancing the modal feature representation. For example, when the visual branch detects a smoke region, the cross-attention mechanism dynamically increases the attention weight of the audio branch to concurrent abnormal combustion noise, achieving synergistic enhancement of the "visual-audio" abnormal features.

[0039] The three enhanced features of the cross-attention output are concatenated along the channel dimension to obtain cross-modal fusion features. And compressed to through a shared fully connected layer To achieve semantic alignment between modalities ; in Indicates feature splicing, It is a fully connected layer.

[0040] In cross-modal fusion features Based on this, a spatiotemporal attention network is constructed to capture the temporal dynamic characteristics of environmental changes, enabling the modeling of gradual risks such as smoke diffusion and sudden temperature rise.

[0041] First, the cross-modal fused features are input into the multi-layer Transformer encoder of the spatiotemporal attention network. Each encoder layer contains multi-head self-attention and a feedforward neural network (FFN). The specific process is as follows: S31. Using the cross-modal fusion features as input to the spatiotemporal attention network, position encoding is performed on the cross-modal fusion features to obtain position-encoded features. That is, the cross-modal fusion features... Add learnable positional coding .

[0042] S33. The positional encoding features are concatenated based on a multi-head self-attention mechanism to obtain multi-head concatenated features. Among them, there are... Through multi-head self-attention mechanism, Mapped to Grouped queries - key-value pairs, attention is calculated for each group as follows: ; in The final output is the head-joining feature of the multi-head concatenation result: , ( ) represents the feature splicing function.

[0043] S33. Perform nonlinear transformation on the multi-head splicing features based on the feedforward network, and calculate the time step weights of the multi-head splicing features through the time attention vector.

[0044] Specifically, the multi-head splicing features of the attention output are processed by two fully connected layers of the feedforward network, and a nonlinear transformation is performed to satisfy: ,in, b1 and b2 are learnable bias coefficients for the activation function. and This represents the learnable weight coefficients.

[0045] Time step weights are generated using time attention vectors. Focus on key time segments (such as the time when the smoke first appears): ; in This is a learnable temporal attention vector.

[0046] S34. Calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

[0047] Specifically, the time series characteristics satisfy: ; This invention uses this mechanism to capture the causal relationship between temperature and smoke concentration over time. For example, when smoke appears after the temperature continues to rise, the feature weight of the corresponding time step is automatically increased.

[0048] The present invention, through steps S31-S34, can accurately focus on the key time segment of the first mutation of abnormal indicators and automatically capture the causal relationship between temperature and smoke concentration.

[0049] In this embodiment of the invention, Shannon entropy quantification is introduced to quantify the reliability of each modality, achieving dynamic weight allocation and thus forcibly focusing on high-reliability key features with low entropy values. Modal uncertainty measurement is applied to the temporal features of the visual, audio, and environmental modalities. The steps to calculate the entropy value are as follows: S35. Normalize the enhanced visual features, audio features, and environmental features respectively, and calculate the probability distributions corresponding to the enhanced visual features, audio features, and environmental features respectively to obtain the distribution results. Specifically, normalize each modal feature along the channel dimension into a probability distribution. ,in visual features Audio features and environmental characteristics 'c' represents the index of the feature channel.

[0050] S36. Calculate the Shannon entropy based on the distribution results to obtain the Shannon entropy value. Specifically, the Shannon entropy calculation satisfies the following conditions: ; Low entropy values ​​indicate a concentrated distribution of features, which corresponds to key information such as visual features of smoke and high-frequency components of abnormal audio.

[0051] S37. Adjust and fuse the weights corresponding to the enhanced visual features, audio features, and environmental features according to the Shannon entropy values ​​to obtain the Shannon fused features. Specifically, by using exponential normalization to generate modal fusion weights, the model is forced to focus on low-entropy modalities: ; in, This is a temperature hyperparameter that adjusts the influence of entropy on the weights.

[0052] The final fused features satisfy the following conditions: ; For example, when the visual modality detects smoke ( During a sudden drop, Rapidly increase to over 70% to ensure that smoke characteristics dominate the fusion.

[0053] S4. The spatiotemporal time series features and the Shannon fusion features are fused to obtain the final fusion features.

[0054] In this embodiment of the invention, the spatiotemporal and temporal characteristics representing the long-term cross-modal evolution law are used. Combined with Shannon's fusion features, which represent a state of instantaneous high reliability. To achieve the following: This allows for the final fusion of features, enabling the classifier to combine dynamic trend analysis with instantaneous anti-interference capabilities.

[0055] S5. Perform risk probability prediction based on the final fusion features to obtain the prediction result; when the prediction result indicates an abnormal in-vehicle environment, issue an early warning notification.

[0056] In this embodiment of the invention, the final fused features are obtained through a classifier. Achieve two-level risk classification (normal / abnormal); the classifier architecture uses a two-layer fully connected network + Softmax. Its output probability vector This corresponds to the "normal" and "abnormal" categories. When an abnormal state occurs, a red indicator light will be triggered to alarm and notify the occupants of the vehicle.

[0057] In this embodiment of the invention, to verify the effectiveness of the intelligent in-vehicle environment monitoring of the present invention, a comparative experiment was constructed to evaluate the present invention and existing typical solutions based on two core indicators: risk identification accuracy and false alarm rate. The details are as follows: Since there is currently no fully publicly available multimodal synchronous dataset that includes in-vehicle cockpit scenarios, this invention employs a semi-synthetic multimodal data construction method to verify the algorithm while ensuring that the data distribution closely approximates the real in-vehicle environment. The specific construction process is as follows: (1) Visual data: Public smoke and fire video datasets (D-Fire, KMUFireSmoke) were selected, and video clips containing smoke diffusion and open flames were extracted. The key areas of the cockpit were cropped according to the perspective of the vehicle camera. The frame rate was uniformly 25fps and the resolution was uniformly 1920×1080. (2) Audio data: Abnormal acoustic segments related to burning, explosion, and glass breakage were selected from the public acoustic database (Freesound) and aligned with the above visual segments by timestamp, with a sampling rate of 44.1kHz. (3) Environmental sensor data: Based on the typical range of real vehicle-mounted environmental sensors, 6-dimensional time series data are generated by program synthesis, including temperature, humidity, PM2.5, CO, formaldehyde and smoke concentration. The normal operating condition data follows a Gaussian distribution, and the abnormal operating condition data is superimposed with step increase or steep increase characteristics in the corresponding time window, and ±5% random noise is added to simulate sensor measurement error. (4) Time alignment: The three types of modal data are uniformly divided into 10-second sliding windows with a time step and aligned by high-precision timestamps. The maximum alignment error is controlled within ±5ms, which is consistent with the hardware synchronization mechanism described in this invention.

[0058] Three representative solutions in the field of in-vehicle environment monitoring were selected as baselines for comparison: Table 1: Comparison of processing methods among different approaches

[0059] This invention is evaluated using the following three commonly used indicators: Accuracy: The proportion of correctly identified samples out of the total number of samples; Recall: The proportion of samples that are actually anomalous that are correctly identified, reflecting the number of missed detections; False Alarm Rate (FPR): The proportion of normal samples that are incorrectly identified as abnormal, reflecting the stability of the system.

[0060] Under the same experimental conditions, the performance comparison of each scheme is shown in the table below: Table 2: Performance Comparison of Various Methods

[0061] The results above show that: Compared with traditional early and late fusion schemes, this invention improves accuracy by more than 7% and reduces false alarm rate by more than 5 percentage points, indicating that multi-head attention can uncover semantic relationships between visual, audio, and environmental parameters and reduce interference from irrelevant features. Compared to a unidirectional cross-modal attention scheme, this invention improves recall by approximately 4.7%, indicating that the trimodal pairwise interactive feature enhancement mechanism can capture the synergistic change patterns of smoke diffusion, temperature surge, and abnormal sounds. In terms of false alarm rate, the present invention has a false alarm rate of only 3.2%, which is lower than that of the baseline schemes. This is consistent with the design intention of the entropy-driven adaptive fusion strategy to dynamically focus on low-entropy and high-reliability features in complex noise environments.

[0062] The experimental results demonstrate that this invention, through the synergistic effect of bidirectional cross-modal semantic interaction, spatiotemporal joint modeling, and entropy-driven adaptive fusion, outperforms the baseline scheme in terms of accuracy, timeliness, and anti-interference capability in in-vehicle environment risk identification. This helps to address the shortcomings of traditional methods in multimodal fusion efficiency, spatiotemporal feature extraction, and key information focusing. Compared with existing technologies, this invention, by collecting in-vehicle environmental, audio, and visual data across all dimensions and achieving deep semantic interaction between different modalities through a multi-head attention mechanism, enables visual, audio, and environmental features to mutually guide and enhance each other. This effectively solves the problem of insufficient cross-modal feature interaction in existing technologies and constructs a more complete in-vehicle environment representation model.

[0063] Secondly, this invention uses a spatiotemporal attention network to jointly model cross-modal fusion features in both temporal and spatial dimensions. At the temporal level, it captures the temporal patterns of environmental changes through dynamic gating recurrent units, while at the spatial level, it utilizes a multi-scale convolutional attention network to achieve hierarchical feature extraction for complex scenes. This spatiotemporal feature fusion technology forms a refined spatiotemporal dynamic representation, overcoming the limitations of existing models' fragmented spatiotemporal modeling and significantly improving the ability to capture features of abnormal events inside the vehicle.

[0064] This invention dynamically allocates weights to the enhanced visual features, audio features, and environmental features based on Shannon entropy, thereby significantly improving the adaptability to complex working conditions and effectively solving the problem of weak scene adaptability in existing fixed-weight fusion methods. This invention effectively improves the accuracy, real-time performance, and robustness of in-vehicle environment risk identification.

[0065] Example 2 This invention also provides an intelligent in-vehicle environment monitoring system, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of the intelligent in-vehicle environment monitoring system 200 provided in an embodiment of the present invention, which includes: The preprocessing module 201 is used to acquire in-vehicle environment time-series data and preprocess the in-vehicle environment time-series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data.

[0066] The preprocessing module 201 includes the following sub-units: The alignment unit is used to timestamp-align the in-vehicle environment timing data using a hardware synchronization clock. The visual modality unit is used to normalize the visual data in the time-series data of the in-vehicle environment after time-stamp alignment based on the target detection model to obtain the visual modality data. The audio modal unit is used to convert the audio data in the time-series data of the in-vehicle environment after time-stamp alignment into a Mel spectrogram through a short-time Fourier transform, and to filter the Mel spectrogram to obtain the audio modal data. The environment modality unit is used to perform standardization and sliding window processing on the environmental data in the time-series data of the in-vehicle environment after timestamp alignment to obtain the environment modality data.

[0067] The feature extraction module 202 is used to extract features from the multimodal data to obtain visual features, audio features, and environmental features.

[0068] The feature extraction module 202 includes the following sub-units: The visual extraction unit is used to extract features from the visual modality data based on the residual network and to obtain the visual features through global average pooling compression processing. An audio extraction unit is used to extract features from the audio modality data based on a convolutional neural network to obtain the audio features; An environment extraction unit is used to map the environmental modal data to a high-dimensional feature space to obtain the environmental features.

[0069] The capture module 203 is used to enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and to interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; to capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain spatiotemporal temporal features; and to dynamically allocate the weights corresponding to the enhanced visual features, audio features, and environmental features based on Shannon entropy to obtain Shannon fusion features.

[0070] The capture module 203 includes the following sub-units: The encoding unit is used to take the cross-modal fusion features as input to the spatiotemporal attention network and perform position encoding on the cross-modal fusion features to obtain position-encoded features; The splicing unit is used to splice the positional encoding features based on a multi-head self-attention mechanism to obtain multi-head spliced ​​features; The transformation unit is used to perform nonlinear transformation on the multi-head splicing features based on the feedforward network, and to calculate the time step weights of the multi-head splicing features through the time attention vector; The calculation unit is used to calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

[0071] The capture module 203 further includes the following sub-units: The probability distribution unit is used to normalize the enhanced visual features, audio features, and environmental features respectively, and calculate the probability distributions corresponding to the enhanced visual features, audio features, and environmental features respectively to obtain the distribution results; Shannon calculation unit, used to calculate Shannon entropy based on the distribution results, to obtain Shannon entropy value; The Shannon fusion unit adjusts and fuses the weights corresponding to the enhanced visual features, audio features, and environmental features according to the Shannon entropy value to obtain the Shannon fused features.

[0072] The fusion module 204 is used to fuse the spatiotemporal time series features and the Shannon fusion features to obtain the final fused features; The risk prediction module 205 is used to predict the risk probability based on the final fusion features and obtain the prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, an early warning notification is issued.

[0073] The in-vehicle environment intelligent monitoring system 200 can implement the steps in the in-vehicle environment intelligent monitoring method in the above embodiments and achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0074] Example 3 This invention also provides a computer device, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and an in-vehicle environment intelligent monitoring program stored in the memory 302 and capable of running on the processor 301.

[0075] The processor 301 calls the in-vehicle environment intelligent monitoring program stored in the memory 302 and executes the steps in the in-vehicle environment intelligent monitoring method provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps: S1. Acquire time-series data of the in-vehicle environment and preprocess the time-series data of the in-vehicle environment to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data; In this embodiment of the invention, a multimodal sensor array (environmental sensors, vehicle-mounted camera, microphone) is used to collect time-series data of the in-vehicle environment, real-time acquiring multi-dimensional data such as environmental data (temperature, humidity, light intensity), visual data (visual images), and audio data (speech signals). For example, the system deploys six types of environmental sensors, including temperature, humidity, and PM2.5 sensors (outputting time-series data). ), 1920×1080 resolution / 25fps vehicle-mounted camera (capturing visual images) ) and a microphone with a 44.1kHz sampling rate (for acquiring audio signals) ).

[0076] Step S1 includes the following sub-steps: S11. The in-vehicle environment timing data is timestamped using a hardware synchronization clock (error ±5ms). The hardware synchronization clock is a system that synchronizes clock signals between different devices using physical layer signals or dedicated hardware circuits, ensuring consistency of physical layer signals or distributed devices on a time base.

[0077] S12. Based on the object detection model, the visual data in the time-series data of the in-vehicle environment after timestamp alignment is normalized to obtain the visual modality data; wherein, the visual data is normalized to [0,1] after being extracted and cropped from the key areas of the cockpit by the pre-trained object detection model, and the normalization formula is: .

[0078] The audio data in the in-vehicle environment time-series data, after being timestamped, is converted into a Mel spectrogram using a Short-Time Fourier Transform (STFT). The Mel spectrogram is then filtered to obtain the audio modal data. Specifically, the Mel spectrogram is filtered by multiple filters to output audio modal data of dimension T×100×40 (where T is the feature matrix of time-series length).

[0079] The environmental data in the in-vehicle environment time series data after timestamp alignment is standardized and processed using a sliding window to obtain the environmental modal data. Specifically, after Z-score standardization, the environmental data is used to generate environmental modal data (T×6 time series) through a 10-second sliding window.

[0080] S2. Perform feature extraction on the multimodal data to obtain visual features, audio features, and environmental features; In this embodiment of the invention, step S2 includes the following sub-steps: S21. The visual modality data is extracted based on the residual network (ResNet50), and the visual features are obtained by global average pooling compression.

[0081] Specifically, a residual network is used to extract features from the visual modality data, and a first feature matrix is ​​output. Visual features are compressed using global average pooling. ( To unify feature dimensions.

[0082] S22. Extract features from the audio modal data based on a convolutional neural network (CNN) to obtain the audio features.

[0083] Specifically, a second feature matrix is ​​output by extracting features from the audio modal data using a convolutional neural network. After pooling, it is converted into audio features. .

[0084] S23. Map the environmental modal data to a high-dimensional feature space to obtain the environmental features. Specifically, the environmental modal data is mapped to a high-dimensional feature space through a fully connected layer. Mapped to environmental features This enables dimensional alignment between modalities.

[0085] S3. Enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and then interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain spatiotemporal temporal features; dynamically allocate the weights corresponding to the enhanced visual features, audio features, and environmental features based on Shannon entropy to obtain Shannon fusion features. In this embodiment of the invention, a multi-head cross-attention mechanism is constructed to achieve bidirectional information interaction among visual features, audio features, and environmental features. The inter-modal cross-attention calculation is as follows: for any two modalities... (Query modality) and (Key / value modality).

[0086] Cross-attention output for: .

[0087] ; in For the projection matrix, The key dimension is defined as follows. Subsequently, in order to perform interactive fusion of the three features, cross-attention operations were performed on the visual features, audio features, and environmental features respectively to obtain the enhanced features.

[0088] Visual Enhancement: Visual Features that Integrate Audio and Environmental Information ; Audio Enhancement: Audio Features that Integrate Visual and Environmental Information ; Environmental Enhancement: Environmental Features that Integrate Visual and Audio Information , Furthermore, the original modal features are preserved through residual connections to avoid information loss. This invention utilizes a multi-head cross-attention mechanism to enable each modality to explicitly focus on key regions of other modalities (such as the location of abnormal audio sources corresponding to visual smoke regions). Compared to traditional fixed fusion strategies, it can dynamically capture cross-modal semantic associations. For example, when the visual branch detects smoke, cross-attention will enhance the attention weight of the audio branch to concurrent abnormal combustion noise, achieving synergistic enhancement of "visual-audio" abnormal features.

[0089] This invention constructs a cross-modal semantic interaction mechanism based on bidirectional cross-attention, enabling three pairs of bidirectional information interactions: visual-audio, visual-environment, and audio-environment. Specifically, for any two modalities m (query modality) and n (key / value modality), a cross-modal attention weight matrix is ​​calculated through cross-attention. This allows the features of the query modality to dynamically adjust their focus based on the content of the key / value modality, thereby enhancing the modal feature representation. For example, when the visual branch detects a smoke region, the cross-attention mechanism dynamically increases the attention weight of the audio branch to concurrent abnormal combustion noise, achieving synergistic enhancement of the "visual-audio" abnormal features.

[0090] The three enhanced features of the cross-attention output are concatenated along the channel dimension to obtain cross-modal fusion features. And compressed to through a shared fully connected layer To achieve semantic alignment between modalities ; in Indicates feature splicing, It is a fully connected layer.

[0091] In cross-modal fusion features Based on this, a spatiotemporal attention network is constructed to capture the temporal dynamic characteristics of environmental changes, enabling the modeling of gradual risks such as smoke diffusion and sudden temperature rise.

[0092] First, the cross-modal fused features are input into the multi-layer Transformer encoder of the spatiotemporal attention network. Each encoder layer contains multi-head self-attention and a feedforward neural network (FFN). The specific process is as follows: S31. Using the cross-modal fusion features as input to the spatiotemporal attention network, position encoding is performed on the cross-modal fusion features to obtain position-encoded features. That is, the cross-modal fusion features... Add learnable positional coding .

[0093] S32. The positional encoding features are concatenated based on a multi-head self-attention mechanism to obtain multi-head concatenated features. Among them, there are... Through multi-head self-attention mechanism, Mapped to Grouped queries - key-value pairs, attention is calculated for each group as follows: ; in The final output is the head-joining feature of the multi-head concatenation result: , ( ) represents the feature splicing function.

[0094] S33. Perform nonlinear transformation on the multi-head splicing features based on the feedforward network, and calculate the time step weights of the multi-head splicing features through the time attention vector.

[0095] Specifically, the multi-head splicing features of the attention output are processed by two fully connected layers of the feedforward network, and a nonlinear transformation is performed to satisfy: ,in, b1 and b2 are learnable bias coefficients for the activation function. and This represents the learnable weight coefficients.

[0096] Time step weights are generated using time attention vectors. Focus on key time segments (such as the time when the smoke first appears): ; in This is a learnable temporal attention vector.

[0097] S34. Calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

[0098] Specifically, the time series characteristics satisfy: ; This invention uses this mechanism to capture the causal relationship between temperature and smoke concentration over time. For example, when smoke appears after the temperature continues to rise, the feature weight of the corresponding time step is automatically increased.

[0099] The present invention, through steps S31-S34, can accurately focus on the key time segment of the first mutation of abnormal indicators and automatically capture the causal relationship between temperature and smoke concentration.

[0100] In this embodiment of the invention, Shannon entropy quantification is introduced to quantify the reliability of each modality, achieving dynamic weight allocation and thus forcibly focusing on key features with low entropy values ​​and high reliability. Modal uncertainty measurement is applied to the temporal features of the visual, audio, and environmental modalities. The steps to calculate the entropy value are as follows: S35. Normalize the enhanced visual features, audio features, and environmental features respectively, and calculate the probability distributions corresponding to the enhanced visual features, audio features, and environmental features respectively to obtain the distribution results. Specifically, normalize each modal feature along the channel dimension into a probability distribution. ,in visual features Audio features and environmental characteristics 'c' represents the index of the feature channel.

[0101] S36. Calculate the Shannon entropy based on the distribution results to obtain the Shannon entropy value. Specifically, the Shannon entropy calculation satisfies the following conditions: ; Low entropy values ​​indicate a concentrated distribution of features, which corresponds to key information such as visual features of smoke and high-frequency components of abnormal audio.

[0102] S37. Adjust and fuse the weights corresponding to the enhanced visual features, audio features and environmental features according to the Shannon entropy value to obtain the Shannon fused features.

[0103] Specifically, by using exponential normalization to generate modality fusion weights, the model is forced to focus on low-entropy modes: ; in, This is a temperature hyperparameter that adjusts the influence of entropy on the weights.

[0104] The final fused features satisfy the following conditions: ; For example, when the visual modality detects smoke ( During a sudden drop, Rapidly increase to over 70% to ensure that smoke characteristics dominate the fusion.

[0105] S4. The spatiotemporal time series features and the Shannon fusion features are fused to obtain the final fusion features.

[0106] In this embodiment of the invention, the spatiotemporal and temporal characteristics representing the long-term cross-modal evolution law are used. Combined with Shannon's fusion features, which represent a state of instantaneous high reliability. To achieve the following: This allows for the final fusion of features, enabling the classifier to combine dynamic trend analysis with instantaneous anti-interference capabilities.

[0107] S5. Perform risk probability prediction based on the final fusion features to obtain the prediction result; when the prediction result indicates an abnormal in-vehicle environment, issue an early warning notification.

[0108] In this embodiment of the invention, the final fused features are obtained through a classifier. Achieve two-level risk classification (normal / abnormal); the classifier architecture uses a two-layer fully connected network + Softmax. Its output probability vector This corresponds to the "normal" and "abnormal" categories. When an abnormal state occurs, a red indicator light will be triggered to alarm and notify the occupants of the vehicle.

[0109] The computer device 300 provided in this embodiment of the invention can implement the steps in the intelligent monitoring method of the in-vehicle environment as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0110] Example 4 This invention also provides a computer-readable storage medium storing an intelligent in-vehicle environment monitoring program. When executed by a processor, the intelligent in-vehicle environment monitoring program implements the various processes and steps of the intelligent in-vehicle environment monitoring method provided in this invention and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0111] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0112] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0114] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form without departing from the spirit and scope of the claims of the present invention, and all such changes are within the protection scope of the present invention.

Claims

1. A method for intelligent monitoring of the in-vehicle environment, characterized in that, The intelligent monitoring method for the in-vehicle environment includes the following steps: S1. Acquire time-series data of the in-vehicle environment and preprocess the time-series data of the in-vehicle environment to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data; S2. Perform feature extraction on the multimodal data to obtain visual features, audio features, and environmental features; S3. Enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and then interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain spatiotemporal temporal features; dynamically allocate the weights corresponding to the enhanced visual features, audio features, and environmental features based on Shannon entropy to obtain Shannon fusion features. S4. The spatiotemporal time series features and the Shannon fusion features are fused to obtain the final fused features; S5. Perform risk probability prediction based on the final fusion features to obtain the prediction result; when the prediction result indicates an abnormal in-vehicle environment, issue an early warning notification.

2. The intelligent monitoring method for the in-vehicle environment as described in claim 1, characterized in that, Step S1 includes the following sub-steps: S11. The in-vehicle environment timing data is timestamped using a hardware synchronization clock. S12. Based on the target detection model, the visual data in the time series data of the in-vehicle environment after the timestamp alignment is normalized to obtain the visual modal data. The audio data in the time-series data of the in-vehicle environment after timestamp alignment is converted into a Mel spectrogram through short-time Fourier transform, and the Mel spectrogram is filtered to obtain the audio modal data. The environmental data in the in-vehicle environment time series data after timestamp alignment is standardized and subjected to sliding window processing to obtain the environmental modal data.

3. The intelligent monitoring method for the in-vehicle environment as described in claim 1, characterized in that, Step S2 includes the following sub-steps: The visual modality data is feature extracted based on a residual network, and the visual features are obtained by global average pooling compression. The audio features are obtained by extracting features from the audio modal data using a convolutional neural network. The environmental modal data is mapped to a high-dimensional feature space to obtain the environmental features.

4. The intelligent monitoring method for the in-vehicle environment as described in claim 1, characterized in that, Step S3 includes the following sub-steps: S31. Using the cross-modal fusion features as input to the spatiotemporal attention network, position encoding is performed on the cross-modal fusion features to obtain position-encoded features; S32. The position encoding features are concatenated based on a multi-head self-attention mechanism to obtain multi-head concatenated features; S33. Perform nonlinear transformation on the multi-head splicing features based on a feedforward network, and calculate the time step weights of the multi-head splicing features using a time attention vector; S34. Calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

5. The intelligent monitoring method for the in-vehicle environment as described in claim 1, characterized in that, Step S3 includes the following sub-steps: S35. Normalize the enhanced visual features, audio features, and environmental features respectively, and calculate the probability distributions corresponding to the enhanced visual features, audio features, and environmental features respectively to obtain the distribution results; S36. Calculate the Shannon entropy based on the distribution results to obtain the Shannon entropy value; S37. Adjust and fuse the weights corresponding to the enhanced visual features, audio features and environmental features according to the Shannon entropy value to obtain the Shannon fused features.

6. An intelligent monitoring system for in-vehicle environment, characterized in that, include: The preprocessing module is used to acquire time-series data of the in-vehicle environment and preprocess the time-series data of the in-vehicle environment to obtain multimodal data; the multimodal data includes visual modal data, audio modal data and environmental modal data; The feature extraction module is used to extract features from the multimodal data to obtain visual features, audio features, and environmental features; The capture module is used to enhance the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism, and to interactively fuse the enhanced visual features, audio features, and environmental features to obtain cross-modal fusion features; The temporal dynamic features in the cross-modal fusion features are captured based on the spatiotemporal attention network to obtain spatiotemporal temporal features; And to dynamically allocate the weights corresponding to the enhanced visual features, the audio features and the environmental features based on Shannon entropy, respectively, to obtain Shannon fusion features; The fusion module is used to fuse the spatiotemporal time-series features and the Shannon fusion features to obtain the final fused features; The risk prediction module is used to predict the risk probability based on the final fusion features and obtain the prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, an early warning notification is issued.

7. The intelligent in-vehicle environment monitoring system as described in claim 6, characterized in that, The feature extraction module includes the following sub-units: The visual extraction unit is used to extract features from the visual modality data based on the residual network and to obtain the visual features through global average pooling compression processing. An audio extraction unit is used to extract features from the audio modality data based on a convolutional neural network to obtain the audio features; An environment extraction unit is used to map the environmental modal data to a high-dimensional feature space to obtain the environmental features.

8. The intelligent in-vehicle environment monitoring system as described in claim 6, characterized in that, The capture module includes the following sub-units: The encoding unit is used to take the cross-modal fusion features as input to the spatiotemporal attention network and perform position encoding on the cross-modal fusion features to obtain position-encoded features; The splicing unit is used to splice the positional encoding features based on a multi-head self-attention mechanism to obtain multi-head spliced ​​features; The transformation unit is used to perform nonlinear transformation on the multi-head splicing features based on the feedforward network, and to calculate the time step weights of the multi-head splicing features through the time attention vector; The calculation unit is used to calculate the spatiotemporal temporal features by combining the multi-head splicing features after nonlinear transformation with the time step weights.

9. A computer device, characterized in that, include: The system includes a memory, a processor, and an in-vehicle environment intelligent monitoring program stored in the memory and executable on the processor. When the processor executes the in-vehicle environment intelligent monitoring program, it implements the steps of the in-vehicle environment intelligent monitoring method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an intelligent in-vehicle environment monitoring program, which, when executed by a processor, implements the steps of the intelligent in-vehicle environment monitoring method as described in any one of claims 1-5.

Images