Intelligent monitoring method and system for environment in vehicle and storage medium

Abstract

The invention is suitable for the technical field of automobile intelligent control, and particularly relates to an in-vehicle environment intelligent monitoring method and system and a storage medium. The in-vehicle environment intelligent monitoring method comprises the following steps: acquiring in-vehicle environment time sequence data, and preprocessing the in-vehicle environment time sequence data to obtain multi-modal data; performing feature extraction on the multi-modal data; performing interactive fusion on the visual features, the audio features and the environment features based on a multi-head cross attention mechanism, and capturing time sequence dynamic features in cross-modal fusion features based on a space-time attention network; dynamically distributing the weight corresponding to each feature in the time sequence features based on the Shannon entropy; performing risk probability prediction according to the final fusion feature to obtain a prediction result; and when the prediction result is that the in-vehicle environment is abnormal, an early warning notification is sent out. Compared with the prior art, the method is higher in precision, real-time performance and robustness of in-vehicle environment risk identification.

Description

Technical Field

[0001] The present invention is applicable to the field of automobile intelligent control technology, and in particular relates to a method, system and storage medium for intelligent monitoring of an in-vehicle environment. Background Art

[0002] With the rapid development of intelligent transportation and automotive electronics, in-vehicle environmental safety is gaining increasing attention. The in-vehicle environment includes multi-dimensional information such as temperature, humidity, PM2.5, hazardous gases (such as CO and formaldehyde), smoke, and unusual noises. Accurate monitoring of this information is crucial for protecting the health of drivers and passengers and preventing accidents such as fires and poisoning.

[0003] However, the shortcomings of existing in-vehicle environment monitoring methods are as follows:

[0004] (1) Multimodal fusion lacks semantic interaction: Existing fusion methods do not establish semantic associations between cross-modal features and cannot effectively utilize the complementary information between visual, audio, and environmental sensor data, resulting in feature redundancy and loss of key information. For example, there is a strong correlation between smoke images in visual data and abnormal popping sounds in audio data and temperature rise signals from environmental sensors. However, existing methods have difficulty effectively capturing this cross-modal dependency, resulting in insufficient feature expression after fusion.

[0005] (2) Insufficient spatiotemporal feature extraction: Existing methods independently model the time series characteristics and spatial distribution characteristics of environmental data, failing to form a joint spatiotemporal representation and failing to capture the spatiotemporal coupling patterns of environmental changes. In-vehicle environmental data has significant spatiotemporal dynamic characteristics, such as the spatial distribution and temporal evolution of smoke diffusion, and the temporal trend of odor concentration. Existing methods often use fixed-structure neural networks (such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs)) to process spatial or temporal features separately, lacking the ability to jointly model spatiotemporal features and making it difficult to capture the dynamic patterns of environmental changes.

[0006] (3) Insufficient focus on key information: Existing methods do not dynamically evaluate the importance of features in each modality and dimension during feature processing, resulting in low model learning efficiency and weak anti-interference ability. In the in-vehicle environment, some features (such as the color and texture of smoke and the peak concentration of abnormal odors) play a decisive role in risk identification. However, existing methods treat all features equally and lack an adaptive focusing mechanism for key information, which makes the model prone to false positives or missed detections in complex noisy environments.

[0007] Therefore, there is an urgent need for a new in-vehicle environment intelligent monitoring method, system and storage medium to solve the above technical problems. Summary of the Invention

[0008] The present invention provides a method, system and storage medium for intelligent monitoring of the in-vehicle environment, the purpose of which is to improve the accuracy, real-time performance and robustness of in-vehicle environmental risk identification.

[0009] In a first aspect, the present invention provides a method for intelligently monitoring an in-vehicle environment, the method comprising the following steps:

[0010] S1. Acquire in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data;

[0011] S2. Extracting features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fusing the visual features, the audio features, and the environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features;

[0012] S3. Capturing the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features;

[0013] S4. Dynamically assigning a weight corresponding to each feature in the time series feature based on Shannon entropy to obtain a final fusion feature;

[0014] S5. Perform risk probability prediction based on the final fusion features to obtain a prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, issue a warning notification.

[0015] Preferably, step S1 includes the following sub-steps:

[0016] S11, aligning the time stamps of the in-vehicle environment time series data using a hardware synchronization clock;

[0017] S12. Normalizing the visual data in the in-vehicle environment time series data after time stamp alignment based on the target detection model to obtain the visual modality data;

[0018] Converting the audio data in the in-vehicle environment time series data after time stamp alignment into a Mel-spectrogram through short-time Fourier transform, and filtering the Mel-spectrogram to obtain the audio modal data;

[0019] The environmental data in the in-vehicle environmental time series data after time stamp alignment is subjected to standardization processing and sliding window processing to obtain the environmental modal data.

[0020] Preferably, step S2 includes the following sub-steps:

[0021] Extracting features from the visual modality data based on a residual network and performing global average pooling compression processing to obtain the visual features;

[0022] Performing feature extraction on the audio modal data based on a convolutional neural network to obtain the audio features;

[0023] The environmental modal data is mapped to a high-dimensional feature space to obtain the environmental features.

[0024] Preferably, step S3 includes the following sub-steps:

[0025] S31, using the cross-modal fusion feature as the input of the spatiotemporal attention network, performing position encoding on the cross-modal fusion feature to obtain a position encoding feature;

[0026] S32, splicing the position encoding features based on a multi-head self-attention mechanism to obtain a multi-head splicing feature;

[0027] S33, performing a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculating the time step weight of the multi-head splicing feature by scaling dot product attention;

[0028] S34. Calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

[0029] Preferably, step S4 includes the following sub-steps:

[0030] S41, normalizing the time series features, calculating the probability distribution corresponding to each feature in the time series features, and obtaining a distribution result;

[0031] S42. Calculate the Shannon entropy based on the distribution result to obtain a Shannon entropy value;

[0032] S43. Adjust the weight corresponding to each feature in the time series feature according to the Shannon entropy value to obtain the final fusion feature.

[0033] In a second aspect, the present invention further provides an intelligent in-vehicle environment monitoring system, comprising:

[0034] A preprocessing module, configured to obtain in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data;

[0035] A feature extraction module is used to extract features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fuse the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features;

[0036] A capture module, configured to capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features;

[0037] A weight allocation module is used to dynamically allocate the weight corresponding to each feature in the time series feature based on Shannon entropy to obtain the final fusion feature;

[0038] The risk prediction module is used to predict the risk probability based on the final fusion features to obtain a prediction result; when the prediction result is that the in-vehicle environment is abnormal, an early warning notification is issued.

[0039] Preferably, the feature extraction module includes the following subunits:

[0040] A visual extraction unit, configured to extract features from the visual modality data based on a residual network and obtain the visual features through global average pooling compression processing;

[0041] an audio extraction unit, configured to perform feature extraction on the audio modality data based on a convolutional neural network to obtain the audio features;

[0042] The environment extraction unit is used to map the environmental modal data into a high-dimensional feature space to obtain the environmental features.

[0043] Preferably, the capture module includes the following subunits:

[0044] an encoding unit, configured to use the cross-modal fusion feature as an input of the spatiotemporal attention network, perform position encoding on the cross-modal fusion feature, and obtain a position encoding feature;

[0045] A splicing unit, configured to splice the position encoding features based on a multi-head self-attention mechanism to obtain a multi-head splicing feature;

[0046] A transformation unit, configured to perform a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculate the time step weight of the multi-head splicing feature by scaling dot product attention;

[0047] A calculation unit is used to calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

[0048] In a third aspect, the present invention also provides a computer device comprising: a memory, a processor, and an in-vehicle environment intelligent monitoring program stored in the memory and runnable on the processor, wherein when the processor executes the in-vehicle environment intelligent monitoring program, the steps in the in-vehicle environment intelligent monitoring method as described in any one of the above embodiments are implemented.

[0049] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a program for intelligent monitoring of the in-vehicle environment is stored. When the program for intelligent monitoring of the in-vehicle environment is executed by a processor, the steps of the method for intelligent monitoring of the in-vehicle environment as described in any one of the above embodiments are implemented.

[0050] Compared with the existing technology, the present invention collects full-dimensional environmental data, audio data and visual data in the car, and realizes deep semantic interaction of different modal data through a bidirectional cross-modal attention mechanism, so that visual, voice and environmental features can guide and enhance each other, effectively solving the problem of insufficient cross-modal feature interaction in the existing technology and constructing a more complete in-car environment representation model.

[0051] Secondly, the present invention uses a spatiotemporal attention network to jointly model cross-modal fusion features in both the temporal and spatial dimensions. At the temporal level, a dynamic gated recurrent unit is used to capture the temporal patterns of environmental changes. At the spatial level, a multi-scale convolutional attention network is used to extract hierarchical features from complex scenes. This spatiotemporal feature fusion technology forms a refined spatiotemporal dynamic representation, breaking through the limitations of existing models that separate spatiotemporal modeling and significantly improving the ability to capture features of unusual events within the vehicle.

[0052] This invention dynamically assigns weights to each of the time series features based on Shannon entropy, significantly improving its adaptability to complex operating conditions and effectively addressing the weak adaptability of existing fixed-weight fusion methods. This invention effectively improves the accuracy, real-time performance, and robustness of in-vehicle environmental risk identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be described in detail below with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and easier to understand through the detailed description made with reference to the following drawings. In the accompanying drawings:

[0054] Figure 1 This is a flowchart of the method for intelligent monitoring of the in-vehicle environment provided by an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the structure of the intelligent monitoring system for vehicle interior environment provided by an embodiment of the present invention;

[0056] Figure 3It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Example 1

[0059] Please refer to Figure 1 The present invention provides a method for intelligently monitoring an in-vehicle environment, the method comprising the following steps:

[0060] S1. Acquire in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data;

[0061] In the embodiment of the present invention, a multimodal sensor array (environmental sensor, vehicle-mounted camera, microphone) is used to collect time series data of the vehicle environment, and multi-dimensional data such as environmental data (temperature, humidity, light), visual data (visual image), and audio data (voice signal) are collected in real time. For example, by deploying six types of environmental sensors (output time series data) such as temperature, humidity, PM2.5, etc. ), a vehicle-mounted camera with a resolution of 1920×1080 / 25fps frame rate (collecting visual images V) and a microphone with a sampling rate of 44.1kHz (collecting audio signals A).

[0062] Step S1 includes the following sub-steps:

[0063] S11. Align the timestamps of the in-vehicle environment timing data (with an error of ±5ms) using a hardware-synchronized clock. A hardware-synchronized clock is a system that synchronizes clock signals between different devices using physical layer signals or dedicated hardware circuits to ensure consistency in the time base of physical layer signals or distributed devices.

[0064] S12. Normalize the visual data in the time series data of the in-vehicle environment after timestamp alignment based on the target detection model to obtain the visual modal data; wherein the visual data is extracted by the pre-trained target detection model and the key area of ​​the cockpit is cropped and normalized to [0, 1]. The normalization formula is:

[0065]

[0066] S13. Convert the audio data in the time series data of the in-vehicle environment after the timestamp alignment into a mel-spectrogram through a short-time Fourier transform (STFT), and filter the mel-spectrogram to obtain the audio modal data. Specifically, the mel-spectrogram is filtered through multiple filters to output audio modal data with a dimension of T×100×40 (where T is the feature matrix of the time series length).

[0067] S14. Normalize and perform sliding window processing on the environmental data in the time series data of the in-vehicle environment after the timestamp alignment to obtain the environmental modal data. Specifically, after the environmental data is normalized by Z-score, the environmental modal data (T×6 time series) is generated using a 10-second sliding window.

[0068] S2. Extracting features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fusing the visual features, the audio features, and the environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features;

[0069] In this embodiment of the present invention, inter-modal information enhancement is achieved by introducing a bidirectional cross-attention mechanism, wherein step S2 includes the following sub-steps:

[0070] S21. Perform feature extraction on the visual modality data based on a residual network (ResNet50), and obtain the visual features through global average pooling compression processing.

[0071] Specifically, the visual modality data is subjected to feature extraction through the residual network, and a first feature matrix is ​​outputted. Compressed into visual features by global average pooling (D is the unified feature dimension).

[0072] S22. Perform feature extraction on the audio modality data based on a convolutional neural network (CNN) to obtain the audio features.

[0073] Specifically, the audio modal data is subjected to feature extraction by a convolutional neural network, and a second feature matrix is ​​output. Converted into audio features after pooling

[0074] S23, mapping the environmental modal data to a high-dimensional feature space to obtain the environmental features. Specifically, the environmental modal data S′ is mapped to the environmental features through a fully connected layer. Achieve inter-modality dimension alignment.

[0075] Then, a multi-head cross-attention mechanism is constructed to achieve bidirectional information interaction between visual features, audio features, and environmental features. The cross-attention between modalities is calculated as follows: for any two modalities m (query modality) and n (key / value modality).

[0076] Cross attention output Z m←n For: Q m =X m W q ,K n =X n W k ,V n =X n W v .

[0077] Z m←n =A m,n V n ;

[0078] in is the projection matrix, d k is the key dimension. Subsequently, in order to interactively fuse the three features, cross-attention operations are performed on the visual features, audio features, and environmental features to obtain enhanced features.

[0079] Visual enhancement: Visual features X′ that fuse audio and environmental information v =Z v←a +Z v←e +X v ;

[0080] Audio enhancement: Audio features X′ that integrate visual and environmental information a =Z a←v +Z a←e +X a ;

[0081] Environmental enhancement: Environmental features X′ that integrate visual and audio information e =Z e←v +Z e←a +X e ,

[0082] And the original modal features are retained through residual connections to avoid information loss.

[0083] Then the three enhanced features of the cross attention output are spliced ​​in the channel dimension to obtain the cross-modal fusion feature and compressed to Achieving semantic alignment between modalities

[0084] F fusion =FC([X′ v ; X′a ; X′ e ]);

[0085] Where [·;·] represents feature concatenation, and FC represents a fully connected layer. The cross-attention mechanism enables each modality to explicitly focus on key areas of the other modality (e.g., visual smoke regions corresponding to the location of abnormal audio sound sources), dynamically capturing cross-modal semantic associations. For example, when the visual branch detects smoke, the cross-attention mechanism increases the audio branch's attention weight to the concurrent abnormal combustion noise, achieving a synergistic enhancement of the "visual-audio" anomaly features.

[0086] S3. Capturing the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features;

[0087] In this embodiment of the present invention, the cross-modal fusion feature output in step S2 Based on this, a spatiotemporal attention network is constructed to capture the temporal dynamic characteristics of environmental changes and to model gradual risks such as smoke diffusion and sudden temperature rise.

[0088] First, the cross-modal fusion features are input into the multi-layer Transformer encoder of the spatiotemporal attention network. Each layer of the encoder contains multi-head self-attention and a feedforward neural network (FFN). The specific process is as follows:

[0089] S31, taking the cross-modal fusion feature as the input of the spatiotemporal attention network, performing position encoding on the cross-modal fusion feature to obtain position encoding feature. fnsion Adding learnable positional encodings

[0090] S32, based on the multi-head self-attention mechanism, the position encoding features are spliced ​​to obtain multi-head splicing features. pos =F fusion +P, through the multi-head self-attention mechanism, F pos Mapped to h groups of query-key-value pairs, the attention of each group is calculated as:

[0091]

[0092] in The final output is the head splicing feature of the multi-head splicing result: MultiHead=Concat(Head1,…,Head h )W O .

[0093] S33. Perform a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculate the time step weight of the multi-head splicing feature by scaling dot product attention.

[0094] Specifically, the multi-head concatenated features of the attention output are subjected to a nonlinear transformation through two fully connected layers of the feedforward network, satisfying: FFN(X) = σ(XW1+b1)W2+b2, and time-dependent modeling.

[0095] Generate time-step weight α by scaling dot-product attention t , focusing on key timing fragments (such as the time when smoke first appears):

[0096]

[0097] where u t is a learnable temporal attention vector.

[0098] S34. Calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

[0099] Specifically, the timing characteristics meet the following requirements:

[0100] T t =α t ⊙F pos (t),T final =LayerNorm(T t +MultiHead(E pos ));

[0101] Through this mechanism, the present invention can capture the causal relationship between temperature and smoke concentration in the time dimension. For example, when smoke appears after the temperature continues to rise, the feature weight of the corresponding time step is automatically increased.

[0102] S4. Dynamically assigning a weight corresponding to each feature in the time series feature based on Shannon entropy to obtain a final fusion feature;

[0103] In the embodiment of the present invention, Shannon entropy is introduced to quantify the reliability of each modality and realize dynamic weight distribution. Modal uncertainty measurement is used to measure the temporal characteristics X′ of the three modalities of vision, audio and environment. v ,X′ a ,X′ e The steps to calculate the entropy value are as follows:

[0104] S41, normalize the time series features, calculate the probability distribution corresponding to each feature in the time series features, and obtain the distribution result. Specifically, normalize each modal feature along the channel dimension to a probability distribution Among them, m∈{v,a,e}, namely visual features v, audio features a and environmental features e.

[0105] S42. Perform Shannon entropy calculation based on the distribution result to obtain a Shannon entropy value. Specifically, the Shannon entropy calculation satisfies the following conditions:

[0106]

[0107] Low entropy values ​​indicate concentrated feature distribution, which corresponds to key information such as smoke visual features and high-frequency components of abnormal audio.

[0108] S43. Adjust the weight corresponding to each feature in the time series feature according to the Shannon entropy value to obtain the final fusion feature.

[0109] Specifically, by using exponential normalization to generate modal fusion weights, the model is forced to focus on low-entropy modalities:

[0110]

[0111] Among them, λ is the temperature hyperparameter, which adjusts the influence of entropy on weight.

[0112] The final fusion features meet the following conditions:

[0113] F final =w v X′ v +w a X′ a +w e X′ e ;

[0114] For example, when the visual modality detects smoke (E v When the v Rapidly rising to over 70%, ensuring the smoke signature dominates the fusion.

[0115] S5. Perform risk probability prediction based on the final fusion features to obtain a prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, issue a warning notification.

[0116] In the embodiment of the present invention, the final fusion feature F is final Implement secondary risk classification (normal / abnormal); the classifier architecture uses a two-layer fully connected network + Softmax: Its output probability vector Corresponding to the "normal" and "abnormal" categories. When an abnormal state occurs, the red indicator light will be triggered to alarm and notify the people in the car.

[0117] Compared with the existing technology, the present invention collects full-dimensional environmental data, audio data and visual data in the car, and realizes deep semantic interaction of different modal data through a bidirectional cross-modal attention mechanism, so that visual, voice and environmental features can guide and enhance each other, effectively solving the problem of insufficient cross-modal feature interaction in the existing technology and constructing a more complete in-car environment representation model.

[0118] Secondly, the present invention uses a spatiotemporal attention network to jointly model cross-modal fusion features in both the temporal and spatial dimensions. At the temporal level, a dynamic gated recurrent unit is used to capture the temporal patterns of environmental changes. At the spatial level, a multi-scale convolutional attention network is used to extract hierarchical features from complex scenes. This spatiotemporal feature fusion technology forms a refined spatiotemporal dynamic representation, breaking through the limitations of existing models that separate spatiotemporal modeling and significantly improving the ability to capture features of unusual events within the vehicle.

[0119] This invention dynamically assigns weights to each of the time series features based on Shannon entropy, significantly improving its adaptability to complex operating conditions and effectively addressing the weak adaptability of existing fixed-weight fusion methods. This invention effectively improves the accuracy, real-time performance, and robustness of in-vehicle environmental risk identification.

[0120] Example 2

[0121] The present invention also provides an intelligent monitoring system for the vehicle interior environment. Figure 2 , FIG is a schematic diagram of the structure of an intelligent in-vehicle environment monitoring system 200 provided by an embodiment of the present invention, which includes:

[0122] 201. A preprocessing module, configured to obtain in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data.

[0123] The pre-processing module 201 includes the following sub-units:

[0124] 2011. An alignment unit, configured to align the time stamps of the in-vehicle environment time series data through a hardware synchronization clock;

[0125] 2012. A visual modality unit, configured to perform normalization processing on the visual data in the in-vehicle environment time series data after the timestamps are aligned based on a target detection model to obtain the visual modality data;

[0126] 2013. An audio modality unit, configured to convert the audio data in the in-vehicle environment time series data after time stamp alignment into a mel-spectrogram through short-time Fourier transform, and perform filtering on the mel-spectrogram to obtain the audio modality data;

[0127] 2014. An environmental modality unit, configured to perform standardization processing and sliding window processing on the environmental data in the in-vehicle environmental time series data after the timestamps are aligned, to obtain the environmental modality data.

[0128] 202. A feature extraction module is used to extract features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fuse the visual features, the audio features, and the environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features.

[0129] The feature extraction module 202 includes the following subunits:

[0130] 2021. A visual extraction unit, configured to extract features from the visual modality data based on a residual network, and obtain the visual features through global average pooling compression processing;

[0131] 2022. An audio extraction unit, configured to perform feature extraction on the audio modality data based on a convolutional neural network to obtain the audio features;

[0132] 2023. An environment extraction unit, configured to map the environment modal data to a high-dimensional feature space to obtain the environment features.

[0133] 203. A capture module is used to capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features.

[0134] The capture module 203 includes the following subunits:

[0135] 2031. An encoding unit, configured to use the cross-modal fusion feature as an input of the spatiotemporal attention network, perform position encoding on the cross-modal fusion feature, and obtain a position encoding feature;

[0136] 2032. A splicing unit, configured to splice the position encoding features based on a multi-head self-attention mechanism to obtain a multi-head splicing feature;

[0137] 2033. A transformation unit, configured to perform a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculate a time step weight of the multi-head splicing feature by scaling dot product attention;

[0138] 2034. A calculation unit, configured to calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

[0139] 204. A weight allocation module is configured to dynamically allocate a weight corresponding to each feature in the time series feature based on Shannon entropy to obtain a final fusion feature.

[0140] The weight allocation module 204 includes the following subunits:

[0141] 2041. A probability distribution unit, configured to normalize the time series features, calculate a probability distribution corresponding to each feature in the time series features, and obtain a distribution result;

[0142] 2042. A Shannon calculation unit, configured to perform Shannon entropy calculation based on the distribution result to obtain a Shannon entropy value;

[0143] 2043. A Shannon fusion unit, configured to adjust the weight corresponding to each feature in the time series feature according to the Shannon entropy value to obtain the final fusion feature.

[0144] 205. A risk prediction module is configured to perform risk probability prediction based on the final fusion features to obtain a prediction result; and issue a warning notification when the prediction result indicates that the in-vehicle environment is abnormal.

[0145] The in-vehicle environment intelligent monitoring system 200 can implement the steps in the in-vehicle environment intelligent monitoring method in the above embodiment and can achieve the same technical effects. Please refer to the description in the above embodiment and will not be repeated here.

[0146] Example 3

[0147] The embodiment of the present invention also provides a computer device, please refer to Figure 3 , Figure 3 300 is a structural diagram of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and an intelligent in-vehicle environment monitoring program stored in the memory 302 and executable on the processor 301.

[0148] The processor 301 calls the in-vehicle environment intelligent monitoring program stored in the memory 302 to execute the steps of the in-vehicle environment intelligent monitoring method provided by the embodiment of the present invention. Figure 1 , specifically including the following steps:

[0149] S1. Acquire in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data;

[0150] In the embodiment of the present invention, a multimodal sensor array (environmental sensor, vehicle-mounted camera, microphone) is used to collect time series data of the vehicle environment, and multi-dimensional data such as environmental data (temperature, humidity, light), visual data (visual image), and audio data (voice signal) are collected in real time. For example, by deploying six types of environmental sensors (output time series data) such as temperature, humidity, PM2.5, etc. ), a vehicle-mounted camera with a resolution of 1920×1080 / 25fps frame rate (collecting visual images V) and a microphone with a sampling rate of 44.1kHz (collecting audio signals A).

[0151] Step S1 includes the following sub-steps:

[0152] S11. Align the timestamps of the in-vehicle environment timing data (with an error of ±5ms) using a hardware-synchronized clock. A hardware-synchronized clock is a system that synchronizes clock signals between different devices using physical layer signals or dedicated hardware circuits to ensure consistency in the time base of physical layer signals or distributed devices.

[0153] S12. Normalize the visual data in the time series data of the in-vehicle environment after timestamp alignment based on the target detection model to obtain the visual modal data; wherein the visual data is extracted by the pre-trained target detection model and the key area of ​​the cockpit is cropped and normalized to [0, 1]. The normalization formula is:

[0154]

[0155] S13. Convert the audio data in the time series data of the in-vehicle environment after the timestamp alignment into a mel-spectrogram through a short-time Fourier transform (STFT), and filter the mel-spectrogram to obtain the audio modal data. Specifically, the mel-spectrogram is filtered through multiple filters to output audio modal data with a dimension of T×100×40 (where T is the feature matrix of the time series length).

[0156] S14. Normalize and perform sliding window processing on the environmental data in the time series data of the in-vehicle environment after the timestamp alignment to obtain the environmental modal data. Specifically, after the environmental data is normalized by Z-score, the environmental modal data (T×6 time series) is generated using a 10-second sliding window.

[0157] S2. Extracting features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fusing the visual features, the audio features, and the environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features;

[0158] In this embodiment of the present invention, inter-modal information enhancement is achieved by introducing a bidirectional cross-attention mechanism, wherein step S2 includes the following sub-steps:

[0159] S21. Perform feature extraction on the visual modality data based on a residual network (ResNet50), and obtain the visual features through global average pooling compression processing.

[0160] Specifically, the visual modality data is subjected to feature extraction through the residual network, and a first feature matrix is ​​outputted. Compressed into visual features by global average pooling (D is the unified feature dimension).

[0161] S22. Perform feature extraction on the audio modality data based on a convolutional neural network (CNN) to obtain the audio features.

[0162] Specifically, the audio modal data is subjected to feature extraction by a convolutional neural network, and a second feature matrix is ​​output. Converted into audio features after pooling

[0163] S23, mapping the environmental modal data to a high-dimensional feature space to obtain the environmental features. Specifically, the environmental modal data S′ is mapped to the environmental features through a fully connected layer. Achieve inter-modality dimension alignment.

[0164] Then, a multi-head cross-attention mechanism is constructed to achieve bidirectional information interaction between visual features, audio features, and environmental features. The cross-attention between modalities is calculated as follows: for any two modalities m (query modality) and n (key / value modality).

[0165] Cross attention output Z m←n For: Q m =X m W q ,K n =X n W k ,V n =X n W v .

[0166] Z m←n =A m,n V n ;

[0167] in is the projection matrix, d kis the key dimension. Subsequently, in order to interactively fuse the three features, cross-attention operations are performed on the visual features, audio features, and environmental features to obtain enhanced features.

[0168] Visual enhancement: Visual features X′ that fuse audio and environmental information v =Z v←a +Z v←e +X v ;

[0169] Audio enhancement: Audio features X′ that integrate visual and environmental information a =Z a←v +Z a←e +X a ;

[0170] Environmental enhancement: Environmental features that integrate visual and audio information ′ e =Z e←v +Z e←a +X e ,

[0171] And the original modal features are retained through residual connections to avoid information loss.

[0172] Then the three enhanced features of the cross attention output are spliced ​​in the channel dimension to obtain the cross-modal fusion feature and compressed to Achieving semantic alignment between modalities

[0173] F fusion =FC([X′ v ; X′ a ; X′ e ]);

[0174] Where [·;·] represents feature concatenation, and FC represents a fully connected layer. The cross-attention mechanism enables each modality to explicitly focus on key areas of the other modality (e.g., visual smoke regions corresponding to the location of abnormal audio sound sources), dynamically capturing cross-modal semantic associations. For example, when the visual branch detects smoke, the cross-attention mechanism increases the audio branch's attention weight to the concurrent abnormal combustion noise, achieving a synergistic enhancement of the "visual-audio" anomaly features.

[0175] S3. Capturing the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features;

[0176] In this embodiment of the present invention, the cross-modal fusion feature output in step S2 Based on this, a spatiotemporal attention network is constructed to capture the temporal dynamic characteristics of environmental changes and to model gradual risks such as smoke diffusion and sudden temperature rise.

[0177] First, the cross-modal fusion features are input into the multi-layer Transformer encoder of the spatiotemporal attention network. Each layer of the encoder contains multi-head self-attention and a feedforward neural network (FFN). The specific process is as follows:

[0178] S31, taking the cross-modal fusion feature as the input of the spatiotemporal attention network, performing position encoding on the cross-modal fusion feature to obtain position encoding feature. fusion Adding learnable positional encodings

[0179] S32, based on the multi-head self-attention mechanism, the position encoding features are spliced ​​to obtain multi-head splicing features. pos =F fusion +P, through the multi-head self-attention mechanism, F pos Mapped to h groups of query-key-value pairs, the attention of each group is calculated as:

[0180]

[0181] in The final output is the head splicing feature of the multi-head splicing result: MultiHead=Concat(Head1,…,Head h )W O .

[0182] S33. Perform a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculate the time step weight of the multi-head splicing feature by scaling dot product attention.

[0183] Specifically, the multi-head concatenated features of the attention output are subjected to a nonlinear transformation through two fully connected layers of the feedforward network, satisfying: FFN(X) = σ(XW1+b1)W2+b2, and time-dependent modeling.

[0184] Generate time-step weight α by scaling dot-product attention t , focusing on key timing fragments (such as the time when smoke first appears):

[0185]

[0186] where u t is a learnable temporal attention vector.

[0187] S34. Calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

[0188] Specifically, the timing characteristics meet the following requirements:

[0189] T t =α t ⊙F pos (t),T final =LayerNorm(T t +MultiHead(F pos ));

[0190] Through this mechanism, the present invention can capture the causal relationship between temperature and smoke concentration in the time dimension. For example, when smoke appears after the temperature continues to rise, the feature weight of the corresponding time step is automatically increased.

[0191] S4. Dynamically assigning a weight corresponding to each feature in the time series feature based on Shannon entropy to obtain a final fusion feature;

[0192] In the embodiment of the present invention, Shannon entropy is introduced to quantify the reliability of each modality and realize dynamic weight distribution. Modal uncertainty measurement is used to measure the temporal characteristics X′ of the three modalities of vision, audio and environment. v ,X′ a ,X′ e The steps to calculate the entropy value are as follows:

[0193] S41, normalize the time series features, calculate the probability distribution corresponding to each feature in the time series features, and obtain the distribution result. Specifically, normalize each modal feature along the channel dimension to a probability distribution Among them, m∈{v,a,e}, namely visual features v, audio features a and environmental features e.

[0194] S42. Perform Shannon entropy calculation based on the distribution result to obtain a Shannon entropy value. Specifically, the Shannon entropy calculation satisfies the following conditions:

[0195]

[0196] Low entropy values ​​indicate concentrated feature distribution, which corresponds to key information such as smoke visual features and high-frequency components of abnormal audio.

[0197] S43. Adjust the weight corresponding to each feature in the time series feature according to the Shannon entropy value to obtain the final fusion feature.

[0198] Specifically, by using exponential normalization to generate modal fusion weights, the model is forced to focus on low-entropy modalities:

[0199]

[0200] Among them, λ is the temperature hyperparameter, which adjusts the influence of entropy on weight.

[0201] The final fusion features meet the following conditions:

[0202] F final =w v x′ v +w a X′ a +w e X′ e ;

[0203] For example, when the visual modality detects smoke (E v When the v Rapidly rising to over 70%, ensuring the smoke signature dominates the fusion.

[0204] S5. Perform risk probability prediction based on the final fusion features to obtain a prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, issue a warning notification.

[0205] In the embodiment of the present invention, the final fusion feature F is final Implement secondary risk classification (normal / abnormal); the classifier architecture uses a two-layer fully connected network + Softmax: Its output probability vector Corresponding to the "normal" and "abnormal" categories. When an abnormal state occurs, the red indicator light will be triggered to alarm and notify the people in the car.

[0206] The computer device 300 provided in the embodiment of the present invention can implement the steps in the intelligent monitoring method of the vehicle interior environment in the above embodiment and can achieve the same technical effects. Please refer to the description in the above embodiment and will not be repeated here.

[0207] Example 4

[0208] An embodiment of the present invention also provides a computer-readable storage medium, which stores an in-vehicle environment intelligent monitoring program. When the in-vehicle environment intelligent monitoring program is executed by a processor, it implements the various processes and steps in the in-vehicle environment intelligent monitoring method provided by an embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0209] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0210] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0211] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0212] The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only a preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms and equivalent changes without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

Claims

1. A method for intelligent monitoring of vehicle interior environment, characterized in that: The method for intelligently monitoring the in-vehicle environment comprises the following steps: S1. Acquire in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data; S2. Extracting features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fusing the visual features, the audio features, and the environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features; S3. Capturing the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features; S4. Dynamically assigning a weight corresponding to each feature in the time series feature based on Shannon entropy to obtain a final fusion feature; S5. Perform risk probability prediction based on the final fusion features to obtain a prediction result; when the prediction result indicates that the in-vehicle environment is abnormal, issue a warning notification.

2. The method for intelligently monitoring the in-vehicle environment according to claim 1, wherein: Step S1 includes the following sub-steps: S11, aligning the time stamps of the in-vehicle environment time series data using a hardware synchronization clock; S12. Normalizing the visual data in the in-vehicle environment time series data after time stamp alignment based on the target detection model to obtain the visual modality data; Converting the audio data in the in-vehicle environment time series data after time stamp alignment into a Mel-spectrogram through short-time Fourier transform, and filtering the Mel-spectrogram to obtain the audio modal data; The environmental data in the in-vehicle environmental time series data after time stamp alignment is subjected to standardization processing and sliding window processing to obtain the environmental modal data.

3. The method for intelligently monitoring the in-vehicle environment according to claim 1, wherein: Step S2 includes the following sub-steps: Extracting features from the visual modality data based on a residual network and performing global average pooling compression processing to obtain the visual features; Performing feature extraction on the audio modal data based on a convolutional neural network to obtain the audio features; The environmental modal data is mapped to a high-dimensional feature space to obtain the environmental features.

4. The method for intelligently monitoring the in-vehicle environment according to claim 1, wherein: Step S3 includes the following sub-steps: S31, using the cross-modal fusion feature as the input of the spatiotemporal attention network, performing position encoding on the cross-modal fusion feature to obtain a position encoding feature; S32, splicing the position encoding features based on a multi-head self-attention mechanism to obtain a multi-head splicing feature; S 33. Performing a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculating the time step weight of the multi-head splicing feature by scaling dot product attention; S34. Calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

5. The method for intelligently monitoring the in-vehicle environment according to claim 1, wherein: Step S4 includes the following sub-steps: S41, normalizing the time series features, calculating the probability distribution corresponding to each feature in the time series features, and obtaining a distribution result; S42. Calculate the Shannon entropy based on the distribution result to obtain a Shannon entropy value; S43. Adjust the weight corresponding to each feature in the time series feature according to the Shannon entropy value to obtain the final fusion feature.

6. An intelligent monitoring system for vehicle interior environment, characterized in that: include: A preprocessing module, configured to obtain in-vehicle environment time series data and preprocess the in-vehicle environment time series data to obtain multimodal data; the multimodal data includes visual modal data, audio modal data, and environmental modal data; A feature extraction module is used to extract features from the multimodal data to obtain visual features, audio features, and environmental features; and interactively fuse the visual features, audio features, and environmental features based on a multi-head cross-attention mechanism to obtain cross-modal fusion features; A capture module, configured to capture the temporal dynamic features in the cross-modal fusion features based on a spatiotemporal attention network to obtain temporal features; A weight allocation module is used to dynamically allocate the weight corresponding to each feature in the time series feature based on Shannon entropy to obtain the final fusion feature; The risk prediction module is used to predict the risk probability based on the final fusion features to obtain a prediction result; when the prediction result is that the in-vehicle environment is abnormal, an early warning notification is issued.

7. The intelligent in-vehicle environment monitoring system according to claim 6, characterized in that: The feature extraction module includes the following subunits: A visual extraction unit, configured to extract features from the visual modality data based on a residual network and obtain the visual features through global average pooling compression processing; an audio extraction unit, configured to perform feature extraction on the audio modality data based on a convolutional neural network to obtain the audio features; The environment extraction unit is used to map the environmental modal data into a high-dimensional feature space to obtain the environmental features.

8. The intelligent in-vehicle environment monitoring system according to claim 6, characterized in that: The capture module includes the following subunits: an encoding unit, configured to use the cross-modal fusion feature as an input of the spatiotemporal attention network, perform position encoding on the cross-modal fusion feature, and obtain a position encoding feature; A splicing unit, configured to splice the position encoding features based on a multi-head self-attention mechanism to obtain a multi-head splicing feature; A transformation unit, configured to perform a nonlinear transformation on the multi-head splicing feature based on a feedforward network, and calculate the time step weight of the multi-head splicing feature by scaling dot product attention; A calculation unit is used to calculate the multi-head splicing feature after nonlinear transformation and the time step weight to obtain the time series feature.

9. A computer device, characterized in that: include: A memory, a processor, and an in-vehicle environment intelligent monitoring program stored in the memory and runnable on the processor, wherein when the processor executes the in-vehicle environment intelligent monitoring program, the steps in the in-vehicle environment intelligent monitoring method as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an in-vehicle environment intelligent monitoring program, and when the in-vehicle environment intelligent monitoring program is executed by the processor, the steps of the in-vehicle environment intelligent monitoring method according to any one of claims 1 to 5 are implemented.

Images