Intelligent fire alarm signal identification method and system based on multi-source fusion perception

Abstract

The application relates to the technical field of signal identification, and provides a smart fire-fighting fire alarm signal identification method and system based on multi-source fusion perception. According to the technical scheme of the application, real-time data in multiple modes are acquired through intelligent sensors, feature extraction is performed on the real-time data to generate a feature matrix in multiple modes, dynamic fusion is performed on the feature matrix and environmental data in the real-time data to form a fusion vector, a thermal imaging space-time sequence is generated based on image information in the environmental data, a space-time tensor feature is generated according to the thermal imaging space-time sequence and the fusion vector, the space-time tensor feature is input into a pre-constructed encoder network, an attention graph representing fire space-time distribution and an error parameter are output, and a fire alarm probability and an early warning level are determined according to the attention graph and the error parameter. The whole-process intelligentization from multi-modal data acquisition to risk quantization early warning is realized, and the accuracy and environmental adaptability of fire alarm identification are significantly improved.

Description

Technical Field

[0001] This application relates to the field of signal recognition technology, and more specifically, to a smart fire alarm signal recognition method and system based on multi-source fusion sensing. Background Technology

[0002] Traditional fire protection systems primarily rely on single sensors, such as smoke detectors and temperature probes, for fire alarm monitoring, resulting in high false alarm rates and delayed response times. For example, kitchen smoke or steam may trigger false alarms, while early smoldering stages are prone to missed alarms because the temperature has not yet reached the threshold. With accelerating urbanization, complex scenarios such as high-rise buildings and underground spaces place higher demands on the accuracy of fire early warning systems, making traditional methods insufficient for modern fire safety needs. The integration of technologies such as the Internet of Things (IoT), artificial intelligence (AI), internet platforms, and big data services provides technical support for smart fire protection. By deploying multimodal sensors, including video, infrared, and gas detectors, and building edge computing nodes, heterogeneous spatiotemporal data can be collected in real time. However, when the types and volumes of collected multi-source data are enormous, extracting key features from this data for fire alarm signal identification becomes a problem that needs to be solved. Summary of the Invention

[0003] This application provides a smart fire alarm signal recognition method and system based on multi-source fusion sensing, which can at least partially solve the problem of how to extract key features from multimodal data for fire alarm signal recognition.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of this application, a smart fire alarm signal recognition method based on multi-source fusion perception is provided, comprising: acquiring multimodal real-time data through intelligent sensors; extracting features from the real-time data to generate a multimodal feature matrix; dynamically fusing the feature matrix and environmental data in the real-time data to form a fusion vector; generating a thermal imaging spatiotemporal sequence based on image information in the environmental data; generating a spatiotemporal tensor feature based on the thermal imaging spatiotemporal sequence and the fusion vector; inputting the spatiotemporal tensor feature into a pre-constructed encoder network to output an attention map representing the spatiotemporal distribution of the fire and error parameters; and determining the fire alarm probability and warning level based on the attention map and the error parameters.

[0006] In this application, based on the aforementioned scheme, the step of acquiring multimodal real-time data through a smart sensor and extracting features from the real-time data to generate a multimodal feature matrix includes: acquiring multimodal real-time data through a smart sensor, wherein the real-time data includes environmental data, acoustic data, and imaging data; performing linear processing on the environmental data in the real-time data to output a first feature; extracting energy features from the acoustic data in the real-time data as a second feature; performing segmentation processing on the imaging data in the real-time data to generate a third feature; and combining and standardizing the first feature, the second feature, and the third feature to generate a multimodal feature matrix.

[0007] In this application, based on the aforementioned scheme, the step of linearly processing the environmental data in the real-time data to output a first feature includes: generating a linear factor based on the environmental data in the real-time data, and performing linear processing on the environmental data using the linear factor to generate the first feature. for:

[0008]

[0009] in, This represents a linear factor dynamically generated based on the distribution of environmental data, where x represents the environmental data.

[0010] In this application, based on the aforementioned scheme, the step of extracting energy features from the acoustic wave data in the real-time data as a second feature includes: decomposing the acoustic wave data in the real-time data to generate a preset number of frequency bands, and extracting the energy parameters of the frequency bands as the second feature. for:

[0011]

[0012] Where i and k represent the frequency band identifiers of the acoustic data, respectively. represent the wavelet coefficients of the i-th and k-th frequency bands, respectively.

[0013] In this application, based on the aforementioned scheme, the step of dynamically fusing the feature matrix and the environmental data in the real-time data to form a fusion vector includes: determining environmental factors through the environmental data in the real-time data; determining the membership parameters corresponding to each feature dimension in the feature matrix based on the environmental factors; calculating the feature parameters corresponding to the feature dimensions based on the membership parameters; and dynamically fusing the feature matrix based on the feature parameters to generate a fusion vector.

[0014] In this application, based on the aforementioned scheme, the step of generating a thermal imaging spatiotemporal sequence based on image information in the environmental data, and generating spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector, includes: generating a thermal imaging spatiotemporal sequence based on image information in the environmental data; and extracting target features from the thermal imaging spatiotemporal sequence and the fusion vector using a preset convolution kernel to generate spatiotemporal tensor features.

[0015] In this application, based on the aforementioned scheme, the step of inputting the spatiotemporal tensor features into a pre-constructed encoder network and outputting an attention map and error parameters representing the spatiotemporal distribution of the fire includes: constructing an encoder network by stacking multiple layers of convolutional kernels; inputting the spatiotemporal tensor features into the encoder network and outputting an attention map and error parameters representing the spatiotemporal distribution of the fire.

[0016] In this application, based on the aforementioned scheme, the step of determining the fire alarm probability and warning level according to the attention map and the error parameter includes: generating a dynamic threshold based on the error parameters corresponding to a preset number of attention maps; generating a fire alarm probability based on the quotient between the error parameter and the dynamic threshold; and generating a warning level based on a preset number of levels and the initialized cluster centers.

[0017] According to one aspect of this application, a smart fire alarm signal recognition system based on multi-source fusion sensing is provided, comprising:

[0018] The acquisition unit is used to acquire real-time multimodal data through intelligent sensors and perform feature extraction on the real-time data to generate a multimodal feature matrix.

[0019] The fusion unit is used to dynamically fuse the feature matrix and the environmental data in the real-time data to form a fusion vector.

[0020] The feature unit is used to generate a thermal imaging spatiotemporal sequence based on image information in the environmental data, and to generate spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector.

[0021] The encoding unit is used to input the spatiotemporal tensor features into a pre-constructed encoder network and output an attention map and error parameters representing the spatiotemporal distribution of the fire.

[0022] The early warning unit is used to determine the fire alarm probability and early warning level based on the attention map and the error parameters.

[0023] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the intelligent fire alarm signal recognition method based on multi-source fusion sensing as described in the above embodiments.

[0024] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the intelligent fire alarm signal recognition method based on multi-source fusion sensing as described in the above embodiments.

[0025] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the intelligent fire alarm signal recognition method based on multi-source fusion sensing provided in the various optional implementations described above.

[0026] This application's technical solution acquires multimodal real-time data through intelligent sensors, extracts features from the real-time data to generate a multimodal feature matrix, dynamically fuses the feature matrix and environmental data from the real-time data to form a fusion vector, generates a thermal imaging spatiotemporal sequence based on image information from the environmental data, and generates spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector, inputs the spatiotemporal tensor features into a pre-constructed encoder network, and outputs an attention map and error parameters representing the spatiotemporal distribution of the fire, and determines the fire alarm probability and warning level based on the attention map and the error parameters. By collaboratively sensing environmental, acoustic, and imaging data through multi-source sensors, a feature matrix covering multi-dimensional features is formed. Combined with dynamic fusion of environmental data, a fusion vector adapting to scene changes is generated. Furthermore, the joint features of the thermal imaging spatiotemporal sequence and the fusion vector are extracted to construct a spatiotemporal tensor feature. The encoder network accurately locates the spatiotemporal region of a fire and quantifies the reconstruction error through layer-by-layer feature abstraction and attention mechanism. Finally, it outputs the fire alarm probability and graded warning through dynamic threshold and fuzzy clustering, realizing the intelligent process from multimodal data collection to risk quantification and warning, and significantly improving the accuracy and environmental adaptability of fire alarm identification.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0029] Figure 1 The flowchart illustrating a method for identifying intelligent fire alarm signals based on multi-source fusion sensing in one embodiment of this application is shown.

[0030] Figure 2 The flowchart illustrating the generation of a multimodal feature matrix is ​​shown in one embodiment of this application.

[0031] Figure 3 The illustration shows a schematic diagram of a smart fire alarm signal recognition system based on multi-source fusion sensing in one embodiment of this application.

[0032] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0034] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] The implementation details of the technical solution of this application are described below:

[0038] Figure 1A flowchart illustrating a smart fire alarm signal recognition method based on multi-source fusion sensing according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, the intelligent fire alarm signal recognition method based on multi-source fusion sensing includes at least steps S110 to S150, which are described in detail below:

[0039] S110: Acquire real-time multimodal data through intelligent sensors, and extract features from the real-time data to generate a multimodal feature matrix.

[0040] In this embodiment, multi-source heterogeneous data, including temperature, smoke concentration, infrared thermal imaging video stream, and environmental parameters such as humidity and wind speed, are collected in real time through an intelligent sensor network. The acquired data is preprocessed using edge computing nodes to remove noise interference and unify spatiotemporal resolution. Then, spatiotemporal features (such as temperature gradient change rate and smoke diffusion speed), statistical features (such as mean and variance), and frequency domain features (such as the spectral energy of the thermal imaging sequence) of each modality are extracted. Optionally, a dynamic weighted fusion algorithm is used to map the multi-dimensional features to a unified feature space, forming a structured multimodal feature matrix, providing high-dimensional, time-aligned input data for subsequent fire alarm identification.

[0041] like Figure 2 As shown, in one embodiment of this application, multimodal real-time data is acquired through a smart sensor, and feature extraction is performed on the real-time data to generate a multimodal feature matrix, including:

[0042] S210, acquires multimodal real-time data through intelligent sensors, including environmental data, acoustic data, and imaging data;

[0043] S220, perform linear processing on the environmental data in the real-time data, and output the first feature;

[0044] S230, extract energy features from the acoustic wave data in the real-time data as a second feature;

[0045] S240, The imaging data in the real-time data is segmented to generate a third feature;

[0046] S250, the first feature, the second feature and the third feature are combined and standardized to generate a multimodal feature matrix.

[0047] In one embodiment of this application, multimodal real-time data is acquired through smart sensors, including collecting environmental data through an environmental sensor array, such as deploying a smoke concentration sensor, a carbon dioxide sensor, an oxygen sensor, and a temperature sensor, and synchronously collecting data at 1-second intervals; deploying a sound wave detector, using a microphone array, to capture sound wave data in a preset frequency band at a preset sampling rate, and generating a spectrum diagram through preprocessing; deploying an infrared thermal imaging matrix, using an uncooled focal plane detector, to capture thermal radiation data at a preset time period, obtain imaging data, and generate a temperature distribution matrix; and deploying a radio electromagnetic signal module to receive signals in a preset frequency band via software radio, and analyze the modulation method and power spectral density.

[0048] Optionally, a wireless network can be built between the smart sensors based on Z-Wave technology. Specifically, this is achieved by setting gateways and networking information in the sensor chips. This allows data to be transmitted between sensors via self-organizing networks or wireless networks, improving data transmission efficiency and security.

[0049] In one embodiment of this application, the environmental data in the real-time data is linearly processed. For example, physical quantities such as smoke concentration (0-1000ppm), carbon dioxide (0-5000ppm), oxygen (0-25%), and temperature (-40~85℃) are linearly processed to generate a first feature. for:

[0050]

[0051] in, This represents a linear factor dynamically adjusted based on the distribution of environmental data, where x represents the environmental data. In this embodiment, the first feature is used to represent the normalized physical quantity. Through the above processing, the nonlinear sensor response is mapped to a linear output, improving the data resolution. For example, the resolution of a temperature sensor in the low-temperature region (-20℃) is improved by 3 times.

[0052] In one embodiment of this application, energy features are extracted from the acoustic wave data in the real-time data. In this process, the acoustic wave data is first decomposed to generate a preset number of frequency bands, and then the energy parameters of the frequency bands are extracted as a second feature. for:

[0053]

[0054] Where i and k represent the frequency band identifiers of the acoustic data, respectively. These represent the wavelet coefficients of the i-th and k-th frequency bands, respectively. In this embodiment, the second feature is used to represent the sound wave energy. Through the above process, the transient features of the explosion sound wave can be extracted, improving the classification accuracy.

[0055] In one embodiment of this application, the imaging data in the real-time data is segmented to generate a third feature. In this embodiment, the third feature is used to represent the area of ​​the hot zone or the temperature range. For example, noise is removed by applying a medium filter, and then the high-temperature region is segmented by an image segmentation algorithm to generate a binary mask of the hot zone.

[0056] In one embodiment of this application, the first feature, the second feature, and the third feature are combined and standardized to generate a multimodal feature matrix. For example, the constructed feature matrix F includes a 4-dimensional first feature, a 1-dimensional second feature, and a 2-dimensional third feature.

[0057] In this embodiment, the raw sensor data is synchronized and aligned, and then nonlinear transformation is used to eliminate range differences, providing a unified input for feature extraction. Normalized data is then processed through wavelet decomposition and entropy calculation to transform the time-domain signal into statistical features, reducing the complexity of subsequent classification. Multimodal features are fused into structured data, which is directly input into a machine learning model for environmental state recognition.

[0058] The above process integrates data from three types of sensors: environment, sound waves, and imaging, to form a feature set covering multiple dimensions such as temperature, humidity, sound wave energy, and object contours. Linear processing of environmental data eliminates dimensional differences, sound wave energy feature extraction captures abnormal sound patterns, and imaging segmentation locates heat sources or smoke areas. After standardization, the combination of these three data points ensures that different modal features are fused at a unified scale, providing a structured and highly usable input foundation for subsequent analysis.

[0059] S120, The feature matrix and the environmental data in the real-time data are dynamically fused to form a fusion vector.

[0060] In this embodiment, by analyzing the spatiotemporal correlation between environmental data and feature matrix in real time, the weight coefficients of each modal feature are dynamically adjusted: for example, when the ambient temperature rises, the influence of thermal imaging features is automatically enhanced, while when the wind speed is high, the contribution of smoke diffusion features is suppressed. Finally, the multimodal features and environmental parameters are superimposed according to weights to form a fusion vector containing environmental adaptive information, providing dynamically optimized input data for subsequent fire alarm judgment.

[0061] In one embodiment of this application, the feature matrix and environmental data in the real-time data are dynamically fused to form a fusion vector, including:

[0062] Environmental factors are determined by environmental data in the real-time data, and membership parameters corresponding to each feature dimension in the feature matrix are determined based on the environmental factors.

[0063] Calculate the feature parameters corresponding to the feature dimension based on the membership parameters;

[0064] The feature matrix is ​​dynamically fused based on the feature parameters to generate a fused vector.

[0065] In one embodiment of this application, after generating a standardized multimodal feature matrix, structured information including environmental sensor values, acoustic energy characteristics, and thermal imaging area distribution is included. Simultaneously, three key environmental parameters—temperature, humidity, and wind speed—are collected in real time as the basis for dynamic adjustment. These environmental parameters are transformed into criteria for judging the importance of different features using fuzzy logic. In this embodiment, the environmental data includes real-time collected ambient temperature, humidity, and wind speed, which serve as the basis for dynamic weight adjustment.

[0066] Environmental factors are determined using environmental data from the real-time data, and membership parameters corresponding to each feature dimension in the feature matrix are determined based on these environmental factors. for:

[0067]

[0068] in, These represent the center value and width generated by clustering historical data, respectively. These represent the left and right boundaries determined based on historical data, respectively. This represents environmental parameters determined based on historical data, where c represents the environmental context vector, and i and m represent feature identifiers and environmental factor identifiers, respectively. This represents gradient operation.

[0069] The importance of each feature dimension is assessed using fuzzy logic rules. For example, when the ambient temperature rises, the focus on smoke concentration and carbon dioxide data is automatically increased; if the wind speed is high, the reliance on sound wave data is reduced. This adjustment is not a linear change, but is achieved through a pre-defined fuzzy rule base. Ultimately, each feature is assigned a feature parameter between 0 and 1, reflecting its contribution to the current environment. In one embodiment of this application, the feature parameter corresponding to the feature dimension is calculated based on the membership parameter. for:

[0070]

[0071] Where i, j, and k represent feature identifiers, These represent the membership parameters corresponding to features j and k, respectively.

[0072] In one embodiment of this application, the feature matrix is ​​dynamically fused based on the feature parameters to generate a fused vector. for:

[0073]

[0074] in, This represents the i-th feature element in the feature matrix. The above calculation uses a standardized feature matrix F to provide a multimodal data baseline. The environmental context vector c drives dynamic weight adjustment, forming an environmentally adaptive fusion representation after dynamic weight adjustment of the multidimensional features. By integrating data from different sources according to their credibility, for example, fire-related features are enhanced in high-temperature environments, while interfering factors (such as wind noise) are suppressed. The fused data retains the integrity of multi-source information and highlights key features through dynamic adjustment, providing more accurate input for subsequent anomaly detection or decision analysis.

[0075] The above process utilizes environmental factors to adjust the weights of each feature dimension. For example, it enhances the influence of thermal imaging features under high temperature and humidity conditions, while suppressing the contribution of smoke diffusion features when wind speed is high. Feature parameters are calculated through membership parameters to achieve a deep correlation between the feature matrix and environmental data, generating a fusion vector that adapts to real-time environmental changes and improving the model's adaptability to complex scenarios.

[0076] S130, generate a thermal imaging spatiotemporal sequence based on the image information in the environmental data, and generate spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector.

[0077] In this embodiment, image data captured by the thermal imaging device at different time points are first collected to form a thermal imaging spatiotemporal sequence arranged in chronological order; simultaneously, a fusion vector containing environmental adaptive information is obtained. Then, the images in the thermal imaging spatiotemporal sequence are aligned and arranged according to spatial location (e.g., height, width) and chronological order to form a three-dimensional spatiotemporal data block. Next, the environmental adaptive information of the fusion vector (e.g., the influence weights of temperature and humidity on features) is fused with the thermal imaging spatiotemporal data block through a dynamic association mechanism, ultimately generating a spatiotemporal tensor feature that simultaneously contains spatiotemporal dynamic patterns and environmental adaptive information, providing structured input for subsequent fire alarm determination.

[0078] In one embodiment of this application, a thermal imaging spatiotemporal sequence is generated based on image information in the environmental data, and spatiotemporal tensor features are generated based on the thermal imaging spatiotemporal sequence and the fusion vector, including:

[0079] A thermal imaging spatiotemporal sequence is generated based on the image information in the environmental data;

[0080] Target features are extracted from the thermal imaging spatiotemporal sequence and the fusion vector using a preset convolution kernel to generate spatiotemporal tensor features.

[0081] In one embodiment of this application, image information from environmental data is acquired to generate a thermal imaging spatiotemporal sequence, which records the temperature distribution of the target area over time. Simultaneously, a fusion vector generated in the previous steps is acquired, which has dynamically adjusted the weights of each sensor's data according to the environment. These two data components together constitute the input basis for this step, similar to simultaneously viewing real-time monitoring footage and historical environmental parameters.

[0082] Using 3D convolution technology, target features are extracted simultaneously in spatial and temporal dimensions. A pre-defined convolution kernel moves simultaneously in four spatial directions (left, right, up, and down) within the thermal imaging image, while also scanning forward and backward along the time axis, extracting core features from the fused vector. This process captures spatiotemporal features, such as the diffusion of high-temperature points from the center outwards, or the dynamic pattern of abnormal temperature regions shrinking over time.

[0083] After extracting the target features, tensor decomposition technology is used to break down the complex target features into a combination of multiple basic components. Secondary information (such as environmental thermal radiation interference) is filtered out, while dominant features (such as the core path of fire spread) are retained, ultimately generating a simpler and more representative feature matrix. The resulting spatiotemporal tensor features are then directly input into subsequent classification or prediction models. Because the spatiotemporal features have been deeply mined and optimized, subsequent analysis can more quickly identify abnormal events, such as equipment overheating or illegal intrusion. For example, in chemical industrial park monitoring, it can accurately locate areas of abnormal temperature, shorten fire early warning response time, and reduce false alarm rates.

[0084] The above process arranges thermal imaging images into a spatiotemporal sequence in chronological order, combines environmental adaptive information from the fused vectors, and extracts spatiotemporal tensor features through convolutional kernels. This process preserves the temporal dynamics of temperature diffusion while incorporating the influence of environmental factors on fire development, resulting in structured spatiotemporal tensor features that can more accurately characterize the spatiotemporal evolution pattern of fire.

[0085] S140, the spatiotemporal tensor features are input into a pre-constructed encoder network, and the attention map and error parameters representing the spatiotemporal distribution of the fire are output.

[0086] In this embodiment, spatiotemporal tensor features are input into a pre-trained encoder network, and spatiotemporal features are extracted layer by layer through multi-layer convolutional kernels: the first layer captures local temperature changes, the second layer fuses diffusion trends, and the last layer integrates global dynamics. The encoder uses residual connections to alleviate gradient vanishing and introduces a spatial attention mechanism to dynamically focus on key areas, such as the high-temperature diffusion front. Finally, it outputs an attention map that identifies the spatiotemporal region of the fire and error parameters that measure the difference between the input and reconstructed data, providing a structured basis for subsequent fire alarm determination.

[0087] In one embodiment of this application, the spatiotemporal tensor features are input into a pre-constructed encoder network, and the output includes an attention map representing the spatiotemporal distribution of the fire and error parameters, including:

[0088] An encoder network is constructed by stacking multiple layers of convolutional kernels.

[0089] The spatiotemporal tensor features are input into the encoder network, and the output is an attention map and error parameters representing the spatiotemporal distribution of the fire.

[0090] In one embodiment of this application, after generating spatiotemporal tensor features, the spatiotemporal dynamic patterns of the thermal imaging sequence, such as temperature diffusion trajectories, are recorded using these features. The encoder extracts deep features from the spatiotemporal tensor features using residual dense blocks. Multiple layers of convolutional kernels are stacked, with each layer capturing spatiotemporal patterns at different scales. Specifically, the first layer of the multi-layer convolutional kernels extracts edge temperature changes, the second layer fuses regional diffusion trends, and the final layer integrates multi-scale features. Residual connections allow the input and output to be directly added, mitigating the gradient vanishing problem in deep networks and ensuring training stability.

[0091] In this embodiment, features at different depths are fused within the residual dense block through cross-layer connections. For example, the first convolutional layer extracts local temperature abrupt changes, the second layer captures the diffusion direction, and the third layer fuses global trends. The output of each residual dense block is a concatenation of the input features and features from all intermediate layers, enhancing feature reuse capabilities. A small gradient is introduced in the negative region through an activation function to maintain feature diversity. The encoder ultimately outputs a high-dimensional hidden state, containing the essence of compressed spatiotemporal features.

[0092] In this embodiment, the decoder introduces a spatial attention mechanism to dynamically focus on key regions. A small convolutional kernel scans the feature map, assigning an attention score to each spatial location. For example, a high-temperature region scores 0.8, and a background region scores 0.2. The scores are normalized by an activation function to form an attention map. The decoder combines the attention map with encoder features, gradually recovering spatiotemporal details through deconvolutional layers to generate a reconstructed tensor.

[0093] Optionally, this process ensures that the model prioritizes fire-related areas and suppresses interference from wind noise, environmental heat radiation, and other factors.

[0094] The final output includes reconstruction error and an attention map. The attention map visualizes the spatiotemporal region the model focuses on, aiding in the localization of the fire source. The reconstruction error measures the difference between the input spatiotemporal tensor features and the reconstructed spatiotemporal tensor features, ensuring that key patterns are preserved.

[0095] Optionally, the reconstruction accuracy and attention distribution can be balanced by the total loss function, for example, by constraining excessive attention concentration through the Hadamard product and determining the regularization coefficient through cross-validation.

[0096] The above process can improve the accuracy of the model in fire detection, reduce the false alarm rate, and achieve precise monitoring.

[0097] The above process uses a multi-layered, stacked convolutional encoder to extract high-order features of the fire's spatiotemporal region through layer-by-layer abstraction, while residual connections alleviate the gradient vanishing problem in deep networks. A spatial attention mechanism dynamically focuses on key areas, such as the high-temperature diffusion front, generating an attention map that identifies the fire's location. Error parameters quantify the difference between the input and reconstructed data; the combination of these two provides dual evidence for fire alarm determination.

[0098] S150, determine the fire alarm probability and warning level based on the attention map and the error parameters.

[0099] In this embodiment, the fire alarm probability is first calculated by combining the differences between the fire spatiotemporal regions identified by the attention map and the reconstruction measured by the error parameters, taking into account historical error fluctuations and environmental attention weights, and dynamically adjusting the threshold: the larger the error and the more concentrated the attention in a region, the higher the fire alarm probability. Subsequently, a fuzzy clustering algorithm is used to classify the warning level into three levels—low, medium, and high—based on the probability values, realizing a closed loop from feature analysis to risk quantification and providing accurate basis for emergency response.

[0100] In one embodiment of this application, determining the fire alarm probability and warning level based on the attention map and the error parameter includes:

[0101] A dynamic threshold is generated based on the error parameters corresponding to a preset number of attention maps.

[0102] The fire alarm probability is generated based on the quotient between the error parameter and the dynamic threshold.

[0103] Based on the preset number of levels and the cluster centers generated during initialization, an early warning level is generated.

[0104] In one embodiment of this application, the reconstruction error reflects the model's ability to reconstruct the spatiotemporal pattern of thermal imaging, while the attention map identifies the key areas of focus for the model, such as the high-temperature diffusion front. A dynamic threshold is generated based on a preset number of attention maps and their corresponding error parameters, using a dynamic threshold function that combines the average value and fluctuation amplitude of historical reconstruction errors with the distribution of the current attention map. for:

[0105]

[0106] Where n and N represent the identifiers and total number of the attention map, respectively. Let H and W represent the preset error factor and weight factor, respectively, and let H and W represent the height and width of the attention map A, respectively. These represent the error parameters corresponding to the nth and Nth attention maps, respectively. This represents standardized calculations. The above calculations allow for setting flexible thresholds for temperature alerts under different weather conditions, avoiding misjudgments caused by fixed thresholds when the environment changes.

[0107] In one embodiment of this application, the fire alarm probability is higher by comparing the current reconstruction error with a threshold. The degree of anomaly is quantified by a probability value, and the fire alarm probability P is generated based on the quotient between the error parameter and the dynamic threshold:

[0108]

[0109] in, This represents the activation function. This represents the error parameter at time t. The above calculation converts the difference into a probability value between 0 and 1, facilitating subsequent classification processing.

[0110] In one embodiment of this application, based on a preset number of levels and the initially generated cluster centers, the warning level D is generated as follows:

[0111]

[0112] in, The membership parameter represents the i-th feature of the context vector c. This represents the cluster center corresponding to the i-th feature.

[0113] The above process, using error parameters and attention map A as input, forms a closed loop of feature extraction, anomaly detection, and probability assessment. By encoding and decoding spatiotemporal features, dynamic thresholds and probability calculations are used to transform them into actionable early warning information, ultimately driving the decision-making system. This achieves a complete process from data anomaly detection to risk quantification, providing an intelligent and adaptive solution for fire alarm early warning.

[0114] This application's technical solution acquires multimodal real-time data through intelligent sensors, extracts features from the real-time data to generate a multimodal feature matrix, dynamically fuses the feature matrix and environmental data from the real-time data to form a fusion vector, generates a thermal imaging spatiotemporal sequence based on image information from the environmental data, and generates spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector, inputs the spatiotemporal tensor features into a pre-constructed encoder network, and outputs an attention map and error parameters representing the spatiotemporal distribution of the fire, and determines the fire alarm probability and warning level based on the attention map and the error parameters. By collaboratively sensing environmental, acoustic, and imaging data through multi-source sensors, a feature matrix covering multi-dimensional features is formed. Combined with dynamic fusion of environmental data, a fusion vector adapting to scene changes is generated. Furthermore, the joint features of the thermal imaging spatiotemporal sequence and the fusion vector are extracted to construct a spatiotemporal tensor feature. The encoder network accurately locates the spatiotemporal region of a fire and quantifies the reconstruction error through layer-by-layer feature abstraction and attention mechanism. Finally, it outputs the fire alarm probability and graded warning through dynamic threshold and fuzzy clustering, realizing the intelligent process from multimodal data collection to risk quantification and warning, and significantly improving the accuracy and environmental adaptability of fire alarm identification.

[0115] The following describes embodiments of the intelligent fire alarm signal recognition system based on multi-source fusion sensing of this application, which can be used to execute the intelligent fire alarm signal recognition method based on multi-source fusion sensing in the above embodiments of this application. It is understood that the intelligent fire alarm signal recognition system based on multi-source fusion sensing can be a computer program (including program code) running on a computer device; for example, the intelligent fire alarm signal recognition system based on multi-source fusion sensing is an application software. This intelligent fire alarm signal recognition system based on multi-source fusion sensing can be used to execute the corresponding steps in the method provided in the embodiments of this application. For details not disclosed in the embodiments of the intelligent fire alarm signal recognition system based on multi-source fusion sensing of this application, please refer to the embodiments of the intelligent fire alarm signal recognition method based on multi-source fusion sensing described above.

[0116] Figure 3 A block diagram of a smart fire alarm signal recognition system based on multi-source fusion sensing according to an embodiment of this application is shown.

[0117] Reference Figure 3 As shown, an intelligent fire alarm signal recognition system based on multi-source fusion sensing according to an embodiment of this application includes:

[0118] The acquisition unit 310 is used to acquire real-time multimodal data through a smart sensor and perform feature extraction on the real-time data to generate a multimodal feature matrix.

[0119] The fusion unit 320 is used to dynamically fuse the feature matrix and the environmental data in the real-time data to form a fusion vector.

[0120] Feature unit 330 is used to generate a thermal imaging spatiotemporal sequence based on image information in the environmental data, and to generate spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector;

[0121] The encoding unit 340 is used to input the spatiotemporal tensor features into a pre-constructed encoder network and output an attention map and error parameters representing the spatiotemporal distribution of the fire.

[0122] The early warning unit 350 is used to determine the fire alarm probability and early warning level based on the attention map and the error parameters.

[0123] In this application, based on the aforementioned scheme, the step of acquiring multimodal real-time data through a smart sensor and extracting features from the real-time data to generate a multimodal feature matrix includes: acquiring multimodal real-time data through a smart sensor, wherein the real-time data includes environmental data, acoustic data, and imaging data; performing linear processing on the environmental data in the real-time data to output a first feature; extracting energy features from the acoustic data in the real-time data as a second feature; performing segmentation processing on the imaging data in the real-time data to generate a third feature; and combining and standardizing the first feature, the second feature, and the third feature to generate a multimodal feature matrix.

[0124] In this application, based on the aforementioned scheme, the step of linearly processing the environmental data in the real-time data to output a first feature includes: generating a linear factor based on the environmental data in the real-time data, and performing linear processing on the environmental data using the linear factor to generate the first feature. for:

[0125]

[0126] in, This represents a linear factor dynamically generated based on the distribution of environmental data, where x represents the environmental data.

[0127] In this application, based on the aforementioned scheme, the step of extracting energy features from the acoustic wave data in the real-time data as a second feature includes: decomposing the acoustic wave data in the real-time data to generate a preset number of frequency bands, and extracting the energy parameters of the frequency bands as the second feature. for:

[0128]

[0129] Where i and k represent the frequency band identifiers of the acoustic data, respectively. represent the wavelet coefficients of the i-th and k-th frequency bands, respectively.

[0130] In this application, based on the aforementioned scheme, the step of dynamically fusing the feature matrix and the environmental data in the real-time data to form a fusion vector includes: determining environmental factors through the environmental data in the real-time data; determining the membership parameters corresponding to each feature dimension in the feature matrix based on the environmental factors; calculating the feature parameters corresponding to the feature dimensions based on the membership parameters; and dynamically fusing the feature matrix based on the feature parameters to generate a fusion vector.

[0131] In this application, based on the aforementioned scheme, the step of generating a thermal imaging spatiotemporal sequence based on image information in the environmental data, and generating spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector, includes: generating a thermal imaging spatiotemporal sequence based on image information in the environmental data; and extracting target features from the thermal imaging spatiotemporal sequence and the fusion vector using a preset convolution kernel to generate spatiotemporal tensor features.

[0132] In this application, based on the aforementioned scheme, the step of inputting the spatiotemporal tensor features into a pre-constructed encoder network and outputting an attention map and error parameters representing the spatiotemporal distribution of the fire includes: constructing an encoder network by stacking multiple layers of convolutional kernels; inputting the spatiotemporal tensor features into the encoder network and outputting an attention map and error parameters representing the spatiotemporal distribution of the fire.

[0133] In this application, based on the aforementioned scheme, the step of determining the fire alarm probability and warning level according to the attention map and the error parameter includes: generating a dynamic threshold based on the error parameters corresponding to a preset number of attention maps; generating a fire alarm probability based on the quotient between the error parameter and the dynamic threshold; and generating a warning level based on a preset number of levels and the initialized cluster centers.

[0134] This application's technical solution acquires multimodal real-time data through intelligent sensors, extracts features from the real-time data to generate a multimodal feature matrix, dynamically fuses the feature matrix and environmental data from the real-time data to form a fusion vector, generates a thermal imaging spatiotemporal sequence based on image information from the environmental data, and generates spatiotemporal tensor features based on the thermal imaging spatiotemporal sequence and the fusion vector, inputs the spatiotemporal tensor features into a pre-constructed encoder network, and outputs an attention map and error parameters representing the spatiotemporal distribution of the fire, and determines the fire alarm probability and warning level based on the attention map and the error parameters. By collaboratively sensing environmental, acoustic, and imaging data through multi-source sensors, a feature matrix covering multi-dimensional features is formed. Combined with dynamic fusion of environmental data, a fusion vector adapting to scene changes is generated. Furthermore, the joint features of the thermal imaging spatiotemporal sequence and the fusion vector are extracted to construct a spatiotemporal tensor feature. The encoder network accurately locates the spatiotemporal region of a fire and quantifies the reconstruction error through layer-by-layer feature abstraction and attention mechanism. Finally, it outputs the fire alarm probability and graded warning through dynamic threshold and fuzzy clustering, realizing the intelligent process from multimodal data collection to risk quantification and warning, and significantly improving the accuracy and environmental adaptability of fire alarm identification.

[0135] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0136] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the function and scope of use of the embodiments of this application.

[0137] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes based on a program stored in a read-only memory 402 or a program loaded from a storage section 408 into a random access memory 403, such as executing the intelligent fire alarm signal recognition method based on multi-source fusion sensing described in the above embodiment. The random access memory 403 also stores various programs and data required for system operation. The central processing unit 401, the read-only memory 402, and the random access memory 403 are interconnected via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0138] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0139] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0140] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0143] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0144] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the intelligent fire alarm signal recognition method based on multi-source fusion sensing described in the above embodiments.

[0145] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0146] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0147] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0148] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for intelligent fire alarm signal identification based on multi-source fusion perception, characterized in that, The method comprises the following steps: obtaining real-time data of multiple modalities through intelligent sensors, and performing feature extraction on the real-time data to generate a feature matrix of multiple modalities; performing dynamic fusion on the feature matrix and environmental data in the real-time data to form a fusion vector; generating a thermal imaging space-time sequence based on image information in the environmental data, and generating a space-time tensor feature according to the thermal imaging space-time sequence and the fusion vector; inputting the space-time tensor feature into a pre-constructed encoder network to output an attention map representing the space-time distribution of fire and an error parameter; determining a fire alarm probability and an early warning level according to the attention map and the error parameter; wherein the dynamic fusion of the feature matrix and the environmental data in the real-time data to form the fusion vector comprises: determining an environment factor through environment data in the real-time data, determining a membership parameter corresponding to each feature dimension in the feature matrix respectively based on the environment factor is: ； wherein, respectively represent a center value and a width generated by clustering historical data, respectively represent a left boundary and a right boundary determined based on historical data, represent an environmental parameter determined based on historical data, c represents an environmental context vector, i and m respectively represent a feature identifier and an environmental factor identifier, represents a gradient operation; calculating a feature parameter corresponding to the feature dimension based on the membership parameter; performing dynamic fusion on the feature matrix based on the feature parameter to generate a fusion vector. 2.The intelligent fire alarm signal recognition method based on multi-source fusion perception according to claim 1, characterized in that, The method of obtaining real-time data of multiple modalities through intelligent sensors, and performing feature extraction on the real-time data to generate a feature matrix of multiple modalities comprises: obtaining real-time data of multiple modalities through intelligent sensors, wherein the real-time data comprises environmental data, sound wave data, and imaging data; performing linear processing on the environmental data in the real-time data to output a first feature; extracting energy features from the sound wave data in the real-time data as a second feature; performing segmentation processing on the imaging data in the real-time data to generate a third feature; combining and standardizing the first feature, the second feature, and the third feature to generate a feature matrix of multiple modalities. 3.The intelligent fire alarm signal recognition method based on multi-source fusion perception according to claim 2, characterized in that, The linear processing on the environmental data in the real-time data to output a first feature comprises: generate a linear factor according to environment data in the real-time data, linearly process the environment data through the linear factor to generate a first feature is: ； wherein, represents a linear factor dynamically generated according to a distribution of the environment data, x represents the environment data.

4. The multi-source fusion perception-based intelligent fire alarm signal identification method according to claim 3, characterized in that, The extraction of energy features from the sound wave data in the real-time data as a second feature comprises: The sound wave data in the real-time data is decomposed to generate a preset number of frequency bands, and an energy parameter of the frequency band is extracted as a second feature : ； where i and k represent the band identification of the acoustic data, respectively, Wi,kand Wk,i represent the wavelet coefficients of the i-th and k-th band, respectively. 5.The intelligent fire alarm signal recognition method based on multi-source fusion perception according to claim 1, characterized in that, The generation of a thermal imaging space-time sequence based on image information in the environmental data, and the generation of a space-time tensor feature according to the thermal imaging space-time sequence and the fusion vector comprise: generating a thermal imaging space-time sequence based on image information in the environmental data; extracting target features from the thermal imaging space-time sequence and the fusion vector through a pre-set convolution kernel to generate a space-time tensor feature. 6.The intelligent fire alarm signal recognition method based on multi-source fusion perception according to claim 1, characterized in that, The inputting of the space-time tensor feature into a pre-constructed encoder network to output an attention map representing the space-time distribution of fire and an error parameter comprises: constructing an encoder network through a multi-layer convolution kernel stack; inputting the space-time tensor feature into the encoder network to output an attention map representing the space-time distribution of fire and an error parameter.

7. The multi-source fusion perception-based intelligent fire alarm signal identification method according to claim 1, characterized in that, The determination of a fire alarm probability and an early warning level according to the attention map and the error parameter comprises: generating a dynamic threshold based on error parameters corresponding to a preset number of attention maps; generating a fire alarm probability based on the quotient between the error parameter and the dynamic threshold; generating an early warning level based on a preset number of levels and an initialized cluster center.

8. A smart fire alarm signal identification system based on multi-source fusion perception, characterized in that, The method comprises the following steps: an acquisition unit is configured to obtain real-time data of multiple modalities through intelligent sensors, and perform feature extraction on the real-time data to generate a feature matrix of multiple modalities; a fusion unit is configured to perform dynamic fusion on the feature matrix and environmental data in the real-time data to form a fusion vector; The feature unit is configured to generate a thermal imaging spatio-temporal sequence based on image information in the environmental data, and generate a spatio-temporal tensor feature according to the thermal imaging spatio-temporal sequence and the fusion vector; The encoding unit is configured to input the spatio-temporal tensor feature into a pre-constructed encoder network, and output an attention map and an error parameter representing a fire spatio-temporal distribution; The early warning unit is configured to determine a fire alarm probability and an early warning level according to the attention map and the error parameter; The feature matrix and the environmental data in the real-time data are dynamically fused to form a fusion vector, including: determining an environment factor through environment data in the real-time data, determining a membership parameter corresponding to each feature dimension in the feature matrix respectively based on the environment factor is: ； wherein, respectively represent a center value and a width generated by clustering historical data, respectively represent a left boundary and a right boundary determined based on historical data, represent an environmental parameter determined based on historical data, c represents an environmental context vector, i and m respectively represent a feature identifier and an environmental factor identifier, represents a gradient operation; calculating a feature parameter corresponding to the feature dimension based on the membership parameter; dynamically fusing the feature matrix based on the feature parameter to generate a fusion vector. 9.The intelligent fire alarm signal recognition system based on multi-source fusion perception according to claim 8, characterized in that, The multi-modal real-time data are acquired by intelligent sensors, and feature extraction is performed on the real-time data to generate a multi-modal feature matrix, including: The multi-modal real-time data are acquired by intelligent sensors, and the real-time data include environmental data, sound wave data, and imaging data; The environmental data in the real-time data are linearly processed to output a first feature; Energy features are extracted from the sound wave data in the real-time data as a second feature; The imaging data in the real-time data are segmented to generate a third feature; The first feature, the second feature, and the third feature are combined and standardized to generate a multi-modal feature matrix.

Images