Environment detection and recognition method and system based on image recognition

Abstract

The application discloses an environment detection and identification method and system based on image recognition, collects multi-modal image data composed of an environment image and an environment temperature distribution image; an improved convolutional neural network is established based on EfficientNet as a backbone to extract spatial static features of the multi-modal image data, and a RNN recurrent neural network is used to extract time dynamic features in the multi-modal image data; the weights of the spatial static features and the time dynamic features are adjusted through a self-adaptive cross-modal attention mechanism, and are fused to obtain fused feature data; the fused feature vector is input into an environment state classifier to identify an environment state, an environment detection report is generated according to an identification result, and a corresponding environment early warning mechanism is triggered. The application can accurately locate abnormalities and distinguish interference, and the environment detection precision is improved compared with traditional single-mode environment detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an environment detection and recognition method and system based on image recognition. Background Technology

[0002] Traditional environmental monitoring methods have significant limitations in industrial production, data center maintenance, and other scenarios. Single sensors can only acquire data from a single dimension; for example, a temperature sensor measures only a single point, making it unable to pinpoint the source of anomalies. They are also prone to false alarms and missed alarms, and require dense deployment, resulting in high costs and difficult maintenance. Manual inspections are difficult to conduct 24 / 7 monitoring, and inspections of large-scale scenarios are time-consuming, making it difficult to promptly capture sudden anomalies. Furthermore, results are influenced by human experience, leading to significant subjective errors. While existing technologies include image recognition-based detection methods, some only use single-modal images and cannot integrate key information such as temperature. Their accuracy is insufficient for specific scenarios, resulting in poor detection performance in different environments and failing to meet the real-time, accurate detection requirements of complex scenarios. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing an environment detection and recognition method and system based on image recognition.

[0004] To achieve the above objectives, the technical solution of the present invention further includes the following steps in the above-mentioned environment detection and recognition method based on image recognition:

[0005] Collect multimodal image data consisting of environmental images and environmental temperature distribution images;

[0006] An improved convolutional neural network is built based on EfficientNet as the backbone to extract the spatial static features of multimodal image data, and the temporal dynamic features of the multimodal image data are extracted using an RNN recurrent neural network.

[0007] The weights of spatial static features and temporal dynamic features are adjusted by an adaptive cross-modal attention mechanism and then fused to obtain fused feature data.

[0008] The fused feature vector is input into a Transformer-based environmental state classifier to identify the environmental state. Based on the identification results, an environmental monitoring report is generated and a corresponding environmental early warning mechanism is triggered.

[0009] Furthermore, in the above-mentioned environment detection and recognition method based on image recognition, the acquisition of multimodal image data composed of environmental images and environmental temperature distribution images includes:

[0010] The environmental image and environmental temperature distribution image are acquired by the sensor. Gaussian filtering is used to remove noise from the environmental image. Adaptive histogram equalization is used to enhance the contrast of the environmental image. Finally, the image size is normalized to obtain the preprocessed environmental image.

[0011] The ambient temperature distribution image is calibrated, and isolated noise points in the heat map are removed by median filtering. The temperature data is then mapped to a grayscale image to obtain a temperature grayscale image.

[0012] Based on the timestamp at the time of acquisition, the preprocessed environmental image and the temperature grayscale image are spatially aligned to obtain multimodal image data.

[0013] Furthermore, in the above-mentioned environment detection and recognition method based on image recognition, the step of establishing an improved convolutional neural network based on EfficientNet as the backbone to extract spatial static features of multimodal image data and using an RNN recurrent neural network to extract temporal dynamic features from the multimodal image data includes:

[0014] Based on EfficientNet-B4 as the backbone network, a BiFPN bidirectional feature pyramid network module is added to the transition layer between feature extraction and feature fusion.

[0015] By fusing features from different layers of EfficientNet through bidirectional paths, shallow features capture detailed information related to edges and textures, while deep features capture global information related to object categories and scene semantics.

[0016] Furthermore, in the above-mentioned environment detection and recognition method based on image recognition, the step of establishing an improved convolutional neural network based on EfficientNet as the backbone to extract spatial static features of multimodal image data and using an RNN recurrent neural network to extract temporal dynamic features from the multimodal image data includes:

[0017] The preprocessed environmental image and temperature grayscale image are input into the improved EfficientNet through channel stitching. The network extracts features step by step through convolutional layers, batch normalization layers, and Swish activation function, and outputs a spatial static feature vector.

[0018] The output spatial static features are subjected to L2 regularization, and key features are enhanced by the SE attention module to suppress interference from irrelevant background features, thus obtaining spatial static features.

[0019] Furthermore, in the above-mentioned environment detection and recognition method based on image recognition, the step of establishing an improved convolutional neural network based on EfficientNet as the backbone to extract spatial static features of multimodal image data and using an RNN recurrent neural network to extract temporal dynamic features from the multimodal image data includes:

[0020] According to the acquisition time sequence, the spatial static feature vectors of N consecutive frames are combined into a time series to form time series data with dimensions [N, 2048].

[0021] A bidirectional LSTM network was selected as the time feature extraction network, with two hidden layers. The information transmission in the time series was dynamically adjusted through forget gate, input gate, and output gate.

[0022] The output of the last time step of the bidirectional LSTM is compressed to obtain a time dynamic feature vector, which includes at least the temperature change rate and the duration of the abnormal state.

[0023] Furthermore, in the above-mentioned environment detection and recognition method based on image recognition, the step of adjusting the weights of spatial static features and temporal dynamic features through an adaptive cross-modal attention mechanism and fusing them to obtain fused feature data includes:

[0024] The cosine similarity between the spatial static feature vector and the temporal dynamic feature vector is calculated after unifying their dimensions through a fully connected layer.

[0025] A weighted prediction network is constructed using two fully connected layers and a sigmoid activation function. The inputs are modal similarity and preliminary features of the current frame environment, and the outputs are two weight values. Feature fusion is performed based on the weight values ​​to obtain fused feature data.

[0026] Furthermore, in the above-mentioned image recognition-based environment detection and identification method, the step of inputting the fused feature vector into a Transformer-based environment state classifier to identify the environment state, generating an environment detection report based on the identification result, and triggering a corresponding environment early warning mechanism includes:

[0027] A Transformer classifier is constructed with a 6-layer encoder as the core and a multi-head self-attention module consisting of 8 heads.

[0028] The fused feature vector is used as input to the Transformer encoder. The semantic relationships between features are mined through the self-attention module. The environment state probability distribution is output through the fully connected layer and the Softmax activation function. The category with the highest probability is selected as the environment state recognition result.

[0029] Furthermore, in the image recognition-based environment detection and recognition system, the environment detection and recognition system includes the following modules:

[0030] The image data acquisition module is used to acquire multimodal image data consisting of environmental images and environmental temperature distribution images;

[0031] The image feature extraction module is used to build an improved convolutional neural network based on EfficientNet as the backbone to extract the spatial static features of multimodal image data and to extract the temporal dynamic features of the multimodal image data using an RNN recurrent neural network.

[0032] The image feature fusion module is used to adjust the weights of spatial static features and temporal dynamic features through an adaptive cross-modal attention mechanism, and then fuse them to obtain fused feature data.

[0033] The environmental detection and identification module is used to input the fused feature vector into the Transformer-based environmental state classifier to identify the environmental state, generate an environmental detection report based on the identification results, and trigger the corresponding environmental early warning mechanism.

[0034] Furthermore, in the image recognition-based environment detection and recognition system, the image feature fusion module includes the following sub-modules:

[0035] The computation submodule is used to calculate the cosine similarity between the spatial static feature vector and the temporal dynamic feature vector after unifying their dimensions through a fully connected layer.

[0036] The fusion submodule is used to construct a weighted prediction network consisting of two fully connected layers and a sigmoid activation function. The input is modal similarity and preliminary features of the current frame environment, and the output is two weight values. Feature fusion is performed based on the weight values ​​to obtain fused feature data.

[0037] Furthermore, in the image recognition-based environment detection and recognition system, the environment detection and recognition module includes the following sub-modules:

[0038] Construct a submodule to build a Transformer classifier with a 6-layer encoder as the core and a multi-head self-attention module of 8 heads.

[0039] The output submodule is used to take the fused feature vector as input to the Transformer encoder, mine the semantic association between features through the self-attention module, output the probability distribution of the environment state through the fully connected layer and the Softmax activation function, and select the category with the highest probability as the environment state recognition result.

[0040] Its beneficial effects lie in the fact that by fusing RGB and temperature images and standardizing them to achieve synergy between visual and physical information, it can accurately locate anomalies and distinguish interference, improving the accuracy of environmental detection compared to traditional single-modality methods. Employing an improved EfficientNet + bidirectional LSTM to extract spatiotemporal features, it can statically locate anomalies and dynamically track trends, predicting severe anomalies 5-10 minutes in advance, reducing the probability of accidents. Through an adaptive attention mechanism, it dynamically adjusts feature weights, significantly improving scene adaptability. Based on Transformer classification and a three-level early warning system, it improves operational efficiency and saves manpower and resources. It can also generate structured reports to facilitate data backtracking and model optimization, driving enterprises to shift from passive response to proactive prevention and improving environmental management levels. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0042] Figure 1 This is a schematic diagram of the first embodiment of the environment detection and recognition method based on image recognition in this invention.

[0043] Figure 2 This is a schematic diagram of a second embodiment of the environment detection and recognition method based on image recognition in this invention.

[0044] Figure 3 This is a schematic diagram of the first embodiment of the image recognition-based environment detection and recognition system in this invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms "one," "an," and "this" used herein may also include the plural forms. It should be further understood that the terminology used in this specification includes the presence of features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0047] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, an environment detection and recognition method based on image recognition includes the following steps:

[0048] Step 101: Acquire multimodal image data consisting of environmental images and environmental temperature distribution images;

[0049] Specifically, in this embodiment, environmental images and environmental temperature distribution images are acquired through sensors. Gaussian filtering is used to remove noise from the environmental images, and adaptive histogram equalization is used to enhance the contrast of the environmental images before the image size is normalized to obtain a preprocessed environmental image. The environmental temperature distribution image is calibrated, and median filtering is used to remove isolated noise points in the heat map before the temperature data is mapped to a grayscale image to obtain a temperature grayscale image. Based on the timestamp at the time of acquisition, the preprocessed environmental image and the temperature grayscale image are spatially aligned to obtain multimodal image data.

[0050] In the multi-source data acquisition phase, equipment selection and deployment must closely align with the scenario requirements to ensure comprehensive and accurate data acquisition. When selecting RGB industrial cameras, prioritize models with high dynamic range and HDR functionality. These cameras can clearly capture scene details in strong light or backlight environments. For example, the strong light generated by welding operations in a workshop will not cause image overexposure, and they can accurately record visual information such as the trajectory of rising smoke and color changes on the surface of equipment. For infrared thermal imagers, choose models with a resolution of at least 384×288 pixels. Their detection temperature range covers -20℃ to 500℃, clearly showing temperature gradients caused by uneven heat dissipation from equipment. For example, the location of hotspots caused by localized overheating of servers in a server room can be accurately pinpointed to the centimeter level. During deployment, both types of equipment must be fixed at the same horizontal height in the same detection area using brackets, such as on a beam in the workshop ceiling or on a load-bearing column in the corner of the server room. Before installation, use a laser level to calibrate the lens angle to ensure that the field of view of the RGB industrial camera completely overlaps with that of the infrared thermal imager, avoiding data mismatch due to differences in the acquisition range. The acquisition parameters need to be dynamically adjusted according to the scenario. In static environments such as unattended warehouses, a acquisition frequency of 1 frame / 5 seconds can save storage resources while ensuring data validity. In dynamic environments such as production line operation areas, a frequency of 1 frame / second can capture abnormal changes in material handling and equipment operation in real time. The RGB image resolution is fixed at 224×224 because this size is highly compatible with the input specifications of the subsequent EfficientNet network, which can reduce feature loss caused by image scaling. The infrared thermal imager outputs 16-bit depth temperature data, which can record temperature differences more accurately than 8-bit data. For example, it can accurately distinguish between normal operating temperature and abnormal temperature when the difference is only 0.3℃. Moreover, the timestamp of each frame of data needs to be accurate to the millisecond level, which provides a foundation for subsequent multimodal image data alignment. The initial data screening employs a combination of automated and manual sampling. Algorithms automatically identify blurry frames; frames with an image sharpness score below a preset threshold (e.g., 0.8) are marked as blurry, often indicating lens obstruction by dust or moisture. Abnormal temperature frames are identified by checking for five or more consecutive frames with zero temperatures or values ​​exceeding the device's detection range; for example, infrared thermal imagers may display zero-temperature data when powered off. Manual sampling randomly selects 10% of the screened data to check for algorithmic errors, ensuring that the accuracy of the retained valid data is no less than 99%.

[0051] Multimodal image data preprocessing is crucial for improving subsequent feature extraction. It requires separate processing for RGB images and temperature distribution images based on their characteristics. In RGB image preprocessing, a 3×3 convolution kernel is used for Gaussian filtering. This size effectively filters Gaussian noise in the image, such as noise caused by electromagnetic interference from motors in a workshop, without excessively blurring image edge details. Adaptive histogram equalization processes the image in blocks, enhancing weak visual features such as smoke and water stains while avoiding overall brightness distortion. For example, slight water stains in a warehouse corner are not easily noticeable before processing, but their outlines and extent are clearly presented after processing. When the image size is normalized to 224×224, a bilinear interpolation algorithm is used to ensure a smooth transition of pixel information during image stretching. Pixel values ​​are normalized to the [0,1] range by dividing the original pixel values ​​of 0-255 by 255, eliminating the influence of different pixel value ranges on network training and enabling the model to learn features more stably. The temperature distribution image preprocessing first involves equipment calibration. Standard isothermal blocks at 50℃ and 100℃ are placed in the detection area, and the measured values ​​from the infrared thermal imager are compared with the standard values. A linear correction algorithm is used to adjust the equipment parameters to ensure that the temperature error is controlled within ±0.5℃. For example, for a standard isothermal block at 50℃, the thermal imager measurement should be between 49.5℃ and 50.5℃. Median filtering also uses a 3×3 convolution kernel to effectively remove isolated noise points in the heatmap, such as instantaneous high temperatures caused by flying insects, avoiding such interference from subsequent temperature feature analysis. When mapping the temperature data to an 8-bit grayscale image, a linear mapping formula is used to map the temperature range of -20℃ to 500℃ to a grayscale value range of 0-255. This facilitates inputting the data into the network in conjunction with RGB images and allows for a direct reflection of temperature changes through grayscale differences, while retaining the original 16-bit temperature data to provide accurate temperature values ​​for subsequent feature association. Multimodal image data alignment is based on timestamps. First, RGB images and temperature distribution images with timestamp errors within ±10 milliseconds are selected. Then, an image registration algorithm based on feature point matching is used to extract key feature points in the two types of images, such as device corners and fixed markers. The positional offset caused by installation angle differences is corrected by calculating the transformation matrix. Finally, the position of the same object in the scene is completely corresponding in the two types of images. The resulting RGB-temperature multimodal image data pairs lay the foundation for subsequent spatiotemporal feature fusion.

[0052] Step 102: Based on EfficientNet as the backbone, build an improved convolutional neural network to extract the spatial static features of multimodal image data, and use an RNN recurrent neural network to extract the temporal dynamic features of multimodal image data.

[0053] Specifically, this embodiment uses EfficientNet-B4 as the backbone network, and adds a BiFPN bidirectional feature pyramid network module to the transition layer between feature extraction and feature fusion. Features from different levels of EfficientNet are fused through bidirectional paths, where shallow features capture edge and texture-related details, and deep features capture global information related to object categories and scene semantics. The preprocessed environmental image and temperature grayscale image are input into the improved EfficientNet via channel concatenation. The network extracts features step-by-step through convolutional layers, batch normalization layers, and the Swish activation function, outputting a spatial static feature vector. The output spatial static features are then subjected to L2 regularization, and a SE attention module is used to enhance key features and suppress interference from irrelevant background features, resulting in the final spatial static features. According to the acquisition time sequence, the spatial static feature vectors of N consecutive frames are combined into a time series to form time series data with dimensions [N, 2048]. A bidirectional LSTM is selected as the time feature extraction network, with two hidden layers. The information transmission in the time series is dynamically adjusted through forget gate, input gate, and output gate. The output of the last time step of the bidirectional LSTM is dimensionally compressed to obtain the time dynamic feature vector, which includes at least the temperature change rate and the duration of the abnormal state.

[0054] The core of spatial static feature extraction based on the improved EfficientNet lies in accurately capturing key static information in the environment through network structure optimization and multimodal image data collaboration. Regarding network structure improvement, EfficientNet-B4 was chosen as the basic backbone network because it achieves a balance between parameter quantity and performance. Compared to EfficientNet-B3, its feature extraction capability is improved by more than 15%, while the number of parameters only increases by 20%, avoiding excessive hardware computational burden. The BiFPN module added to the network neck employs a bidirectional path design: the upward path is responsible for transmitting deep global semantic features from EfficientNet, such as scene-level information like overall device overheating, to the shallower layers, supplementing the semantic correlation of shallow features; the downward path transmits detailed features from the shallow layers, such as the edge texture of the overheated area, to the deeper layers, enhancing the detailed expression of deep features. For example, when detecting a scene where a device exhibits localized high temperature accompanied by discoloration of the casing, BiFPN can fuse the deep semantic features of device anomalies with the detailed features of the shallow discoloration area's edges, making subsequent feature extraction more accurate. In the multimodal image data input stage, a channel stitching method is used to combine a 3-channel RGB image with a 1-channel temperature grayscale image into 4-channel data. This method can completely preserve the original feature association between the two types of images. The 3 channels of the RGB image correspond to red, green, and blue information respectively, which can present visual features such as device color changes and smoke color; the 1 channel of the temperature grayscale image maps the temperature level with grayscale values, which can intuitively reflect the distribution of high-temperature areas. After inputting into the improved EfficientNet, the network uses convolutional layers, 3×3 convolutional kernels to extract local features, and batch normalization layers to accelerate training convergence and avoid gradient vanishing. The Swish activation function, compared with ReLU, can better activate the synergistic effect of weak features, gradually transforming the 4-channel original data into high-dimensional features. The final output 2048-dimensional spatial static feature vector contains not only visual static information such as device shape and smoke location, but also thermal static information such as temperature distribution pattern and high-temperature point coordinates.

[0055] Based on RNN and LSTM, the focus of temporal dynamic feature extraction is to uncover the changing patterns of environmental states over time, providing temporal support for anomaly early warning. In the time series construction phase, a sequence of N=10 consecutive spatial static feature vectors is selected. This is a result of comprehensively considering detection accuracy and real-time performance. If N < 10, insufficient data may prevent the capture of slow changes, such as a gradual temperature increase of 1°C per minute. If N > 10, it will increase data processing latency, making it difficult to meet real-time early warning requirements. The 10 feature frames are arranged in chronological order of acquisition, forming a time series with dimensions [10, 2048]. Each feature frame corresponds to a time step, such as a 1-second interval between steps. The 10 steps cover a 10-second time period, which can completely record the dynamic process of smoke appearing from nothing and temperature changing from normal to abnormal. In terms of LSTM network configuration, bidirectional LSTM is chosen to overcome the limitation of unidirectional LSTM, which can only utilize historical information and features from previous time steps. The forward LSTM extracts historical dynamic information from frames 1 to 10, such as the temperature change trend in the previous 5 seconds, while the backward LSTM extracts future dynamic information from frames 10 to 1, such as the temperature change trend in the next 5 seconds. Combining the two allows for a more comprehensive capture of temporal correlations. The two hidden layers, each with 1024 hidden units, allow for deeper feature learning through multi-layer nonlinear transformations: the first hidden layer extracts basic temporal features, such as the rate of temperature change, while the second hidden layer further explores complex dynamic patterns, such as the acceleration of the rate of temperature change. The synergistic effect of the forget gate, input gate, and output gate is the core advantage of LSTM. When processing transient light changes, such as sudden flickering of workshop lights, the forget gate reduces the corresponding feature weights to below 0.1 to reduce interference. When processing critical dynamic information such as continuously rising temperature, the input gate increases the feature weights to above 0.8 to ensure that effective information is preserved. The output gate then passes the filtered dynamic information to the next layer. In the temporal feature output stage, the forward and backward hidden states of the last time step of the bidirectional LSTM are each concatenated with 1024 dimensions to form a 2048-dimensional feature. This is then compressed to 1024 dimensions through a fully connected layer. This reduces the computational load of subsequent processing and enhances key dynamic features, such as the rate of temperature change and the duration of abnormal states, through dimensional compression. The final 1024-dimensional temporal dynamic feature vector accurately reflects temporal information such as whether the rate of temperature increase exceeds the limit and whether the smoke diffusion speed is accelerated, providing important support for subsequent feature fusion and anomaly recognition.

[0056] Step 103: Adjust the weights of spatial static features and temporal dynamic features through an adaptive cross-modal attention mechanism, and then fuse them to obtain fused feature data;

[0057] Specifically, in this embodiment, the spatial static feature vector and the temporal dynamic feature vector are unified in dimension through a fully connected layer, and the cosine similarity of the two types of features is calculated. A weight prediction network is constructed by two fully connected layers and a sigmoid activation function. The input is modal similarity and preliminary features of the current frame environment, and the output is two weight values. Feature fusion is performed based on the weight values ​​to obtain fused feature data.

[0058] Attention weight calculation is the core of achieving adaptive fusion of multimodal features. By accurately assessing modal importance and dynamically learning weights, it ensures the priority utilization of highly discriminative features. In the modal importance assessment stage, the dimensionality difference between spatial static features and temporal dynamic features must first be addressed. Therefore, the dimensions of the two types of features are first unified through fully connected layers: a 2048→1024 fully connected layer is designed for spatial static features, and a 1024→1024 fully connected layer is designed for temporal dynamic features. Only linear transformations are performed to preserve the original feature distribution, and the two layers share training parameters to avoid the impact of parameter differences on feature correlation. After dimensionality unification, cosine similarity is used to calculate the correlation between the two types of features. Its core is to measure the cosine value of the angle between the two vectors, with a value range of [-1, 1]. The closer the value is to 1, the higher the correlation.

[0059] Adaptive weight learning relies on a weight prediction network, which employs a lightweight structure of two fully connected layers plus a sigmoid activation function. This ensures weight prediction accuracy while avoiding excessive computation. The first fully connected layer has an input dimension of 1, representing modal similarity + K and the initial feature dimension of the current frame environment, where K is set according to the initial feature type. The output dimension of the first fully connected layer is set to 64, and the nonlinear expression is enhanced by the ReLU activation function. The second fully connected layer has an input dimension of 64 and an output dimension of 2, corresponding to w1 (spatial feature weight) and w2 (temporal feature weight), respectively. Finally, the output value is mapped to the [0,1] interval by the sigmoid activation function, and the normalization operation w1 = w1 / (w1+w2) and w2 = w2 / (w1+w2) ensures that w1+w2 = 1. For example, in a static scenario like a drone room, where environmental conditions change slowly and temporal dynamic features have low discriminative power, the initial features are normal temperature and no smoke. The weight prediction network would adjust w1 to 0.7-0.8 and w2 to 0.2-0.3. In a dynamic scenario like a production workshop, where equipment operates frequently, temporal features such as the rate of temperature rise are more critical. w1 would drop to 0.3-0.4 and w2 would rise to 0.6-0.7. This dynamic weight adjustment relies on a frame-level real-time update mechanism. Each time a frame of multimodal image data is processed, the modal similarity and initial features of the current frame are re-inputted, allowing the weight prediction network to iteratively update w1 and w2 in real time. For instance, in a sudden smoke scene, if the spatial features of the current frame suddenly show visual smoke features, with a sudden drop in grayscale values, the initial features would mark this as an abnormal smoke situation. The network would instantly increase w1 to above 0.9 within 1-2 processing cycles, ensuring that static smoke features participate in the fusion process first, avoiding missed detections due to the lag in temporal features and the time required for smoke diffusion.

[0060] Multimodal feature fusion and optimization are key to transforming spatial static features and temporal dynamic features into a unified representation. Through weighted fusion and feature optimization, the completeness and accuracy of environmental state representation are improved. In the weighted feature fusion stage, w1 and w2, calculated using attention weights, are the core, and a weighted sum is achieved using the formula: Fusion Feature = w1 × Spatial Static Feature + w2 × Temporal Dynamic Feature. It's important to note that the spatial static features here have been normalized to 1024 dimensions through a fully connected layer, consistent with the temporal dynamic features, allowing for direct element-wise multiplication and addition. For example, when w1 = 0.8 and w2 = 0.2, 80% of the information in the fused feature comes from spatial static features, such as smoke location and high-temperature region coordinates, while 20% comes from temporal dynamic features, such as smoke diffusion rate and temperature change trend, highlighting both static anomalies in the current frame and preserving dynamic change patterns. If w1 = 0.3 and w2 = 0.7, the fused feature primarily uses temporal dynamic features, such as the rate of temperature increase, supplemented by spatial features, such as whether the high-temperature region is expanding, adapting to the needs of dynamic scenes.

[0061] Feature fusion optimization addresses the issues of distribution shift and insufficient nonlinearity in the fused features through a combination of batch normalization and ReLU activation. During batch normalization, the mean and variance of the fused features are calculated for each batch of 32 frames, standardizing the features to a distribution with a mean of 0 and a variance of 1. This avoids feature value fluctuations caused by previous weighted fusion, such as excessively high feature values ​​in some frames due to a large w1 value, thus accelerating the convergence of the subsequent Transformer classifier training. For example, if the difference between smoke concentration and background interference in the fused features is negative, indicating that the interference outweighs the effective information, ReLU will suppress it; if the difference between temperature anomaly and normal threshold is positive, indicating significant effective information, it will be fully preserved. The final 1024-dimensional fused feature vector contains both spatial static information such as device shape and high-temperature location, and temporal dynamic information such as temperature change rate and smoke diffusion trend, comprehensively representing the current state and changing patterns of the environment, providing high-quality feature input for subsequent environmental state classification. Step 104: Input the fused feature vector into the Transformer-based environmental state classifier to identify the environmental state, generate an environmental monitoring report based on the identification results, and trigger the corresponding environmental early warning mechanism.

[0062] Specifically, in this embodiment, a 6-layer encoder is used as the core, and the multi-head self-attention module is set to 8 heads to construct a Transformer classifier. The fused feature vector is used as the input of the Transformer encoder. The semantic relationship between features is mined through the self-attention module. The environment state probability distribution is output through a fully connected layer and a Softmax activation function. The category with the highest probability is selected as the environment state recognition result.

[0063] Based on the Transformer, environmental state classification achieves accurate and reliable state recognition by leveraging the semantic association mining capabilities of the encoder. The Transformer classifier is centered around a 6-layer encoder, with each layer's multi-head self-attention module having 8 heads, capable of simultaneously capturing the associations across different dimensions of the fused features. The feedforward neural network's hidden layer dimension is set to 2048, which deepens feature expression through non-linear transformations, converting the 1024-dimensional fused features into more discriminative high-dimensional features. Logically, after the final fused features are input into the encoder, the self-attention module calculates the association weights between feature elements. For example, the weights of features like temperature > 60℃ and reddening device surfaces are significantly increased, thus associating them with the device's overheating state. The fully connected layer maps the encoder output to the probability of a preset category, and the Softmax activation function ensures the probability sum is 1, selecting the category with the highest probability as the result. Classification result verification is achieved through confidence level screening. When the confidence level is < 0.8, the last 5 frames of historical data are retrieved for a second judgment. If the historical data shows no anomalies, the result is corrected to normal, significantly reducing the false positive rate.

[0064] An environmental monitoring report generation and tiered early warning mechanism enables efficient output of monitoring results and rapid risk response. Report content is tailored to operational needs, with monitoring time accurate to the second for easy tracking of anomaly occurrences. Clearly marked monitoring areas allow for quick on-site location. Key characteristic data, such as anomaly area coordinates and maximum temperature values, provide accurate references for operations and maintenance. Historical comparison data helps determine if anomalies are sporadic; for example, if similar anomalies occur three times in the past hour, potential equipment malfunctions need to be investigated. Output formats support both local and remote access. Local PDF storage facilitates long-term archiving and review, while remote API push allows operations and maintenance personnel to receive reports in real-time without on-site viewing. Tiered early warnings match response measures according to risk: low-risk warnings only provide local alerts to avoid interference; medium-risk warnings push photos and handling suggestions to accelerate processing; and high-risk warnings link on-site equipment to prevent escalation. The warning cancellation logic balances automation and manual intervention: three consecutive normal frames indicate anomaly cancellation, while manual confirmation is suitable for scenarios requiring manual repair. After cancellation, information is recorded in the report, forming a complete detection-early warning-handling closed loop.

[0065] Its beneficial effects lie in the fact that by fusing RGB and temperature images and standardizing them to achieve synergy between visual and physical information, it can accurately locate anomalies and distinguish interference, improving the accuracy of environmental detection compared to traditional single-modality methods. Employing an improved EfficientNet + bidirectional LSTM to extract spatiotemporal features, it can statically locate anomalies and dynamically track trends, predicting severe anomalies 5-10 minutes in advance, reducing the probability of accidents. Through an adaptive attention mechanism, it dynamically adjusts feature weights, significantly improving scene adaptability. Based on Transformer classification and a three-level early warning system, it improves operational efficiency and saves manpower and resources. It can also generate structured reports to facilitate data backtracking and model optimization, driving enterprises to shift from passive response to proactive prevention and improving environmental management levels.

[0066] Please see Figure 2 In image recognition-based environmental detection and identification methods, the acquisition of multimodal image data composed of environmental images and environmental temperature distribution images includes the following steps:

[0067] Step 201: Acquire environmental images and environmental temperature distribution images through sensors, remove noise from the environmental images using Gaussian filtering, enhance the contrast of the environmental images through adaptive histogram equalization, and then normalize the image size to obtain a preprocessed environmental image.

[0068] Step 202: The ambient temperature distribution image is calibrated. After removing isolated noise points in the heat map by median filtering, the temperature data is mapped to a grayscale image to obtain a temperature grayscale image.

[0069] Step 203: Based on the timestamp at the time of acquisition, spatially align the preprocessed environmental image with the temperature grayscale image to obtain multimodal image data.

[0070] The above describes embodiments of the image recognition-based environment detection and identification method of the present invention. Please refer to [link / reference]. Figure 3 In an image recognition-based environmental detection and identification system, the system includes the following modules:

[0071] The image data acquisition module is used to acquire multimodal image data consisting of environmental images and environmental temperature distribution images;

[0072] The image feature extraction module is used to build an improved convolutional neural network based on EfficientNet as the backbone to extract the spatial static features of multimodal image data and to extract the temporal dynamic features of multimodal image data using an RNN recurrent neural network.

[0073] The image feature fusion module is used to adjust the weights of spatial static features and temporal dynamic features through an adaptive cross-modal attention mechanism, and then fuse them to obtain fused feature data.

[0074] The environmental detection and identification module is used to input the fused feature vector into the Transformer-based environmental state classifier to identify the environmental state, generate an environmental detection report based on the identification results, and trigger the corresponding environmental early warning mechanism.

[0075] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An environment detection and recognition method based on image recognition, characterized in that, The environment detection identification method comprises the following steps: Collecting multi-modal image data composed of environment images and environment temperature distribution images; Taking EfficientNet-B4 as a basis backbone network, a BiFPN bidirectional feature pyramid network module is added in the transition layer of feature extraction and feature fusion; the features of different levels of EfficientNet are fused through the upper and lower bidirectional paths, wherein the shallow layer feature captures the detail information related to edge and texture, and the deep layer feature captures the global information related to object category and scene semantics; the preprocessed environment image and the temperature gray image are input into the improved EfficientNet through channel splicing, and the network gradually extracts features through convolution layers, batch normalization layers and Swish activation functions, and outputs a spatial static feature vector; the output spatial static feature is subjected to L2 regularization processing, and at the same time, the key features are strengthened and the interference of irrelevant background features is suppressed through an SE attention module, to obtain a spatial static feature; according to the time sequence of collection, the spatial static feature vectors of continuous N frames are combined to form a time sequence data with a dimension of [N, 2048]; a bidirectional LSTM is selected as a time feature extraction network, and 2 layers of hidden layers are set to dynamically adjust the information transmission in the time sequence through the forget gate, input gate and output gate; the output of the last time step of the bidirectional LSTM is dimensionally compressed to obtain a time dynamic feature vector, which at least includes a temperature change rate and an abnormal state duration; The weights of the spatial static feature and the time dynamic feature are adjusted through an adaptive cross-modal attention mechanism, and are fused to obtain fusion feature data; The fusion feature vector is input into an environment state classifier based on Transformer to identify the environment state, and an environment detection report is generated according to the identification result and a corresponding environment warning mechanism is triggered.

2. The image recognition-based environment detection identification method of claim 1, wherein, The multi-modal image data composed of environment images and environment temperature distribution images is collected, comprising: The environment images and the environment temperature distribution images are obtained through sensors, the noise of the environment images is removed by Gaussian filtering, the contrast of the environment images is enhanced by adaptive histogram equalization, and then the image size is normalized to obtain preprocessed environment images; The environment temperature distribution images are calibrated, the isolated noise points in the thermal map are removed by median filtering, and then the temperature data is mapped into a gray image to obtain a temperature gray image; Based on the time stamp at the time of collection, the preprocessed environment image and the temperature gray image are spatially aligned to obtain multi-modal image data.

3. The image recognition-based environment detection identification method of claim 1, wherein, The weights of the spatial static feature and the time dynamic feature are adjusted through an adaptive cross-modal attention mechanism, and are fused to obtain fusion feature data, comprising: The spatial static feature vector and the time dynamic feature vector are unified in dimension through a full connection layer, and the cosine similarity of the two types of features is calculated; A weight prediction network is constructed by taking 2 layers of full connection layers and Sigmoid activation functions, the input is the modal similarity and the preliminary feature of the current frame environment, and the output is two weight values, according to which the feature fusion is performed to obtain fusion feature data.

4. The image recognition-based environment detection identification method of claim 1, wherein, The fused feature vector is input into a Transformer-based environment state classifier to identify an environment state, an environment detection report is generated according to an identification result, and a corresponding environment early warning mechanism is triggered, including: Taking a 6-layer encoder as the core and setting 8 heads for the multi-head self-attention module, a Transformer classifier is constructed; The fused feature vector is taken as the input of the Transformer encoder, the semantic correlation between features is mined through the self-attention module, the environment state probability distribution is output through the full connection layer and the Softmax activation function, and the class with the highest probability is selected as the environment state identification result.

5. An environment detection and recognition system based on image recognition, characterized in that The environment detection and identification system comprises the following modules: An image data acquisition module is configured to acquire multi-modal image data composed of an environment image and an environment temperature distribution image; An image feature extraction module is configured to take EfficientNet-B4 as a basic backbone network, add a BiFPN bidirectional feature pyramid network module in a transition layer between feature extraction and feature fusion, fuse features at different levels of the EfficientNet through an up-down bidirectional path, wherein shallow features capture edge and texture related detailed information, and deep features capture object class and scene semantic related global information, input the preprocessed environment image and the temperature grayscale image into the improved EfficientNet through a channel splicing manner, and gradually extract features through a convolution layer, a batch normalization layer and a Swish activation function, and output a spatial static feature vector; the output spatial static feature is subjected to L2 regularization processing, and meanwhile, key features are strengthened and irrelevant background features are inhibited through an SE attention module, so as to obtain a spatial static feature; In a time sequence according to a collection time sequence, spatial static feature vectors of N continuous frames are combined to form time sequence data with a dimension of [N, 2048]; a bidirectional LSTM is selected as a time feature extraction network, 2 layers of hidden layers are set, information in the time sequence is dynamically adjusted through a forgetting gate, an input gate and an output gate; the output of the last time step of the bidirectional LSTM is subjected to dimension compression to obtain a time dynamic feature vector, which at least includes a temperature change rate and an abnormal state duration; An image feature fusion module is configured to adjust the weights of the spatial static feature and the time dynamic feature through an adaptive cross-modal attention mechanism, and fuse the spatial static feature and the time dynamic feature to obtain fused feature data; An environment detection and identification module is configured to input the fused feature vector into a Transformer-based environment state classifier to identify an environment state, generate an environment detection report according to an identification result, and trigger a corresponding environment early warning mechanism.

6. The image recognition based environment detection recognition system of claim 5, wherein, The image feature fusion module comprises the following sub-modules: A calculation sub-module is configured to calculate the cosine similarity of the spatial static feature vector and the time dynamic feature vector after the two types of features are unified in dimension through a full connection layer; A fusion sub-module is configured to take 2 layers of full connection layers and a Sigmoid activation function to construct a weight prediction network, take the modal similarity and the current frame environment preliminary feature as the input, and take two weight values as the output, so as to fuse the features according to the weight values to obtain fused feature data.

7. The image recognition based environment detection recognition system of claim 5, wherein, The environment detection and recognition module comprises the following sub-modules: A construction sub-module, configured to construct a Transformer classifier with a 6-layer encoder as a core and 8 heads of a multi-head self-attention module; An output sub-module, configured to take the fusion feature vector as an input of a Transformer encoder, mine semantic correlations between features through a self-attention module, output an environment state probability distribution through a fully connected layer and a Softmax activation function, and select a class with the highest probability as an environment state recognition result.

Images