Pedestrian smoking state recognition system and method based on bimodal

Abstract

The invention provides a bimodal-based pedestrian smoking state recognition system and method. The system comprises an image acquisition module, a data preprocessing module, a feature extraction and fusion module and a smoking detection module. According to the method, the high-precision detection and judgment of the smoking behavior are realized by fusing the complementary sensing characteristics of the visible light and the infrared image. According to the method, a double-flow collaborative deep neural network structure is adopted, semantic features of a visible light image and temperature features of an infrared image are extracted respectively, a cross-modal channel attention mechanism is introduced into a feature layer, and self-adaptive fusion and space focusing of multi-modal information are achieved. According to the method, accurate geometric calibration is not needed, and end-to-end smoking detection and identification can be realized only by ensuring collection time synchronization.

Description

TECHNICAL FIELD

[0001] The present application relates to the field, and particularly to a pedestrian smoking state recognition system and method based on a dual mode. BACKGROUND

[0002] With the rapid development of power industry inspection robots and intelligent monitoring technology, the intelligent level of production safety and behavior recognition is continuously improved. In high-risk places such as power, public safety and industrial production, smoking behavior not only violates safety management specifications, but also may cause fire, explosion and other major safety accidents. Therefore, how to realize automatic and accurate recognition of personnel smoking behavior in complex environment has become an important research direction in the field of intelligent safety monitoring and intelligent inspection.

[0003] The existing smoking detection method mainly relies on visible light cameras to collect images, and identifies the cigarette butt or smoking posture through target detection or action recognition algorithm. However, visible light imaging is easily affected by factors such as light change, occlusion, shooting angle and reflection, resulting in a significant decrease in recognition accuracy in low light, night or complex background environment. At the same time, the small size and unstable brightness of the cigarette butt cause frequent false detection and missed detection, which is difficult to meet the actual application requirements.

[0004] Infrared thermal imaging technology can effectively capture heat source information, is not limited by light conditions, and can detect the temperature distribution characteristics of human body and heat-emitting objects at night or in dim environment. Using infrared images can easily locate the high temperature area of the cigarette butt and the mouth, providing new judgment basis for smoking behavior recognition. However, infrared images lack detailed texture and shape information, making it difficult to accurately distinguish human posture and hand movement, and single use is also prone to misjudgment. SUMMARY

[0005] The present application provides a pedestrian smoking state recognition system and method based on a dual mode, which does not require complex geometric calibration and only needs to ensure synchronization at the time of acquisition. The system extracts human key points, gestures and mouth region features through the visible light branch, extracts temperature distribution and high temperature point information through the infrared branch, realizes bidirectional fusion and enhancement at the feature layer, so that the model can adaptively focus on the "hand-mouth-high temperature point" area, thereby realizing high-precision and robust recognition of smoking behavior.

[0006] The present application provides a pedestrian smoking state recognition system based on a dual mode, which includes an image acquisition module, a data preprocessing module, a feature extraction and fusion module, and a smoking detection module.

[0007] Image acquisition module: the inspection robot synchronously acquires scene image data during task execution, and adds a timestamp for each frame of image. The system matches the synchronous frame according to the timestamp to ensure that the network input is the scene data at the same time, realizing the spatiotemporal consistency of the mode; Data preprocessing module: size adjustment and normalization operation are performed on visible light image and infrared image; Feature extraction and fusion module: a double-flow deep neural network is used to extract features from visible light and infrared images respectively, the visible light branch focuses on extracting semantic information of human key points, hand and mouth regions, and the infrared branch focuses on extracting temperature distribution and high temperature point features; Smoking detection module: after receiving the fused features, the module performs multi-layer convolution operation and detection head reasoning to output candidate detection results.

[0008] Further improvement, the scene image data collected by the image acquisition module includes visible light and infrared images, which are synchronously collected by a visible light camera and an infrared thermal imager respectively.

[0009] The application also provides a pedestrian smoking state recognition method based on double modalities, which comprises the following steps: Step 1) image acquisition; Step 1.1) synchronously collect image data of the target scene by installing a patrol robot, and realize frame-level alignment through a unified time reference, and add a time stamp to each frame of data to ensure that different modal images correspond to the same time scene; Step 1.2) the system establishes an image cache queue according to the time stamp information, and selects synchronous frames to input the network using the nearest matching strategy; Step 2) data preprocessing; Step 2.1) size adjustment and pixel normalization are performed on the collected visible light and infrared images; Step 2.2) temperature mapping and pseudo-color enhancement are performed on the infrared image to highlight the thermal features of the smoking area, and noise suppression and abnormal frame rejection are performed on the two types of images; Step 3) feature extraction and fusion; Step 3.1) a double-flow deep neural network structure is used to extract features from visible light and infrared images respectively; the visible light branch focuses on extracting semantic features of human key points, hands and mouth regions; the infrared branch focuses on extracting temperature distribution and high temperature point features; Step 3.2) cross-modal attention mechanism is introduced at the feature layer to perform bidirectional alignment and weighted fusion of the feature map; the system calculates the attention weight according to the correlation between the features of the two modalities, dynamically adjusts the fusion ratio, and realizes adaptive focusing and enhancement of multi-modal information; Step 4) smoking detection and determination: Step 4.1) the two features obtained in step 3) are fused, and a convolution layer is used to obtain the fused features, and then the features are sent to the detection head module; Step 4.2) Perform non-maximum suppression (NMS) on the above prediction results to remove redundant boxes, and output the final detection result according to the confidence threshold. If smoking is detected, it means that someone is smoking in the current frame.

[0010] As a further improvement, the unified time reference mentioned in step 1.1) is synchronized using a hardware trigger signal or a system clock.

[0011] As a further improvement, in step 1.2), if there is frame loss or delay during the selection of synchronization frame pairs for input to the network, the system will automatically interpolate or skip frames to fill in the gaps, ensuring the continuity and stability of data input.

[0012] Further improvements are made to the normalization process described in step 2.1): the input resolution is unified to 640x640, the difference in the field of view is eliminated by unifying the resolution and scaling, and the normalization operation of subtracting the mean and dividing the variance is performed on the image pixels to improve the consistency of the distribution of network input data and reduce the impact of changes in illumination.

[0013] Further improvements are made to the feature extraction process for visible light and infrared images described in step 3.1), which is as follows: Both the visible light and infrared feature extraction branches adopt the ResNet network structure. Each branch contains four sub-modules, and each sub-module contains a small feature extraction unit, namely a convolutional layer, a batch normalization (BN) layer, and an activation layer. The sub-modules are downsampled through convolutional layers with strides. First, a 1x1 convolution is used for dimensionality upsampling, followed by a 3x3 convolution with a stride of 2 for downsampling, and then a 1x1 convolution for dimensionality reduction. The final feature map size is halved.

[0014] Further improvements are made, and the specific process of bidirectional alignment and weighted fusion in step 3.2) is as follows: global average pooling is performed on the visible light and infrared feature maps respectively, and then two one-dimensional vectors are obtained. Then, a concat operation is performed to obtain the channel attention description. Next, two one-dimensional vectors of channel attention are obtained through two FC layers. Finally, the original feature map and the obtained channel attention are multiplied channel by channel to obtain a new feature map.

[0015] The beneficial effects of this invention are as follows: 1. This method comprehensively utilizes the complementary characteristics of visible light and infrared images to construct a two-stream deep neural network structure to extract semantic and temperature features from both modalities. It then achieves adaptive collaboration and spatial focusing between features through a cross-modal channel attention fusion mechanism. This method requires no complex geometric calibration, only ensuring synchronization during data acquisition.

[0016] 2. The system extracts key human body features, gestures, and mouth area features through the visible light branch, and extracts temperature distribution and high-temperature point information through the infrared branch. It achieves bidirectional fusion and enhancement at the feature layer, enabling the model to adaptively focus on the "hand-mouth-high-temperature point" area, thereby achieving high-precision and robust recognition of smoking behavior. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the software execution process. Figure 2 This is a diagram of the dual-modal smoking recognition network structure. Figure 3 This is a structural diagram of the Cross-Model Attention module; Figure 4 This is a structural diagram of the Conv2D stride (downsampling) module; Figure 5 This is a schematic diagram of the detection head structure; Figure 6 An illustrative image taken by the inspection robot. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The purpose of this invention is to provide a smoking detection system based on dual-modal visual fusion, which integrates the structural semantics of visible light and the temperature information of infrared light. It exhibits good illumination robustness and behavior judgment capabilities, and can stably identify smoking behavior in complex scenes. Specifically, it includes the following modules: Image acquisition module: Scene image data is simultaneously acquired using a visible light camera and an infrared thermal imager. During task execution, the inspection robot simultaneously acquires visible light and infrared images and adds a timestamp to each frame. The system matches synchronization frames based on the timestamps, ensuring that the network input is scene data from the same moment, achieving modal spatiotemporal consistency.

[0021] Data preprocessing module: The main focus is on resizing and normalizing visible light and infrared images. First, the two images are resized and padded to ensure consistent spatial dimensions for the input network. Then, the image pixels are normalized by subtracting the mean and dividing by the variance to obtain the effective input for the network.

[0022] Feature extraction and fusion module: A two-stream deep neural network was used to extract features from visible light and infrared images respectively. The visible light branch focused on extracting semantic information from key points of the human body and the hand and mouth regions, while the infrared branch focused on extracting temperature distribution and high-temperature features. A cross-modal attention mechanism was introduced during the feature extraction process to achieve adaptive fusion and spatial focusing of multimodal information through alignment and weighting. The fused joint features were then further fed into the smoking detection module.

[0023] Smoking detection module: After receiving the fused features, this module performs multi-layer convolutional operations and detection head inference to output candidate detection results. The system performs maximum suppression processing on the detection results to remove duplicate detection boxes, and finally outputs the detection box and judgment result of the smoking target, realizing end-to-end smoking behavior recognition.

[0024] The flowchart of the smoking detection based on dual-modal visual fusion provided by this invention is as follows: Figure 1 As shown, the specific steps include: Step 1: Image Acquisition Step 1.1: Simultaneously acquire image data of the target scene using a visible light camera and an infrared thermal imager mounted on the inspection robot. The two types of sensors achieve frame-level alignment through a unified time reference (such as a hardware trigger signal or system clock synchronization), and a timestamp is added to each frame of data to ensure that different modal images correspond to the same scene at the same time. This step solves the time offset problem in multimodal data acquisition and ensures the synchronization of the fused input.

[0025] Step 1.2: The system establishes an image buffer queue based on timestamp information and selects synchronized frame pairs for input to the network using a nearest-nearest-match strategy. If frame loss or delay occurs, the system automatically interpolates or skips frames to complete the data, ensuring the continuity and stability of the data input. This step improves the robustness of the system's acquisition in dynamic scenes.

[0026] Step 2: Data Preprocessing Step 2.1: Resize and normalize the pixels of the acquired visible light and infrared images. The input resolution is standardized to 640x640. By standardizing the resolution and scaling, differences in the field of view are eliminated. Furthermore, normalization operations (mean subtraction and variance division) are performed on the image pixels to improve the consistency of the network input data distribution and reduce the impact of lighting variations.

[0027] Step 2.2: Perform temperature mapping and pseudo-color enhancement on the infrared image to highlight the thermal characteristics of the smoking area. Figure 6 The displayed data is infrared data enhanced with pseudo-color. Noise suppression and outlier frame removal are performed on both types of images to ensure the quality of the network input data. This step improves the saliency representation and detection stability of the infrared channel.

[0028] Step 3: Feature Extraction and Fusion Step 3.1: A dual-stream deep neural network structure is used to extract features from visible light and infrared images respectively. The visible light branch focuses on extracting semantic features of key points on the human body, as well as areas such as the hands and mouth; the infrared branch focuses on extracting temperature distribution and high-temperature features. This design achieves complementary representation of structural and thermal information. The overall structure is as follows: Figure 2 As shown, both the visible light and infrared feature extraction branches employ the ResNet network structure. Each branch contains four sub-modules, each containing a small feature extraction unit consisting of a convolutional layer, a batch normalization (BN) layer, and an activation layer. Downsampling between sub-modules is achieved through strided convolutional layers, as shown in the diagram. Figure 4 As shown, the dimensionality is first increased by 1x1 convolution, then downsampled by 3x3 convolution with a stride of 2, and then reduced by 1x1 convolution, resulting in a feature map size that is halved.

[0029] Step 3.2: Introduce a cross-modal attention mechanism at the feature layer to perform bidirectional alignment and weighted fusion of feature maps. The system calculates attention weights based on the correlation between intermodal features and dynamically adjusts the fusion ratio to achieve adaptive focusing and enhancement of multimodal information. For example... Figure 3 As shown, global average pooling is performed on the visible light and infrared feature maps respectively, resulting in two one-dimensional vectors. A concat operation is then performed to obtain channel attention descriptions. Next, two fully connected (FC) layers are used to obtain two one-dimensional vectors for the channel attention. Finally, the original feature map and the obtained channel attention are multiplied channel by channel to obtain a new feature map. This step improves the detection's ability to focus on key regions (such as the mouth and hands).

[0030] Step 4: Smoking detection and judgment: Step 4.1: Fuse the two feature streams obtained in Step 3, pass them through a convolutional layer to obtain the fused feature, and then feed the feature into the detection head module. The detection head module is as follows:Figure 5 As shown, the detection head performs a convolution operation on the input feature map, with an output size of 64*64*54. The 54 channels represent 9 sets of predicted values ​​for each pixel. Each set represents a possible detection result. Each set has 6 outputs: the first output indicates whether it is a foreground score, the next 4 represent the predicted bounding box value, and the last indicates whether it is a smoking box score. If both the foreground score and the smoking score exceed a set threshold, the predicted bounding box is considered a smoking box.

[0031] Step 4.2: Perform non-maximum suppression (NMS) on the above prediction results to remove redundant bounding boxes, and output the final detection result based on the confidence threshold. Maximum suppression is used to remove redundant and repetitive predictions. After these operations, if smoking is detected, it indicates that someone is smoking in the current frame.

[0032] In a personnel violation detection project conducted at a substation in Changzhou, smoking violations were previously detected using only visible light, with a detection rate of only 50% and a false detection rate as high as 40%. This patented method utilizes a robot equipped with a dual-light gimbal, which includes two cameras: a visible light camera and an infrared thermal imager. The visible light camera captures visible light images, while the infrared thermal imager captures infrared images. During routine inspections, both cameras capture images and infrared data simultaneously. First, the robot collected approximately 20,000 data samples, including about 5,000 smoking data images. These included both simulated smoking behaviors and actual inspection photos. Then, these data were labeled, primarily with smoking frames. After labeling, the model was trained. Once trained, the model was deployed on the robot and operated during inspections within the substation. During this period, the robot collected approximately 10,000 images, inferred from these images, and obtained approximately 200 smoking images. After comparison with the actual values, a detection rate of 90% and a false detection rate of approximately 10% were obtained, demonstrating the feasibility of this method.

[0033] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, for the device embodiments, the above descriptions are merely preferred embodiments of the present invention. Since they are fundamentally similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention, without departing from the principle of the present invention, should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A pedestrian smoking status recognition system based on dual-modality, characterized in that: It includes an image acquisition module, a data preprocessing module, a feature extraction and fusion module, and a smoking detection module; Image acquisition module: During the operation of the inspection robot, scene image data is collected synchronously and a timestamp is added to each frame of the image. The system matches the synchronous frames according to the timestamp to ensure that the network input is scene data at the same time and achieves modal spatiotemporal consistency. Data preprocessing module: performs size adjustment and normalization operations on visible light and infrared images; Feature extraction and fusion module: The module uses a dual-stream deep neural network to extract features from visible light and infrared images respectively. The visible light branch focuses on extracting semantic information of key points of the human body, hand and mouth regions, while the infrared branch focuses on extracting temperature distribution and high-temperature point features. Smoking detection module: After receiving the fused features, this module performs multi-layer convolution operations and inference with the detection head to output candidate detection results.

2. The pedestrian smoking status recognition system based on dual-modality according to claim 1, characterized in that: The scene image data acquired by the image acquisition module includes visible light and infrared images, which are acquired synchronously by a visible light camera and an infrared thermal imager, respectively.

3. A method for recognizing pedestrian smoking status based on dual-modality, characterized in that... Includes the following steps: Step 1) Image acquisition; Step 1.1) By simultaneously collecting images of the target scene on the inspection robot, the two types of sensors achieve frame-level alignment through a unified time reference, and add a timestamp to each frame of data to ensure that different modal images correspond to the same scene at the same time. Step 1.2) The system establishes an image buffer queue based on timestamp information and selects a pair of synchronized frames for input to the network using a nearest-match strategy; Step 2) Data preprocessing; Step 2.1) Resize and normalize the pixels of the acquired visible light and infrared images; Step 2.2) Perform temperature mapping and pseudo-color enhancement on the infrared image to highlight the thermal characteristics of the smoking area, and at the same time perform noise suppression and abnormal frame removal on both types of images; Step 3) Feature extraction and fusion; Step 3.1) A dual-stream deep neural network structure is used to extract features from visible light and infrared images respectively; the visible light branch focuses on extracting semantic features of key points of the human body, hands, mouth and other regions; the infrared branch focuses on extracting temperature distribution and high temperature point features. Step 3.2) Introduce a cross-modal attention mechanism at the feature layer to perform bidirectional alignment and weighted fusion of feature maps; the system calculates attention weights based on the correlation between features between modalities and dynamically adjusts the fusion ratio to achieve adaptive focusing and enhancement of multimodal information; Step 4) Smoking detection and judgment: Step 4.1) The two features obtained in step 3) are fused, and the fused features are obtained by passing them through a convolutional layer. The features are then sent to the detection head module. Step 4.2) Perform non-maximum suppression (NMS) on the above prediction results to remove redundant boxes, and output the final detection result according to the confidence threshold. If smoking is detected, it means that someone is smoking in the current frame.

4. The pedestrian smoking state recognition method based on bimodality according to claim 3, characterized in that: Step 1.1) The unified time reference is synchronized using a hardware trigger signal or a system clock.

5. The pedestrian smoking state recognition method based on bimodality according to claim 3, characterized in that: In step 1.2), during the process of selecting synchronization frame pairs for input to the network, if there are frame losses or delays, the system will automatically interpolate or skip frames to complete the data input, ensuring the continuity and stability of the data input.

6. The pedestrian smoking state recognition method based on bimodality according to claim 3, characterized in that: The normalization process in step 2.1) specifically involves: unifying the input resolution to 640x640, eliminating field-of-view differences through unified resolution and scaling, and performing normalization operations on image pixels by subtracting the mean and dividing the variance to improve the consistency of the network input data distribution and reduce the impact of illumination changes.

7. The pedestrian smoking state recognition method based on bimodality according to claim 3, characterized in that: Step 3.1) describes the feature extraction process for visible light and infrared images as follows: Both the visible light and infrared feature extraction branches adopt the ResNet network structure. Each branch contains four sub-modules, and each sub-module contains a small feature extraction unit, namely a convolutional layer, a batch normalization (BN) layer, and an activation layer. The sub-modules are downsampled through convolutional layers with strides. First, a 1x1 convolution is used for dimensionality upsampling, then a 3x3 convolution with a stride of 2 is used for downsampling, and then a 1x1 convolution is used for dimensionality reduction. The final feature map size is halved.

8. The pedestrian smoking state recognition method based on dual-modality according to claim 3, characterized in that: Step 3.2) The specific process of bidirectional alignment and weighted fusion is as follows: global average pooling is performed on the visible light and infrared feature maps respectively, and then two one-dimensional vectors are obtained. Then, a concat operation is performed to obtain the channel attention description. Next, two one-dimensional vectors of channel attention are obtained through two FC layers. Finally, the original feature map and the obtained channel attention are multiplied channel by channel to obtain a new feature map.

Images