Smoking detection system based on multi-modal data

Abstract

The invention discloses a smoking detection system based on multi-modal data, which belongs to the technical field of smoking behavior detection and comprises a sensing data capture module, a data transmission module, a smoking multi-modal judgment module, a smoking comprehensive judgment module, a smoking alarm module and the like. According to the method, the probability arrays of the multiple modes are obtained based on the collected multi-mode sensing data according to the multi-mode sensing data, then the SLIA algorithm is used for correcting the probability arrays to obtain the corrected multi-mode probability data, and then the final smoking probability is obtained through calculation based on the corrected multi-mode probability data. And comparing the final smoking probability with a preset smoking probability threshold to obtain a smoking behavior comprehensive judgment result, so that the smoking behavior can be detected through multi-modal data, and the detection accuracy of the smoking behavior is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smoking behavior detection, and in particular to a smoking detection system based on multimodal data. Background Art

[0002] Existing smoking behavior detection technologies mainly use a single detection method. For example, vision-based smoking recognition monitors and analyzes videos in a designated surveillance area to automatically identify and warn of smoking behavior. Sensor-based smoking recognition uses sensors to detect specific signals or changes generated when smoking. These sensors can be actively worn, such as radio frequency proximity sensors, inertial measurement units (IMUs), and respiratory sensors, or they can be passive detection. For example, smoking recognition based on Wi-Fi devices uses changes in Wi-Fi signals to identify smoking behavior. The smoke or movements generated when smoking may affect the propagation path and strength of the Wi-Fi signal, which can be captured and analyzed by the Wi-Fi device.

[0003] Although the existing smoking behavior detection technology has made significant progress in many aspects, it still has some defects: 1. Limitations of target detection: Due to the resolution and shooting distance of surveillance cameras, cigarettes are more likely to exist in the form of sticks in the image, which makes it easy to misdetect sticks. In addition, the shooting angle and the different smoking postures of each person may cause the cigarette target to be blocked, which increases the difficulty of detection. 2. High missed detection rate: A single detection method may not cover all possible smoking scenarios. For example, smoking behavior at a long distance, under occlusion or fast movement may not be accurately detected. 3. Poor environmental adaptability: The smoking detection environment may change with time, weather, season and other factors, and a single detection method may not be able to adapt well to these changes, resulting in reduced detection effect.

[0004] The above problems need to be solved urgently. To this end, a smoking detection system based on multimodal data is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to fuse multimodal data to achieve more accurate smoking behavior detection, and provides a smoking detection system based on multimodal data.

[0006] The present invention solves the above technical problems through the following technical solutions, which include a perception data capture module, a data transmission module, a smoking multimodal judgment module, a smoking comprehensive judgment module and a smoking alarm module;

[0007] The perception data capture module is used to collect multimodal perception data of the detection area;

[0008] The data transmission module is used to transmit the collected multimodal sensing data to the smoking multimodal judgment module;

[0009] The smoking multimodal judgment module is used to process the multimodal perception data to obtain a probability array of multiple modes;

[0010] The smoking comprehensive judgment module is used to correct each probability array using the SLIA algorithm to obtain corrected multimodal probability data, and then calculate the final smoking probability based on the corrected multimodal probability data, compare the final smoking probability with a preset smoking probability threshold, obtain a comprehensive judgment result of smoking behavior, and select whether to generate smoking warning information according to the comprehensive judgment result;

[0011] The smoking alarm module is used to receive smoking alarm information and warn management personnel and parties concerned.

[0012] Furthermore, in the perception data capturing module, the multi-module perception data includes infrared thermal images, personnel images and smoke concentration data of the detection area.

[0013] Furthermore, in the smoking multi-modal judgment module, the infrared thermal image, the person image and the smoke concentration data are processed respectively to obtain probability arrays of the three modes.

[0014] Furthermore, in the smoking multi-modal judgment module, the process of processing the infrared thermal image is as follows:

[0015] S11: dividing the infrared thermal image into N pixels, and obtaining the average temperature of the pixel by calculating the infrared radiation energy of the N pixels;

[0016] S12: Setting the combustion point temperature thresholds to θ1 and θ2, and filtering out the pixels that meet the thresholds; θ1 is the lower limit of the threshold, and θ2 is the upper limit of the threshold;

[0017] S13: Divide the adjacent pixel points that meet the threshold into the same group to obtain multiple pixel groups Z 1 , Z 2 ···Z n ; Z s1 , Z s2 ···Z sn The corresponding Z 1 , Z 2 ···Z n The number of pixels included, where n ≥ 1;

[0018] S14: Set standard value Z sb , Z sb is the pixel size of the standard burning point;

[0019] S15: Calculate the first smoking probability value Rn, when Z si When Rn is equal to 0, si When it is not equal to 0, execute the following formula:

[0020]

[0021] S16: Obtain a first probability array by calculating the plurality of data: [R 1 , R 2 , R 3 …R (n-1) , R n ].

[0022] Furthermore, in the smoking multimodal judgment module, the process of processing the person image is specifically as follows:

[0023] S21: extracting frames from the video stream to be detected to obtain an initial image group to be detected;

[0024] S22: performing image preprocessing on the initial image to be detected helps to extract cigarette features from the image to be detected;

[0025] S23: determining the pre-processed image containing the human face as the image to be detected;

[0026] S24: establishing an image coordinate system, setting the mouth as area A and the cigarette butt as area B through grayscale processing;

[0027] S25: Set the overlap time of regions A and B to M, where M follows a normal distribution;

[0028] S26: Calculate the second smoking probability value:

[0029]

[0030] Among them, μ is the location parameter and σ is the scale parameter;

[0031] S27: Obtain a second probability array by calculating the plurality of data: [P 1 , P 2 , P 3 …P (n-1) , P n ].

[0032] Furthermore, in step S22, the image preprocessing method includes one or more combinations of grayscale, binarization, and dilation processing.

[0033] Furthermore, in the smoking multi-modal judgment module, the process of processing the smoke concentration data is specifically as follows:

[0034] S31: By applying a DC voltage between two metal plate electrodes and placing a piece of isotope americium-241 nearby; americium-241 can emit a small amount of alpha rays, ionizing the nearby air;

[0035] S32: Record the ion current in the form of discrete data to obtain the following array [I 1 ,I 2 ,I 3 ···I n ], where smoke concentration data is represented by ion current data;

[0036] S33: Set the smoking smoke detection threshold to 1 a ,I b , where I a is the lower threshold, I b is the upper threshold value;

[0037] S34: Calculate the third smoking probability value:

[0038]

[0039] Among them, I a ≠I b , φ is the correction parameter;

[0040] S35: Obtain a third probability array by calculating the multiple data: [S 1 , S 2 , S 3 …S (n-1) , S n ].

[0041] Furthermore, in the smoking comprehensive judgment module, the correction process of the SLIA algorithm is as follows:

[0042] S41: Initially, input the probability array corresponding to a certain mode and calculate its average value f(t);

[0043] S42: Find the difference between a single data and f(t), take the positive value, delete the n / 5 rounded partial data with the largest difference, and then find the average value f(t+1), and judge the positive difference between the two average values ​​f(t+1) and f(t). If it is less than the set value, output the average value f(t+1) to end the loop, otherwise continue to execute;

[0044] S43: When the number of output data is less than 1 / 10 of the number of initial data, the operation is terminated and the average value f(t+1) is output.

[0045] Furthermore, in the smoking comprehensive judgment module, the calculation formula of the final smoking probability ζ is as follows:

[0046] ζ=αf(r+1)+βf(p+1)+γf(s+1)

[0047] Among them, f(r+1), f(p+1), and f(s+1) are the average values ​​of the probability arrays of the three modes after correction by the SLIA algorithm, that is, the corrected multimodal probability data, and α, β, and γ are the corresponding correction proportional coefficients.

[0048] Furthermore, in the smoking comprehensive judgment module, when the final smoking probability ζ is greater than or equal to the preset smoking probability threshold ζ 预设 , it indicates that there is smoking behavior and a smoking warning message is generated; otherwise, it indicates that there is no smoking behavior and no smoking warning message is generated.

[0049] Compared with the prior art, the present invention has the following advantages: the smoking detection system based on multimodal data is based on the collected multimodal perception data, and obtains probability arrays of multiple modes according to the multimodal perception data, and then uses the SLIA algorithm to correct each probability array to obtain corrected multimodal probability data, and then calculates the final smoking probability based on the corrected multimodal probability data, compares the final smoking probability with a preset smoking probability threshold, and obtains a comprehensive judgment result of the smoking behavior, which can realize the detection of smoking behavior through multimodal data and improve the detection accuracy of smoking behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of a processing flow of a smoking detection system based on multimodal data in an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the correction process of the SLIA algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0053] like Figure 1 As shown, this embodiment provides a technical solution: a smoking detection system based on multimodal data, including a perception data capture module, a data transmission module, a smoking multimodal judgment module, a smoking comprehensive judgment module, and a smoking alarm module;

[0054] In this embodiment, the perception data capture module is used to collect multimodal perception data of the detection area;

[0055] More specifically, the multi-module perception data includes an infrared thermal image, a personnel image and smoke concentration data of the detection area; wherein the infrared thermal image contains possible cigarettes, and the personnel image contains a person's face and possible cigarettes.

[0056] In this embodiment, the data transmission module is used to transmit the collected multimodal sensing data to the data processing module;

[0057] In this embodiment, the smoking multimodal judgment module is used to process the multimodal perception data to obtain a probability array of multiple modes;

[0058] In this embodiment, the smoking comprehensive judgment module is used to correct each probability array using the SLIA algorithm to obtain corrected multimodal probability data, and then calculate the final smoking probability based on the corrected multimodal probability data, compare the final smoking probability with a preset smoking probability threshold, obtain a comprehensive judgment result of the smoking behavior, and select whether to generate a smoking warning message based on the comprehensive judgment result.

[0059] In this embodiment, the smoking alarm module is used to receive smoking alarm information and warn the management personnel and the parties concerned.

[0060] In this embodiment, in the smoking multi-modal judgment module, the infrared thermal image, the person image and the smoke concentration data are processed respectively to obtain probability arrays of the three modes.

[0061] More specifically, in the smoking multi-modal judgment module, the process of processing the infrared thermal image is as follows:

[0062] S11: dividing the infrared thermal image into N pixels, and obtaining the average temperature of the pixel by calculating the infrared radiation energy of the N pixels;

[0063] S12: Setting the combustion point temperature thresholds to θ1 and θ2, and filtering out the pixels that meet the thresholds; θ1 is the lower limit of the threshold, and θ2 is the upper limit of the threshold;

[0064] S13: Divide the adjacent pixels that meet the threshold into the same group to obtain several pixel groups Z 1 , Z 2 ···Z i (n≥1); Z s1 , Z s2 ···Z sn (n≥1) is the corresponding Z 1 , Z 2 ···Z n (n≥1) The number of pixels contained;

[0065] S14: Set standard value Z sb , Z sb is the pixel size of the standard burning point;

[0066] S15: Calculate the first smoking probability value Rn, when Z si When Rn is equal to 0, si When it is not equal to 0, execute the following formula:

[0067]

[0068] S16: Obtain a first probability array by calculating the plurality of data: [R 1 , R 2 , R 3 …R (n-1) , R n ].

[0069] It should be noted that in infrared thermal images, according to the Stefan-Boltzmann law, the higher the surface temperature of an object, the stronger the infrared radiation naturally radiated by its surface, and vice versa.

[0070] In this embodiment, in the smoking multi-modal judgment module, the process of processing the person image is as follows:

[0071] S21: Extract frames from the video stream to be detected to obtain an initial set of images to be detected:

[0072] S22: performing image preprocessing on the initial image to be detected helps to extract cigarette features from the image to be detected; the image preprocessing method includes at least one or more combinations of grayscale, binarization, and expansion processing;

[0073] S23: determining the pre-processed image containing the facial image of a human face as the image to be detected;

[0074] S24: establishing an image coordinate system, setting the mouth as area A and the cigarette butt as area B through grayscale processing;

[0075] S25: setting the overlap time of regions A and B (the sum of the number of image frames overlapped between regions A and B) to M, where M follows a normal distribution;

[0076] S26: Calculate the second smoking probability value:

[0077]

[0078] Among them, μ is the location parameter, σ is the scale parameter, and the specific values ​​of μ and σ can be obtained according to the training results;

[0079] S27: Obtain a second probability array by calculating the plurality of data: [P 1 , P2 , P 3 …P (n-1) , P n ].

[0080] In this embodiment, in the smoking multi-modal judgment module, the process of processing the smoke concentration data is as follows:

[0081] S31: By applying a DC voltage between two metal plate electrodes and placing a small piece of isotope americium-241 nearby; americium-241 can emit a small amount of alpha rays, ionizing the air nearby. Under the action of the DC voltage between the metal plate electrodes, an ion current will be generated in the air;

[0082] S32: Record the ion current in the form of discrete data to obtain the following array [I 1 ,I 2 ,I 3 ···I n ];

[0083] S33: Set the smoking smoke detection threshold to 1 a ,I b , where I a is the lower threshold, I b is the upper threshold value;

[0084] S34: Calculate the third smoking probability value:

[0085]

[0086] Among them, I a ≠I b , φ is the correction parameter;

[0087] S35: Obtain a third probability array by calculating the multiple data: [S 1 , S 2 , S 3 …S (n-1) , S n ].

[0088] It should be noted that ionic photosensitive elements use smoke particles to detect changes in ionic current in the air. When smoke invades, the smoke particles will absorb ions, causing current changes; resistive photosensitive elements detect the presence of smoke by measuring changes in light after smoke invades.

[0089] like Figure 2 As shown, in this embodiment, in the smoking comprehensive judgment module, the correction process of the SLIA algorithm is as follows:

[0090] Initially, input the probability array corresponding to a certain mode and calculate its average value f(t);

[0091] Calculate the difference between a single data and f(t), take the positive value, delete the n / 5 rounded data with the largest difference, and then calculate its average value f(t+1). Determine the positive difference between the two average values ​​f(t+1) and f(t). If it is less than the set value, indicating that the average values ​​before and after deletion deviate less within the preset range, then output the average value f(t+1) and end the loop. Otherwise, continue to execute.

[0092] When the number of output data is less than 1 / 10 of the number of initial data, the SLIA operation is terminated and the average value f(t+1) is output.

[0093] In this embodiment, in the smoking comprehensive judgment module, the calculation formula of the final smoking probability ζ is as follows:

[0094] ζ=αf(r+1)+βf(p+1)+γf(s+1)

[0095] Among them, f(r+1), f(p+1), and f(s+1) are the average values ​​of the probability arrays of the three modes after correction by the SLIA algorithm, that is, the corrected multimodal probability data, and α, β, and γ are the corresponding correction proportional coefficients.

[0096] In this embodiment, in the smoking comprehensive judgment module, when the final smoking probability ζ is greater than or equal to the preset smoking probability threshold ζ 预设 , it indicates that there is smoking behavior and a smoking warning message is generated; otherwise, it indicates that there is no smoking behavior and no smoking warning message is generated.

[0097] In summary, the smoking detection system based on multimodal data in the above-mentioned embodiment is based on the collected multimodal perception data, and obtains probability arrays of multiple modes according to the multimodal perception data, and then uses the SLIA algorithm to correct each probability array to obtain corrected multimodal probability data, and then calculates the final smoking probability based on the corrected multimodal probability data, and compares the final smoking probability with a preset smoking probability threshold to obtain a comprehensive judgment result of the smoking behavior, which can realize the detection of smoking behavior through multimodal data and improve the detection accuracy of smoking behavior.

[0098] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A smoking detection system based on multimodal data, characterized in that: include: Perception data capture module, data transmission module, smoking multimodal judgment module, smoking comprehensive judgment module and smoking alarm module; The perception data capture module is used to collect multimodal perception data of the detection area; The data transmission module is used to transmit the collected multimodal sensing data to the smoking multimodal judgment module; The smoking multimodal judgment module is used to process the multimodal perception data to obtain a probability array of multiple modes; The smoking comprehensive judgment module is used to correct each probability array using the SLIA algorithm to obtain corrected multimodal probability data, and then calculate the final smoking probability based on the corrected multimodal probability data, compare the final smoking probability with a preset smoking probability threshold, obtain a comprehensive judgment result of smoking behavior, and select whether to generate smoking warning information according to the comprehensive judgment result; The smoking alarm module is used to receive smoking alarm information and warn management personnel and parties concerned.

2. A smoking detection system based on multimodal data according to claim 1, characterized in that: In the perception data capture module, the multi-module perception data includes infrared thermal images, personnel images and smoke concentration data of the detection area.

3. A smoking detection system based on multimodal data according to claim 2, characterized in that: In the smoking multi-modal judgment module, the infrared thermal image, the person image and the smoke concentration data are processed respectively to obtain the probability arrays of the three modes.

4. The smoking detection system based on multimodal data according to claim 3, characterized in that: In the smoking multi-modal judgment module, the process of processing the infrared thermal image is as follows: S11: dividing the infrared thermal image into N pixels, and obtaining the average temperature of the pixel by calculating the infrared radiation energy of the N pixels; S12: Setting the combustion point temperature thresholds to θ1 and θ2, and filtering out the pixels that meet the thresholds; θ1 is the lower limit of the threshold, and θ2 is the upper limit of the threshold; S13: Divide the adjacent pixel points that meet the threshold into the same group to obtain multiple pixel groups Z1, Z2,...Z n ; Z s1 , Z s2 ···Z sn is the corresponding Z1, Z2,...Z n The number of pixels included, where n ≥ 1; S14: Set standard value Z sb , Z sb is the pixel size of the standard burning point; S15: Calculate the first smoking probability value Rn, when Z si When Rn is equal to 0, si When it is not equal to 0, execute the following formula: S16: Obtain a first probability array by calculating the plurality of data: [R1, R2, R3...R (n-1) , R n ].

5. The smoking detection system based on multimodal data according to claim 4, characterized in that: In the smoking multi-modal judgment module, the process of processing the person image is specifically as follows: S21: extracting frames from the video stream to be detected to obtain an initial image group to be detected; S22: performing image preprocessing on the initial image to be detected helps to extract cigarette features from the image to be detected; S23: determining the pre-processed image containing the human face as the image to be detected; S24: establishing an image coordinate system, setting the mouth as area A and the cigarette butt as area B through grayscale processing; S25: Set the overlap time of regions A and B to M, where M follows a normal distribution; S26: Calculate the second smoking probability value: Among them, μ is the location parameter and σ is the scale parameter; S27: Obtain a second probability array by calculating the plurality of data: [P1, P2, P3...P (n-1) , P n ].

6. The smoking detection system based on multimodal data according to claim 5, characterized in that: In step S22, the image preprocessing method includes one or more combinations of grayscale, binarization, and dilation processing.

7. The smoking detection system based on multimodal data according to claim 5, characterized in that: In the smoking multi-modal judgment module, the process of processing the smoke concentration data is specifically as follows: S31: By applying a DC voltage between two metal plate electrodes and placing a piece of isotope americium-241 nearby; americium-241 can emit a small amount of alpha rays, ionizing the nearby air; S32: Record the ion current in the form of discrete data to obtain the following array [I1, I2, I3···I n ], where smoke concentration data is represented by ion current data; S33: Set the smoking smoke detection threshold to 1 a ,I b , where I a is the lower threshold, I b is the upper threshold value; S34: Calculate the third smoking probability value: Among them, I a ≠I b , is the correction parameter; S35: A third probability array is obtained by calculating multiple data: [S1, S2, S3...S (n-1) , S n ].

8. The smoking detection system based on multimodal data according to claim 7, characterized in that: In the smoking comprehensive judgment module, the correction process of the SLIA algorithm is as follows: S41: Initially, input the probability array corresponding to a certain mode and calculate its average value f(t); S42: Find the difference between a single data and f(t), take the positive value, delete the n / 5 rounded partial data with the largest difference, and then find the average value f(t+1), and judge the positive difference between the two average values ​​f(t+1) and f(t). If it is less than the set value, output the average value f(t+1) to end the loop, otherwise continue to execute; S43: When the number of output data is less than 1 / 10 of the number of initial data, the operation is terminated and the average value f(t+1) is output.

9. The smoking detection system based on multimodal data according to claim 8, characterized in that: In the smoking comprehensive judgment module, the calculation formula of the final smoking probability ζ is as follows: ζ=αf(r+1)+βf(p+1)+γf(s+1) Among them, f(r+1), f(p+1), and f(s+1) are the average values ​​of the probability arrays of the three modes after correction by the SLIA algorithm, that is, the corrected multimodal probability data, and α, β, and γ are the corresponding correction proportional coefficients.

10. The smoking detection system based on multimodal data according to claim 9, characterized in that: In the smoking comprehensive judgment module, when the final smoking probability ζ is greater than or equal to the preset smoking probability threshold ζ 预设 , it indicates that there is smoking behavior and a smoking warning message is generated; otherwise, it indicates that there is no smoking behavior and no smoking warning message is generated.

Images