A Deep Learning-Based Method for Detecting Abnormal Behavior in Surveillance Video Data

Abstract

This invention relates to the field of data anomaly detection technology, and more particularly to a method for detecting abnormal behavior in surveillance video data based on deep learning. The method acquires video images of any individual in real time and detects them. When no frontal position information of the individual is detected, the video image is marked as a target image. Multiple consecutive frames of target images are acquired to obtain a target image set. Based on the head and hand position information of any individual in each frame of the target image set, motion coefficients are obtained to identify whether any individual exhibits suspected abnormal behavior. If any individual exhibits suspected abnormal behavior, the abnormal behavior detection result is determined based on the smoke concentration data in the target area and the pixel value differences of pixels in each frame of the target image set. This invention can accurately identify abnormal behavior of employees in a target area.

Description

Technical Field

[0001] This invention relates to the field of data anomaly detection technology, and in particular to a method for detecting abnormal behavior in surveillance video data based on deep learning. Background Technology

[0002] No-smoking and no-fire areas are key areas for safety management, often involving the production, storage, and handling of flammable and explosive materials. For example, chemical plants, oil and gas stations, and warehousing centers strictly prohibit smoking and open flames. These areas not only require the use of numerous high-temperature and high-pressure equipment during production processes but also contain various flammable and explosive chemicals; even slight negligence could lead to fires or explosions. Therefore, strengthening safety production management and strictly monitoring unsafe behaviors (such as smoking and open flame operations) are crucial to ensuring safety.

[0003] Existing technologies for detecting unsafe behaviors of personnel in no-smoking and no-fire areas typically combine surveillance cameras with rule-based detection. The YOLO (YoLaw on Object Detection) algorithm is used to identify abnormal behavior. This algorithm is trained by labeling abnormal and normal images to identify abnormal actions captured by the surveillance camera. However, this algorithm has limitations. For example, when an employee smokes while partially obscured or at a specific angle, the detection system may struggle to capture the complete smoking action or trajectory, leading to missed detections. Furthermore, interference factors such as high-temperature steam, smoke from operating equipment, or dust in no-smoking and no-fire areas can cause inaccurate detection of abnormal behavior by the surveillance camera, thus affecting the accuracy of the detection system.

[0004] Therefore, accurately identifying abnormal behavior of people in no-smoking and no-fire areas has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method for detecting abnormal behavior in surveillance video data based on deep learning, in order to solve the problem of how to accurately identify abnormal behavior of people in no-smoking and no-fire areas.

[0006] This invention provides a method for detecting abnormal behavior in surveillance video data based on deep learning. The method includes the following steps:

[0007] For any person appearing in the surveillance video in the target location, the video image of the person is acquired in real time, and the frontal position information of the person in the video image is detected. When no frontal position information of the person is detected, the video image of the person is marked as a target image. Multiple consecutive frames of the target image are acquired until the number of target images reaches a preset threshold, and a set of target images is obtained.

[0008] Obtain the head position information and hand position information of any person in each frame of the target image set; based on the head position information and hand position information of any person in each frame of the target image set, obtain motion coefficients to characterize the overall motion state of the head and hands of any person; and based on the motion coefficients, identify whether any person has suspected abnormal behavior in the target image set.

[0009] If any person exhibits suspected abnormal behavior in the target image set, the smoke concentration data in the target location and the pixel values ​​of each pixel in each frame of the target image set are obtained. Based on the differences between the smoke concentration data in the target location and the pixel values ​​of each pixel in each frame of the target image set, the abnormal behavior detection result of any person is determined.

[0010] Preferably, after acquiring the video image of any one of the persons in real time and detecting the frontal position information of the person in the video image, the method further includes:

[0011] When the frontal position information of a person is detected, the target detection model is used to identify abnormal behavior of any person in the video image.

[0012] Preferably, the step of obtaining motion coefficients representing the overall motion state of the head and hands of any person in each frame of the target image set, based on the head position information and hand position information of any person, includes:

[0013] For any target image in the target image set, based on the head position information and hand position information of any person in the target image, the relative angle and relative distance between the head and hand of any person in the target image are obtained. The relative angle and relative distance between the head and hand of any person in each frame of the target image set are obtained respectively, and the corresponding relative angle data sequence and relative distance data sequence are obtained.

[0014] The relative angle data sequence is subjected to second-order differencing to obtain a second-order differencing data sequence. The mean of the second-order differencing data sequence is obtained and denoted as the first mean. The perfect square difference between each second-order differencing data in the second-order differencing data sequence and the first mean is calculated to obtain the mean of the perfect square difference, denoted as the second mean. The first ratio of the second mean to a preset stationarity parameter is obtained. The negative of the first ratio is substituted into an exponential function with the natural constant as the base to obtain the first exponential function result.

[0015] For any relative distance in the relative distance data sequence other than the first relative distance, obtain the first difference between the any relative distance and its previous relative distance, obtain the first product of the first difference and the preset distance scaling parameter, perform cosine processing on the first product to obtain the cosine value, obtain the Gaussian function value of the any relative distance, perform weighted summation processing on the cosine value and the Gaussian function value to obtain the weighted summation result of the any relative distance, obtain the weighted summation result of each relative distance in the relative distance data sequence respectively, and obtain the mean of the corresponding weighted summation result, which is denoted as the third mean;

[0016] The result of the first exponential function and the third mean are weighted and summed to obtain the motion coefficients used to characterize the overall motion state of the head and hands of any given person.

[0017] Preferably, identifying whether any person exhibits suspected abnormal behavior in the target image set based on the motion coefficient includes:

[0018] A first motion coefficient threshold and a second motion coefficient threshold are set, wherein the first motion coefficient threshold is less than the second motion coefficient threshold;

[0019] If the motion coefficient is greater than or equal to the first motion coefficient threshold and less than or equal to the second motion coefficient threshold, then it is confirmed that any person in the target image set exhibits suspected abnormal behavior.

[0020] Preferably, acquiring the smoke concentration data within the target location and the pixel values ​​of pixels in each frame of the target image set includes:

[0021] Obtain the sampling time period corresponding to the target image set, denoted as the target sampling time period, and obtain the smoke concentration data sequence in the target location during the target sampling time period;

[0022] Using a preset smoke detection algorithm, the smoke region of the head of any person in each frame of the target image set is obtained, and the pixel value of each pixel in the head smoke region is obtained accordingly.

[0023] Preferably, determining the abnormal behavior detection result of any person based on the smoke concentration data in the target location and the pixel value differences of pixels in each frame of the target image set includes:

[0024] Based on the differences in smoke concentration data in the smoke concentration data sequence, a smoke significance score is obtained within the target sampling period;

[0025] Based on the pixel value differences of pixels in each head smoke region in the target image set, count the number of target images in the target image set that exhibit smoke diffusion characteristics;

[0026] If the smoke salience score is greater than or equal to a preset smoke salience score threshold, and the number is greater than or equal to a first preset number, then it is determined that any one of the persons has abnormal behavior.

[0027] Preferably, obtaining the smoke significance score within the target sampling period based on the differences in smoke concentration data in the smoke concentration data sequence includes:

[0028] The smoke concentration data sequence is subjected to first-order differencing to obtain a corresponding first-order differencing data sequence. The mean of the first-order differencing data sequence is obtained and denoted as the fourth mean. For any first-order differencing data in the first-order differencing data sequence, the perfect square difference between the arbitrary first-order differencing data and the fourth mean is obtained. The sum of the perfect square difference by a constant 1 and a first preset multiple is obtained, and the reciprocal of the sum is denoted as the smoke data score corresponding to the arbitrary first-order differencing data. The smoke data score corresponding to each first-order differencing data in the first-order differencing data sequence is obtained, and the mean of the smoke data scores is obtained. The mean of the smoke data scores is used as the smoke significance score within the target sampling period.

[0029] Preferably, the step of counting the number of target images in the target image set that exhibit smoke diffusion characteristics based on the pixel value differences of pixels in each head smoke region in the target image set includes:

[0030] For any target image in the target image set other than the first frame target image, obtain the overlapping area of ​​the head smoke region in the target image and the head smoke region in the previous target image of the target image. Calculate the difference between the corresponding pixel values ​​of each pixel in the overlapping area in the target image and the previous target image, and obtain the average difference of the overlapping area, which is denoted as the fifth average. Normalize the fifth average to obtain the smoke diffusion degree of the target image.

[0031] The number of pixels in the head smoke region of any target image is obtained and recorded as the first number. The number of pixels in the head smoke region of the previous target image of any target image is obtained and recorded as the second number. The second difference between the first number and the second number is calculated. The negative of the second difference of the second preset multiple is substituted into the exponential function with the natural constant as the base to obtain the second exponential function result. The difference between the constant 1 and the second exponential function result is calculated to obtain the smoke diffusion area of ​​any target image.

[0032] The smoke diffusion degree and smoke diffusion area of ​​any target image are weighted and summed to obtain the smoke diffusion coefficient of any target image.

[0033] Obtain the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set other than the first target image. Based on the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set other than the first target image, count the number of target images in the target image set that have smoke diffusion characteristics.

[0034] Preferably, the step of counting the number of target images with smoke diffusion characteristics in the target image set based on the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set excluding the first frame target image includes:

[0035] If the smoke diffusion degree and smoke diffusion area of ​​any target image are both greater than a constant 0, and the smoke diffusion coefficient of any target image is greater than a preset smoke diffusion coefficient threshold, it is determined that any target image has smoke diffusion characteristics. Target images with smoke diffusion characteristics in the target image set are then filtered to obtain the number of target images with smoke diffusion characteristics in the target image set.

[0036] Preferably, after determining the abnormal behavior detection result of any person based on the smoke concentration data in the target location and the pixel value differences of pixels in each frame of the target image set, the method further includes:

[0037] In real time, new character video images are acquired in a second preset number of consecutive frames following the target image set. If all the new character video images in the second preset number of frames are target images, the new character video images in the second preset number of frames are added to the target image set, and the target images in the first second preset number of frames are removed from the target image set to obtain a new target image set.

[0038] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0039] This invention targets any person appearing in a surveillance video within a target location. It acquires a real-time video image of that person and detects their frontal position within the video image. If no frontal position information is detected, the video image is marked as a target image. Multiple consecutive frames of the target image are acquired until the number of target images reaches a preset threshold, resulting in a target image set. The invention then acquires the head and hand position information of any person in each frame of the target image set. Based on this information, motion coefficients are obtained to characterize the overall motion state of the person's head and hands. These motion coefficients are used to identify whether any person exhibits suspected abnormal behavior within the target image set. If any person exhibits suspected abnormal behavior, the invention acquires smoke concentration data within the target location and pixel values ​​of pixels in each frame of the target image set. Based on the differences between these smoke concentration data and pixel values, the detection result of the abnormal behavior of the person is determined. Specifically, when no frontal position information of a person can be detected, motion coefficients representing the overall motion state of the head and hands of any person in each frame of the target image are obtained for preliminary identification. This allows for accurate identification even when a person is occluded or exhibits abnormal behavior at a specific angle. When any person is identified as exhibiting suspected abnormal behavior, the abnormal behavior is detected based on the smoke concentration data in the target location and the pixel value differences of each pixel in each frame of the target image. This reduces interference from high-temperature steam and other factors on the identification of abnormal behavior, making the monitoring of abnormal behaviors of people in the target location, including but not limited to smoking, more accurate. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a deep learning-based method for detecting abnormal behavior in surveillance video data, provided in Embodiment 1 of the present invention. Detailed Implementation

[0042] Embodiments of this disclosure are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting it.

[0043] It should be noted that the terms "first," "second," etc., used in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0044] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0045] See Figure 1 This is a flowchart of a deep learning-based method for detecting abnormal behavior in surveillance video data, as provided in Embodiment 1 of the present invention. Figure 1 As shown, the method may include:

[0046] Step S101: For any person appearing in the surveillance video in the target location, acquire the person video image in real time, and detect the frontal position information of the person in the person video image. When no frontal position information of the person is detected, mark the person video image as a target image, acquire multiple consecutive frames of the target image until the number of target images reaches a preset threshold, and obtain a target image set.

[0047] No-smoking and no-fire areas include chemical plants, oil and gas stations, and storage centers. Abnormal behaviors of personnel in no-smoking and no-fire areas include smoking and open flame operations. This embodiment takes a chemical plant as the target location and takes the smoking behavior of personnel in the target location as an example to detect the smoking behavior of personnel in the chemical plant.

[0048] Surveillance cameras are deployed in different areas of the chemical plant to ensure that the collected data covers various scenarios. Priority is given to production workshops, equipment maintenance areas, boiler rooms, and rest areas where people may gather. The cameras monitor at a frame rate of 30 FPS, which is not a limitation and can be set according to the specific implementation scenario. Any person appearing in the surveillance video within the chemical plant is designated as a target person. Real-time video images of the target person are acquired, and a spatiotemporal transformer model is used to detect the frontal position information of the target person in the real-time video images.

[0049] The spatiotemporal transformer model is a pre-trained model. Historical video data from the chemical plant is collected, and representative sample data is selected from the historical video data. The labels are refined through manual annotation and video analysis tools to obtain a trained dataset of people's location information. The labels include front and back positions. Then, the spatiotemporal transformer model is used for training to obtain a trained spatiotemporal transformer model. Since the spatiotemporal transformer model is an existing technology, it will not be described in detail here.

[0050] When a person's frontal position information is detected, an object detection model is used to identify any person's abnormal smoking behavior in the video image. Specifically, a YOLO model is trained using a pre-trained smoking behavior dataset to obtain a trained YOLO model (i.e., a trained object detection model). Since YOLO model training is a current technology, it will not be detailed here. The video image of the person whose frontal position information has been detected is fed into the trained object detection model. The model analyzes key parts (head, hands) in the input video image and outputs bounding boxes containing these key parts. These bounding boxes are then matched against the pre-trained smoking behavior dataset to detect whether smoking behavior exists within the bounding box (detecting a hand near the mouth or a cigarette butt). If a match is successful, the target person is determined to have abnormal smoking behavior; if a match is unsuccessful, the target person is determined not to have abnormal smoking behavior.

[0051] When the frontal position of a person cannot be detected, a well-trained object detection model (YOLO model) may struggle to capture the complete smoking action or trajectory, potentially leading to missed detections and impacting the detection results for the target person's smoking behavior. Therefore, when the frontal position of a person cannot be detected, further analysis of key body parts of the target person is necessary to accurately detect their abnormal smoking behavior.

[0052] Since smoking behavior is usually a periodic, repetitive action, involving the hand moving back and forth towards the head, abnormal smoking behavior can be further identified by analyzing the movement trajectories of key body parts of the target individual.

[0053] In this embodiment, the cycle of the target person performing a smoking action is set to 5 seconds. This is not a limitation and can be set according to the specific implementation scenario. When the frontal position information of the person cannot be detected, the real-time acquired video image of the target person is marked as the target image. Multiple consecutive frames of target images are acquired until the number of target images reaches a preset threshold (i.e., 150 frames of target images in the cycle of one smoking action). The 150 consecutive target images are combined into a target image set for analysis of whether the target person has abnormal smoking behavior.

[0054] At this point, the target image set is obtained.

[0055] Step S102: Obtain the head position information and hand position information of any person in each frame of the target image set. Based on the head position information and hand position information of any person in each frame of the target image set, obtain motion coefficients to characterize the overall motion state of the head and hands of any person. Based on the motion coefficients, identify whether any person has suspected abnormal behavior in the target image set.

[0056] After obtaining the target image set, it is necessary to further identify the abnormal smoking behavior of the target person based on the motion trajectory of key parts of their body. Since the key parts directly related to the target person's smoking behavior are their head and hands, skeletal keypoint detection technology can be used to obtain the head and hand position information of the target person in each frame of the target image set. Skeletal keypoint detection technology is an existing technology and will not be elaborated here. Then, based on the head and hand position information of the target person in each frame of the target image set, motion coefficients are obtained to represent the overall motion state of the target person's head and hands. These motion coefficients are then used to identify the target person's abnormal smoking behavior in the target image set.

[0057] Skeletal landmark detection technology can identify the key points of a target person, such as the head and hands, in each frame of the target image and generate a set of key point coordinates. This set includes the coordinates of the target person's head and hands. Therefore, by combining dynamic features over time and analyzing the coordinate changes of these key parts, the target person's motion trajectory can be analyzed. Since the movement of the target person's head and hands changes periodically during smoking—with the relative distance and angle between them periodically increasing and decreasing—the coordinates of the target person's hands and head can be obtained in each frame. Then, based on the changes in the relative distance and angle between these coordinates, motion coefficients representing the overall motion of the target person's head and hands can be obtained. These motion coefficients can then be used to determine whether the target person's motion trajectory matches the motion trajectory of smoking.

[0058] The method for obtaining motion coefficients to characterize the overall motion state of the target person's head and hands, based on the changes in the relative distance and relative angle between the hand and head coordinates in each frame of the target image, is as follows:

[0059] (1) For any target image in the target image set, based on the head position information and hand position information of any person in the target image, obtain the relative angle and relative distance between the head and hand of any person in the target image, and obtain the relative angle and relative distance between the head and hand of any person in each frame of the target image set, and obtain the corresponding relative angle data sequence and relative distance data sequence.

[0060] In one embodiment, taking the i-th frame of the target image set as an example, a coordinate system is constructed in the i-th frame of the target image using skeletal keypoint detection technology to obtain the keypoint coordinates of the head position information and hand position information of the target person in the i-th frame of the target image, which are denoted as (x... head,i ,y head,i ) and (x hand,i ,y hand,i The formula for calculating the relative angle and relative distance between the head and hands of the target person in the i-th frame of the target image is as follows:

[0061]

[0062] Where, θ i x represents the relative angle between the head and hands of the target person in the i-th frame of the target image; hand,i x is the x-coordinate of the hand position information of the target person in the i-th frame of the target image; head,i y is the x-coordinate of the head position information of the target person in the i-th frame of the target image; hand,i y is the ordinate of the hand position information of the target person in the i-th frame of the target image; head,i d represents the ordinate of the head position information of the target person in the i-th frame of the target image; i is the relative distance between the head and hand of the target person in the i-th frame of the target image; arctan() is the arctangent function; i is the index of the target image in the target image set.

[0063] It should be noted that relative angles and relative distances are existing technologies and will not be elaborated upon here.

[0064] Based on the method for obtaining the relative angle and relative distance between the head and hand of the target person in the i-th frame target image, the relative angle and relative distance between the head and hand of the target person in each frame target image of the target image set are obtained respectively. The relative angles between the head and hand of the target person in each frame target image of the target image set are combined into a relative angle data sequence, and the relative distances between the head and hand of the target person in each frame target image of the target image set are combined into a relative distance data sequence.

[0065] (2) Obtain motion coefficients to characterize the overall motion state of the head and hands of any person based on the relative angle data sequence and the relative distance data sequence.

[0066] Specifically, the relative angle data sequence is subjected to second-order differencing to obtain a second-order differencing data sequence. The mean of the second-order differencing data sequence is obtained and denoted as the first mean. The perfect square difference between each second-order differencing data in the second-order differencing data sequence and the first mean is calculated to obtain the mean of the perfect square difference, denoted as the second mean. The first ratio of the second mean to a preset stationarity parameter is obtained. The negative of the first ratio is substituted into an exponential function with the natural constant as the base to obtain the first exponential function result.

[0067] For any relative distance in the relative distance data sequence other than the first relative distance, obtain the first difference between the any relative distance and its previous relative distance, obtain the first product of the first difference and the preset distance scaling parameter, perform cosine processing on the first product to obtain the cosine value, obtain the Gaussian function value of the any relative distance, perform weighted summation processing on the cosine value and the Gaussian function value to obtain the weighted summation result of the any relative distance, obtain the weighted summation result of each relative distance in the relative distance data sequence respectively, and obtain the mean of the corresponding weighted summation result, which is denoted as the third mean;

[0068] The result of the first exponential function and the third mean are weighted and summed to obtain the motion coefficients used to characterize the overall motion state of the head and hands of any given person.

[0069] In one embodiment, the expression for calculating the motion coefficients used to characterize the overall motion state of the target person's head and hands is as follows:

[0070]

[0071] Where C is the motion coefficient of the target person in the target image set; θ ′ j Let be the j-th second-order difference data point in the second-order difference data sequence; j is the index of the second-order difference data point in the second-order difference data sequence; V is the number of second-order difference data points in the second-order difference data sequence; F is the mean of the second-order difference data sequence; τ is the preset stationarity parameter; i is the index of the target image in the target image set; N is the number of target images in the target image set; e is the natural constant; Δd i Let be the difference between the relative distance in the i-th frame of the target image set and the relative distance in the (i-1)-th frame of the target image set; k is the preset distance scaling parameter; cos() is the cosine function; d iμ is the relative distance between the i-th frame target image in the target image set; d λ is the mean of the relative distance data series; d These are preset parameters; α is the weighting coefficient for relative angle changes; β is the weighting coefficient for relative distance changes; γ is the weighting coefficient for periodic distance changes; δ is the weighting coefficient for relative distance; the reference values ​​for α and β are 0.4 and 0.6 respectively, the reference values ​​for γ and δ are 0.5, the reference value for τ is 4, and the reference value for k is... π is the mathematical constant pi, λ d The reference value is twice the standard deviation of the relative distance function sequence. There is no restriction here, and it can be set according to the specific implementation scenario.

[0072] It should be noted that, Let be the Gaussian function value corresponding to the relative distance in the target image of the i-th frame. The Gaussian function is a prior art and will not be described in detail here. This indicates the range of variation in the relative angle between the target person's head and hands. The larger the value, the greater the range of fluctuation in the relative angle between the target's head and hands, and the greater the target's motion coefficient. The smaller the value, the smaller the range of fluctuation of the relative angle between the head and hands of the target person, and the smaller the target person's motion coefficient. Since smoking behavior has a fixed range of motion, that is, the relative angle between the head and hands of the target person who is smoking has a fixed range of fluctuation, the range of fluctuation of the relative angle between the head and hands of the target person that is too large or too small does not conform to the motion trajectory of smoking behavior. In other words, the motion coefficient that is too large or too small does not conform to the motion trajectory of smoking behavior. This indicates the periodicity of motion representing the change in the relative distance between the head and hands of the target person. The larger the value, the greater the period of change in the relative distance between the target's head and hands, and the greater the target's motion coefficient. The smaller the value, the smaller the period of change in the relative distance between the target person's head and hands, and the smaller the target person's motion coefficient. Since smoking behavior has a fixed period, that is, the change in the relative distance between the target person's head and hands has a fixed period, if the period of change in the relative distance between the target person's head and hands is too large or too small, it does not conform to the motion trajectory of smoking behavior. In other words, if the motion coefficient is too large or too small, it does not conform to the motion trajectory of smoking behavior.

[0073] Furthermore, a motion coefficient threshold is set, and the target person is identified as having any suspected abnormal behavior based on the motion coefficient threshold.

[0074] Specifically, a first motion coefficient threshold and a second motion coefficient threshold are set. The first motion coefficient threshold is less than the second motion coefficient threshold. If the motion coefficient is greater than or equal to the first motion coefficient threshold and less than or equal to the second motion coefficient threshold, then it is confirmed that any person in the target image set has suspected abnormal behavior.

[0075] In one embodiment, a first motion coefficient threshold is set to 0.5, and a second motion coefficient threshold is set to 0.9. These are not limited and can be set according to the specific implementation scenario. When the motion coefficient is less than 0.5, it indicates that the fluctuation range of the relative angle between the target person's head and hand is too small, or the period of change in the relative distance between the target person's head and hand is too small, meaning the relative change between the target person's head and hand is relatively drastic, which does not conform to the motion trajectory of smoking behavior, thus confirming that the target person does not smoke. When the motion coefficient is greater than 0.9, it indicates that the fluctuation range of the relative angle between the target person's head and hand is too large, or the period of change in the relative distance between the target person's head and hand is too large, meaning the relative change between the target person's head and hand is weak, which does not conform to the motion trajectory of smoking behavior, thus confirming that the target person does not smoke. When the motion coefficient is greater than or equal to 0.5 and less than or equal to 0.9, it indicates that the relative change between the target person's head and hand conforms to the motion trajectory of smoking behavior, thus confirming that the target person has suspected smoking behavior.

[0076] This confirms whether the target person exhibited any suspicious smoking behavior within the target image set.

[0077] Step S103: If any person exhibits suspected abnormal behavior in the target image set, then obtain the smoke concentration data in the target location and the pixel value of each pixel in each frame of the target image set. Based on the smoke concentration data in the target location and the difference in pixel value of each pixel in each frame of the target image set, determine the abnormal behavior detection result of any person.

[0078] When the motion coefficient is greater than or equal to 0.5 and less than or equal to 0.9, it indicates that the relative changes between the head and hands of the target person are similar to the motion trajectory of smoking behavior, confirming that the target person is suspected of smoking behavior. However, since the motion trajectories of drinking water, wiping sweat, etc. are also similar to the motion trajectory of smoking behavior, it cannot be directly determined that the target person is smoking behavior. Therefore, further analysis is needed to determine whether the target person is smoking behavior.

[0079] Because smoking produces smoke around the head of the person smoking, and infrared sensors deployed within the monitoring area of ​​a chemical plant can collect smoke concentration data, it's possible to determine the presence of smoke around the head of a target person and thus whether they are smoking. However, smoking isn't the only source of smoke in a chemical plant; high-temperature steam and other factors can also affect the smoke concentration data collected by the sensors. Therefore, relying solely on smoke concentration data is insufficient for accurate detection of smoking. Furthermore, smoking not only produces smoke around the head but also exhibits diffusion characteristics. Therefore, combining smoke concentration data from the head area with these diffusion characteristics is crucial for accurately determining whether someone is smoking. In the target image, the pixel values ​​of different pixels change with the smoke concentration. As the smoke diffuses, the smoke concentration decreases, and the pixel values ​​of the pixels in the target image increase. Therefore, the sampling period corresponding to the target image set is obtained and denoted as the target sampling period. The smoke concentration data sequence within the chemical plant during the target sampling period is obtained. Using a preset smoke detection algorithm, the head smoke region of any person in each frame of the target image set is obtained, and the pixel values ​​of the pixels in each head smoke region are obtained for subsequent analysis of whether the target person is smoking.

[0080] In one embodiment, assuming the sampling period corresponding to the target image set is from 2:00:00 PM to 2:00:05 PM, smoke concentration data of the chemical plant is acquired simultaneously at a sampling frequency of once per second during the sampling period in the target image set. The smoke concentration data within the sampling period corresponding to the target image set are combined into a smoke concentration data sequence. This is not limited and can be set according to the specific implementation scenario. The smoke detection algorithm is used to obtain the smoke region of the head of the target person in each frame of the target image set, and the pixel value of the pixel point of the smoke region of the head of the target person in each frame of the target image is obtained. The smoke detection algorithm is existing technology and will not be described in detail here.

[0081] Furthermore, based on the smoke concentration data within the chemical plant and the pixel value differences of each frame of the target image in the target image set, the method for determining whether the target person has engaged in smoking behavior is as follows:

[0082] (1) Based on the differences in smoke concentration data in the smoke concentration data sequence, obtain the smoke significance score within the target sampling period.

[0083] Specifically, the smoke concentration data sequence is subjected to first-order differencing to obtain a corresponding first-order differencing data sequence. The mean of the first-order differencing data sequence is obtained and denoted as the fourth mean. For any first-order differencing data in the first-order differencing data sequence, the perfect square difference between the arbitrary first-order differencing data and the fourth mean is obtained. The sum of the perfect square difference by a constant 1 and a first preset multiple is obtained, and the reciprocal of the sum is denoted as the smoke data score corresponding to the arbitrary first-order differencing data. The smoke data score corresponding to each first-order differencing data in the first-order differencing data sequence is obtained, and the mean of the smoke data scores is obtained. The mean of the smoke data scores is used as the smoke significance score within the target sampling period.

[0084] In one embodiment, the formula for calculating the smoke significance score during the target sampling period is:

[0085]

[0086] Where S is the smoke significance score during the target sampling period; H t μ represents the t-th first-order difference data point in the first-order difference data sequence; t is the index of the first-order difference data point in the first-order difference data sequence; μ H ρ is the mean of the first-order difference data sequence; ρ is the first preset multiple, with a reference value of 2, which is not limited here and can be set according to the specific implementation scenario; T is the number of first-order difference data in the first-order difference data sequence; 1 is a constant.

[0087] It should be noted that due to the diffusion characteristics of smoke, the smoke concentration will vary at each sampling time. Therefore, H t ―μ H The larger the value, the greater the difference between adjacent smoke concentration data in the smoke concentration data sequence, the more it matches the characteristics of smoke, and the higher the smoke significance score.

[0088] (2) Based on the pixel value difference of each head smoke region in the target image set, count the number of target images in the target image set that have smoke diffusion characteristics.

[0089] For any target image in the target image set other than the first frame target image, obtain the overlapping area of ​​the head smoke region in the target image and the head smoke region in the previous target image of the target image. Calculate the difference between the corresponding pixel values ​​of each pixel in the overlapping area in the target image and the previous target image, and obtain the average difference of the overlapping area, denoted as the fifth average. Normalize the fifth average to obtain the smoke diffusion degree of the target image.

[0090] In one embodiment, taking the i-th frame target image in the target image set as an example, the formula for calculating the smoke diffusion degree of the i-th frame target image is:

[0091]

[0092] Where D represents the smoke diffusion level of the target image in frame i; M represents the number of pixels in the overlapping region of the head smoke area in the target image in frame i and frame i-1; I i―1,m Let I be the pixel value of the m-th pixel in the overlapping region in the (i-1)-th frame of the target image; i,m is the pixel value of the m-th pixel in the overlapping region in the i-th frame of the target image; m is the index of the pixel in the overlapping region; i is the index of the target image in the target image set; 255 is the maximum gray value.

[0093] It should be noted that the greater the difference in pixel values ​​between the overlapping areas of the head smoke region in the i-th frame target image and the (i-1)-th frame target image, the greater the degree of smoke diffusion in the i-th frame target image, and the greater the smoke diffusion coefficient of the i-th frame target image.

[0094] b. Obtain the number of pixels in the head smoke region of any target image, denoted as the first number. Obtain the number of pixels in the head smoke region of the previous target image of any target image, denoted as the second number. Calculate the second difference between the first number and the second number. Substitute the negative of the second difference of the second preset multiple into an exponential function with the natural constant as the base to obtain the second exponential function result. Calculate the difference between the constant 1 and the second exponential function result to obtain the smoke diffusion area of ​​any target image.

[0095] In one embodiment, taking the i-th frame target image in the target image set as an example, the formula for calculating the smoke diffusion area of ​​the i-th frame target image is:

[0096]

[0097] Where E is the smoke diffusion area of ​​the target image in the i-th frame; e is the natural constant; p is the second preset multiple, with a reference value of 0.01, which is not limited here and can be set according to the specific implementation scenario; R i R is the first quantity; i―1 is the second quantity; i is the index of the target image in the target image set; 1 is a constant.

[0098] It should be noted that the larger the difference between the first quantity and the second quantity, the larger the area of ​​the head smoke region in the i-th frame target image, the larger the smoke diffusion area of ​​the i-th frame target image, and the larger the smoke diffusion coefficient of the i-th frame target image.

[0099] c. The smoke diffusion degree and smoke diffusion area of ​​any target image are weighted and summed to obtain the smoke diffusion coefficient of any target image.

[0100] In one embodiment, taking the i-th frame target image in the target image set as an example, the smoke diffusion coefficient of the i-th frame target image is calculated:

[0101]

[0102] Among them, A i denoted as the smoke diffusion coefficient of the target image in the i-th frame; D represents the smoke diffusion degree of the target image in the i-th frame; and E represents the smoke diffusion area of ​​the target image in the i-th frame. ω is the weighting coefficient for the degree of smoke diffusion in the target image of the i-th frame; ω is the weighting coefficient for the area of ​​smoke diffusion in the target image of the i-th frame. The reference value for ω is 0.5, and there is no restriction here. It can be set according to the specific implementation scenario.

[0103] It should be noted that the greater the degree of smoke diffusion in the i-th frame target image, the more obvious the smoke diffusion feature in the i-th frame target image, and the greater the smoke diffusion coefficient of the i-th frame target image; the larger the smoke diffusion area in the i-th frame target image, the larger the area of ​​the head smoke region in the i-th frame target image, and the greater the smoke diffusion coefficient of the i-th frame target image.

[0104] d. Obtain the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set except for the first target image. Based on the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set except for the first target image, count the number of target images in the target image set that have smoke diffusion characteristics.

[0105] Specifically, if the smoke diffusion degree and smoke diffusion area of ​​any target image are both greater than a constant 0, and the smoke diffusion coefficient of any target image is greater than a preset smoke diffusion coefficient threshold, it is determined that any target image has smoke diffusion characteristics. Target images with smoke diffusion characteristics in the target image set are then selected to obtain the number of target images with smoke diffusion characteristics in the target image set.

[0106] In one embodiment, taking the i-th frame target image in the target image set as an example, the smoke diffusion coefficient threshold is set to 0.2. This is not limited here and can be set according to the specific implementation scenario. If the smoke diffusion coefficient of the i-th frame target image in the target image set is greater than 0.2, and the smoke diffusion degree and smoke diffusion area of ​​the i-th frame target image are both greater than 0, then it is determined that there is smoke diffusion feature in the i-th frame target image. According to the method for judging whether there is smoke diffusion feature in the i-th frame target image, it is determined whether there is smoke diffusion feature in each frame target image in the target image set. Target images with smoke diffusion feature in the target image set are filtered to obtain the number of target images with smoke diffusion feature in the target image set.

[0107] (3) If the smoke significance score is greater than or equal to the preset smoke significance score threshold, and the number is greater than or equal to the first preset number, then it is determined that any one of the persons has abnormal smoking behavior.

[0108] In one embodiment, the smoke salience score threshold is set to 0.8. This is not limited here and can be set according to the specific implementation scenario. If the smoke salience score in the target sampling period is greater than or equal to 0.8, and there are 90 or more target images with smoke diffusion characteristics in the target sampling period, then it is determined that the target person has abnormal smoking behavior. This is not limited here and can be set according to the specific implementation scenario.

[0109] After determining the abnormal smoking behavior within the target sampling period, new person video images of the second preset number of consecutive frames following the target image set are acquired in real time. If all the new person video images of the second preset number of frames are target images, the new person video images of the second preset number of frames are added to the target image set. The target images of the first second preset number of frames in the target image set are removed to obtain a new target image set. It is then determined whether smoking behavior exists in the new target image set.

[0110] For example: Real-time acquisition of 30 consecutive frames (1 second) of new person video images following the target image set (i.e., 5 seconds). If all 30 consecutive frames (1 second) of new person video images following the target image set (i.e., 5 seconds) are target images, then the target images in the first 30 frames (1 second) of the target image set are removed, and the 30 consecutive frames (1 second) of new person video images following the target image set (i.e., 5 seconds) are added to the target image set, resulting in a new target image set. Based on the method for judging smoking behavior in the target image set, it is determined whether smoking behavior exists in the new target image set. If at least one frame of the 30 consecutive frames (1 second) of new person video images following the target image set (i.e., 5 seconds) detects the frontal position information of a person (i.e., the new person video image is not a target image), then the trained target detection model is directly used to identify the abnormal smoking behavior in the new person video image where the frontal position information of the person was detected.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting abnormal behavior in surveillance video data based on deep learning, characterized in that, The method includes: For any person appearing in the surveillance video within the chemical plant, the video image of that person is acquired in real time, and the frontal position information of the person in the video image is detected. When no frontal position information of the person is detected, the video image of that person is marked as a target image. Multiple consecutive frames of the target image are acquired until the number of target images reaches a preset threshold, thus obtaining a set of target images. Obtain the head position information and hand position information of any person in each frame of the target image set; based on the head position information and hand position information of any person in each frame of the target image set, obtain motion coefficients to characterize the overall motion state of the head and hands of any person; and based on the motion coefficients, identify whether any person has suspected abnormal behavior in the target image set. If any of the individuals exhibits suspected abnormal behavior in the target image set, the smoke concentration data within the chemical plant and the pixel values ​​of each pixel in each frame of the target image set are obtained. Based on the differences between the smoke concentration data within the chemical plant and the pixel values ​​of each pixel in each frame of the target image set, the abnormal behavior detection result of any individual is determined. The step of obtaining motion coefficients to characterize the overall motion state of the head and hands of any person in each frame of the target image set, based on the head and hand position information of any person, includes: For any target image in the target image set, based on the head position information and hand position information of any person in the target image, the relative angle and relative distance between the head and hand of any person in the target image are obtained. The relative angle and relative distance between the head and hand of any person in each frame of the target image set are obtained respectively, and the corresponding relative angle data sequence and relative distance data sequence are obtained. The relative angle data sequence is subjected to second-order differencing to obtain a second-order differencing data sequence. The mean of the second-order differencing data sequence is obtained and denoted as the first mean. The perfect square difference between each second-order differencing data in the second-order differencing data sequence and the first mean is calculated to obtain the mean of the perfect square difference, denoted as the second mean. The first ratio of the second mean to a preset stationarity parameter is obtained. The negative of the first ratio is substituted into an exponential function with the natural constant as the base to obtain the first exponential function result. For any relative distance in the relative distance data sequence other than the first relative distance, obtain the first difference between the any relative distance and its previous relative distance, obtain the first product of the first difference and the preset distance scaling parameter, perform cosine processing on the first product to obtain the cosine value, obtain the Gaussian function value of the any relative distance, perform weighted summation processing on the cosine value and the Gaussian function value to obtain the weighted summation result of the any relative distance, obtain the weighted summation result of each relative distance in the relative distance data sequence respectively, and obtain the mean of the corresponding weighted summation result, which is denoted as the third mean; The weighted sum of the first exponential function result and the third mean is used to obtain the motion coefficients that characterize the overall motion state of the head and hands of any given person.

2. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 1, characterized in that, After acquiring the video image of any person in real time and detecting the frontal position information of the person in the video image, the method further includes: When the frontal position information of a person is detected, the target detection model is used to identify the abnormal smoking behavior of any person in the video image.

3. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 1, characterized in that, The step of identifying whether any person exhibits suspected abnormal behavior in the target image set based on the motion coefficient includes: A first motion coefficient threshold and a second motion coefficient threshold are set, wherein the first motion coefficient threshold is less than the second motion coefficient threshold; If the motion coefficient is greater than or equal to the first motion coefficient threshold and less than or equal to the second motion coefficient threshold, then it is confirmed that any person in the target image set exhibits suspected abnormal behavior.

4. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 1, characterized in that, The step of acquiring the smoke concentration data within the chemical plant and the pixel values ​​of each frame of the target image in the target image set includes: Obtain the sampling time period corresponding to the target image set, denoted as the target sampling time period, and obtain the smoke concentration data sequence within the chemical plant during the target sampling time period; Using a preset smoke detection algorithm, the smoke region of the head of any person in each frame of the target image set is obtained, and the pixel value of each pixel in the head smoke region is obtained accordingly.

5. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 4, characterized in that, The step of determining the abnormal behavior detection result of any person based on the smoke concentration data within the chemical plant and the pixel value differences of pixels in each frame of the target image set includes: Based on the differences in smoke concentration data in the smoke concentration data sequence, obtain the smoke significance score within the target sampling period; Based on the pixel value differences of pixels in each head smoke region in the target image set, count the number of target images in the target image set that exhibit smoke diffusion characteristics; If the smoke salience score is greater than or equal to a preset smoke salience score threshold, and the number is greater than or equal to a first preset number, then it is determined that any one of the individuals has abnormal smoking behavior.

6. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 5, characterized in that, The step of obtaining a smoke significance score for the target sampling period based on the differences in smoke concentration data in the smoke concentration data sequence includes: The smoke concentration data sequence is subjected to first-order differencing to obtain a corresponding first-order differencing data sequence. The mean of the first-order differencing data sequence is obtained and denoted as the fourth mean. For any first-order differencing data in the first-order differencing data sequence, the perfect square difference between the arbitrary first-order differencing data and the fourth mean is obtained. The sum of the perfect square difference by a constant 1 and a first preset multiple is obtained, and the reciprocal of the sum is denoted as the smoke data score corresponding to the arbitrary first-order differencing data. The smoke data score corresponding to each first-order differencing data in the first-order differencing data sequence is obtained, and the mean of the smoke data scores is obtained. The mean of the smoke data scores is used as the smoke significance score within the target sampling period.

7. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 5, characterized in that, The step of counting the number of target images in the target image set that exhibit smoke diffusion characteristics based on the pixel value differences of pixels in each head smoke region in the target image set includes: For any target image in the target image set other than the first frame target image, obtain the overlapping area of ​​the head smoke region in the target image and the head smoke region in the previous target image of the target image. Calculate the difference between the corresponding pixel values ​​of each pixel in the overlapping area in the target image and the previous target image, and obtain the average difference of the overlapping area, which is denoted as the fifth average. Normalize the fifth average to obtain the smoke diffusion degree of the target image. The number of pixels in the head smoke region of any target image is obtained and recorded as the first number. The number of pixels in the head smoke region of the previous target image of any target image is obtained and recorded as the second number. The second difference between the first number and the second number is calculated. The negative of the second difference of the second preset multiple is substituted into the exponential function with the natural constant as the base to obtain the second exponential function result. The difference between the constant 1 and the second exponential function result is calculated to obtain the smoke diffusion area of ​​any target image. The smoke diffusion degree and smoke diffusion area of ​​any target image are weighted and summed to obtain the smoke diffusion coefficient of any target image. Obtain the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set other than the first target image. Based on the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set other than the first target image, count the number of target images in the target image set that have smoke diffusion characteristics.

8. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 7, characterized in that, The step of counting the number of target images with smoke diffusion characteristics in the target image set based on the smoke diffusion degree, smoke diffusion area, and smoke diffusion coefficient of each target image in the target image set excluding the first frame target image includes: If the smoke diffusion degree and smoke diffusion area of ​​any target image are both greater than a constant 0, and the smoke diffusion coefficient of any target image is greater than a preset smoke diffusion coefficient threshold, it is determined that any target image has smoke diffusion characteristics. Target images with smoke diffusion characteristics in the target image set are then filtered to obtain the number of target images with smoke diffusion characteristics in the target image set.

9. The method for detecting abnormal behavior in surveillance video data based on deep learning according to claim 1, characterized in that, After determining the abnormal behavior detection result of any person based on the smoke concentration data within the chemical plant and the pixel value differences of pixels in each frame of the target image set, the method further includes: In real time, new character video images are acquired in a second preset number of consecutive frames following the target image set. If all the new character video images in the second preset number of frames are target images, the new character video images in the second preset number of frames are added to the target image set, and the target images in the first second preset number of frames are removed from the target image set to obtain a new target image set.

Images