Smoking behavior identification method and device, electronic equipment and storage medium

Abstract

The invention discloses a smoking behavior recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the frame extraction of video data, obtaining a video image, inputting the video image into a smoking feature recognition model, and obtaining a feature recognition result; and updating the number of video frames appearing in the smoking features of each category of the suspected smokers in the smoking feature record according to the feature recognition result, calculating the scores of the suspected smokers according to the categories of the smoking features and the number of video frames, and determining the suspected smokers whose scores are greater than a threshold as target smokers. And the alarm information of the smoking behavior in the target no-smoking area is generated, so that the target smoker is determined by continuously calculating the score of the suspected smoker by identifying the smoking characteristics of the types of cigarettes, cigarette ends, smoke, smoking gestures, smoke spitting, cigarette holding and the like and the number of the appearing video frames; compared with a method for judging the smoking behavior by identifying a single smoking feature, the method has the advantages that the misjudgment rate of the smoking behavior is reduced, and the accuracy of smoking behavior identification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual processing, and in particular, to a smoking behavior recognition method, device, electronic device, and storage medium. Background Art

[0002] In a production and manufacturing park, in order to ensure fire safety in the park, no-smoking areas are set in various areas of the park. Monitoring smoking behavior in no-smoking areas is one of the basic means to ensure safe production in the park.

[0003] Currently, video monitoring is mainly used for no-smoking areas. Specifically, high-definition visible light cameras, infrared cameras and other monitoring devices are set in no-smoking areas. After collecting video data through the monitoring devices, image recognition of smoking behavior is performed on the video images and an alarm is given. Existing methods for identifying cigarette butts or smoke in video images to determine whether there is a smoking behavior usually determine that there is a smoking behavior when a cigarette butt or smoke is recognized in the video image, resulting in a relatively high false positive rate. Summary of the Invention

[0004] The present invention provides a smoking behavior recognition method, device, electronic device, and storage medium to solve the problem of high false positive rate in the prior art when determining the existence of a smoking behavior when a cigarette butt or smoke is recognized in a video image.

[0005] In a first aspect, the present invention provides a smoking behavior recognition method, including:

[0006] Receiving multi-channel video data, and extracting frames from each channel of video data to obtain video images, where each channel of video data is data collected for a target no-smoking area;

[0007] For each channel of video data, inputting the video images obtained by frame extraction into a smoking feature recognition model to obtain a feature recognition result;

[0008] Updating the smoking feature record of the target no-smoking area according to the feature recognition result, where the smoking feature record is used to record the number of video frames in which various types of smoking features of suspected smoking persons in the target no-smoking area appear;

[0009] Calculating the score of the suspected smoking person according to the category of the smoking feature and the number of video frames;

[0010] Determining the suspected smoking person with a score greater than a preset threshold as a target smoking person, generating an alarm message indicating that a smoking behavior has occurred in the target no-smoking area, and returning to the step of inputting the video images obtained by frame extraction into the smoking feature recognition model to obtain a feature recognition result.

[0011] Optionally, extracting frames from each channel of video data to obtain video images, including:

[0012] Determine the frame extraction rate for each path of video data, and extract frames from the video data according to the frame extraction rate to obtain video images.

[0013] Optionally, input the video images obtained by frame extraction into a smoking feature recognition model to obtain a feature recognition result, including:

[0014] Input the video images obtained by frame extraction into the smoking feature recognition model;

[0015] Perform smoking feature recognition in the smoking feature recognition model to obtain the personnel ID of the suspected smoking person and smoking features of at least one category.

[0016] Optionally, the feature recognition result includes the personnel ID of the suspected smoking person and smoking features of at least one category. Update the smoking feature record of the target no-smoking area according to the feature recognition result, including:

[0017] Judge whether the personnel ID of the suspected smoking person in the feature recognition result has been recorded in the smoking feature record of the target no-smoking area;

[0018] If so, increment by 1 the number of video frames in which the target smoking feature of the personnel ID of the suspected smoking person appears in the smoking feature record, where the target smoking feature is the smoking feature of the suspected smoking person in the feature recognition result;

[0019] If not, add the personnel ID of the suspected smoking person to the smoking feature record, initialize in the smoking feature record the number of video frames in which the target smoking feature appears for this personnel ID, and record the current time as the start time of the suspected smoking person.

[0020] Optionally, calculate the score of the suspected smoking person according to the category of the smoking feature and the number of video frames, including:

[0021] Obtain the environmental data of the target no-smoking area, and determine the weights of different categories of smoking features based on the current time and the environmental data;

[0022] Calculate the weighted sum using the weight of the smoking feature and the number of video frames in which the smoking feature appears to obtain the score of the suspected smoking person.

[0023] Optionally, determine the target smoking persons as those suspected smoking persons whose scores are greater than a preset threshold, including:

[0024] Obtain the environmental data of the target no-smoking area, and determine the score threshold based on the current time and the environmental data;

[0025] Calculate the target duration using the current time and the start time of each suspected smoker in the smoking feature record;

[0026] For each suspected smoker, when the target duration is less than or equal to a preset duration threshold, determine whether the score of the suspected smoker is greater than or equal to the score threshold;

[0027] If so, determine the suspected smoker as a target smoker, delete the record of the target smoker in the smoking feature record, and set the target smoker as a non-detection object;

[0028] If not, return to the step of receiving multi-channel video data and extracting frames from each channel of video data to obtain video images;

[0029] When the target duration is greater than the preset duration threshold, delete the record of the suspected smoker in the smoking feature record, return to the step of receiving multi-channel video data, and extracting frames from each channel of video data to obtain video images.

[0030] Optionally, generating an alarm message for a smoking behavior occurring in the target no-smoking area includes:

[0031] Obtain the address information of the target no-smoking area, the identity information of the target smoker, and intercept a video segment including the video image from the video data;

[0032] Generate an alarm message including the address information, the identity information, and the video segment.

[0033] In a second aspect, the present invention provides a smoking behavior recognition device, including:

[0034] A video data frame extraction module, configured to receive multi-channel video data and extract frames from each channel of video data to obtain video images, and each channel of video data is data collected for a target no-smoking area;

[0035] A smoking feature recognition module, configured to input the video images obtained by frame extraction into a smoking feature recognition model for each channel of video data to obtain a feature recognition result;

[0036] A smoking feature record update module, configured to update the smoking feature record of the target no-smoking area according to the feature recognition result, where the smoking feature record is used to record the number of video frames in which various types of smoking features of suspected smokers in the target no-smoking area appear;

[0037] A score calculation module, configured to calculate the score of the suspected smoker according to the category of the smoking feature and the number of video frames;

[0038] A target smoking person determination module, configured to determine a suspected smoking person with a score greater than a preset threshold as a target smoking person, generate an alarm message for a smoking behavior occurring in the target no-smoking area, and return to the video data frame extraction module.

[0039] In a third aspect, the present invention provides an electronic device, which includes:

[0040] At least one processor; and

[0041] A memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior recognition method according to any one of the first aspects of the present invention.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions for causing a processor to implement the smoking behavior recognition method according to any one of the first aspects of the present invention when executed.

[0044] After the video data of the target no-smoking area is frame-extracted to obtain video images, the obtained video images are input into a smoking feature recognition model to obtain a feature recognition result. The smoking feature record of the target no-smoking area is updated according to the feature recognition result. The score of the suspected smoking person is calculated according to the category of the smoking feature in the smoking feature record and the number of video frames with the smoking feature. A suspected smoking person with a score greater than the preset threshold is determined as a target smoking person, and an alarm message for a smoking behavior occurring in the target no-smoking area is generated. It realizes the continuous calculation of the score of each suspected smoking person by identifying various types of smoking features such as cigarette butts, cigarette ends, smoke, smoking gestures, exhaling smoke, and holding cigarettes, as well as the number of video frames in which each type of smoking feature appears. The target smoking person is determined by the score. Compared with the situation of judging the existence of a smoking behavior by identifying a single smoking feature, the false positive rate of the smoking behavior is reduced, the accuracy of the smoking behavior recognition is improved, and the smoking behavior occurring in the no-smoking area can be discovered and prevented in time.

[0045] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0047] Figure 1 is a flowchart of a smoking behavior recognition method provided in Embodiment 1 of the present invention;

[0048] Figure 2 is a flowchart of a smoking behavior recognition method provided in Embodiment 2 of the present invention;

[0049] Figure 3 is the overall process of an example of smoking behavior recognition in the present invention;

[0050] Figure 4 is a schematic structural diagram of a smoking behavior recognition device provided in Embodiment 3 of the present invention;

[0051] Figure 5 is a schematic structural diagram of an electronic device provided in Embodiment 4 of the present invention. Specific Embodiments

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0053] Embodiment 1

[0054] Figure 1 is a flowchart of a smoking behavior recognition method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of identifying whether there is someone smoking in a no-smoking area. This method can be executed by a smoking behavior recognition device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. As Figure 1 shown, the smoking behavior recognition method includes:

[0055] S101. Receive multi-channel video data, and extract frames from each channel of video data to obtain video images. Each channel of video data is data collected for a target no-smoking area.

[0056] In this embodiment, a target no-smoking area can be set. The target no-smoking area can be an area where smoking is prohibited in a manufacturing enterprise. Exemplarily, the target no-smoking area can be areas such as production workshops, warehouses, offices, and elevators. Cameras can be set in the target no-smoking area, and video data can be collected by the cameras at a fixed or dynamic frame rate. The video data collected by multiple cameras can be sent to electronic devices such as edge terminals or servers, so that the edge terminal or server receives multiple paths of video data.

[0057] After the edge terminal or server receives multiple paths of video data, frame extraction processing can be performed on each path of video data to obtain video images. Exemplarily, frame extraction can be performed on the video data at a fixed or dynamic frame extraction rate, or key frames (I-frames) in the video data can be extracted. This embodiment does not limit the frame extraction method.

[0058] S102. For each path of video data, input the video image obtained by frame extraction into the smoking feature recognition model to obtain a feature recognition result.

[0059] The smoking features can be behavioral features and / or entity features. Exemplarily, the behavioral features can include smoking gesture, exhaling smoke action, lighting a cigarette, flicking cigarette ash and other behavioral features, and the entity features can include cigarette butts, cigarette sticks, smoke and other features. The smoking feature recognition model can be a neural network model that detects and tracks people and recognizes smoking features. The smoking feature recognition model can be trained with images including various smoking features, and its training method can refer to the supervised training method of neural networks in the prior art, which will not be elaborated here.

[0060] After frame extraction is performed on each path of video data, the video image obtained by frame extraction can be input into the smoking feature recognition model to obtain a feature recognition result. The feature recognition result can include the person ID and at least one category of smoking features.

[0061] S103. Update the smoking feature record of the target no-smoking area according to the feature recognition result. The smoking feature record is used to record the video frame numbers in which various categories of smoking features of suspected smokers in the target no-smoking area appear.

[0062] In this embodiment, a smoking feature record can be pre-configured. The smoking feature record is used to record the ID of the person with smoking features and the video frame numbers in which various categories of smoking features of this person are recognized. In addition, the start time when each person is first detected with smoking features is also recorded. The start time can be the time stamp of the video image when each person is first detected with smoking features.

[0063] After obtaining the feature recognition result, if the personnel ID in the feature recognition result has been recorded in the smoking feature record, it means that the person has been recognized as having a smoking feature before the current frame extraction. The video frame numbers of various types of smoking features in the feature recognition result for this person ID can be incremented by 1 in the smoking feature record. If the personnel ID in the feature recognition result is not recorded in the smoking feature record, it means that the person's smoking feature is detected for the first time. The person ID and smoking features can be added to the smoking feature record, and the video frame number when the added smoking features appear is initialized to 1.

[0064] S104. Calculate the scores of suspected smoking personnel according to the categories of smoking features and the video frame numbers.

[0065] The score represents the score of each suspected smoking personnel as of the frame extraction to obtain the video image. This score is determined by the categories of various smoking features of each suspected smoking personnel and the video frame numbers when they appear. The higher the score of a suspected smoking personnel, the higher the possibility of smoking.

[0066] In one embodiment, weights can be set for each category of smoking features, and the weighted sum is calculated as the score of the suspected smoking personnel by the weights of each category of smoking features and the video frame numbers when they appear. In another embodiment, the weighted average can also be calculated as the score of the suspected smoking personnel by the weights and video frame numbers. This embodiment does not limit the calculation method of the score.

[0067] S105. Determine the suspected smoking personnel with scores greater than the preset threshold as the target smoking personnel, and generate an alarm message for smoking behavior occurring in the target no-smoking area.

[0068] In this embodiment, the threshold can be pre-configured. After calculating the scores of each suspected smoking personnel at the end of each frame extraction, the scores are compared with the preset threshold. If the score is greater than the threshold, the suspected smoking personnel is determined as the target smoking personnel, and an alarm message for smoking behavior occurring in the target no-smoking area is generated. This alarm message can include data such as the identity information of the target smoking personnel, the address of the target no-smoking area, the smoking time, and the video image. After determining the target smoking personnel, the feature record of the target smoking personnel can be deleted from the smoking feature record, and the target smoking personnel is set as a non-detected person so that the target smoking personnel is not subjected to smoking feature recognition in the next frame extraction.

[0069] It should be noted that each time the score of each suspected smoker is compared with the threshold, if the score of the suspected smoker is less than or equal to the threshold, the duration of the smoking feature is calculated by subtracting the start time of the suspected smoker in the smoking feature record from the current time. If the duration of the smoking feature is less than the preset duration threshold (such as 60 seconds), the smoking feature of the suspected smoker will be continuously recognized after the next frame extraction. If the duration of the smoking feature is greater than the preset duration threshold (such as 60 seconds), it means that the score of the suspected smoker is still less than the threshold when the duration threshold is reached. The suspected smoker is not a smoker, and the record of the suspected smoker can be deleted from the smoking feature record, and S101 is returned to continue frame extraction of the video data.

[0070] After frame extraction of the video data in the target no-smoking area of the present invention to obtain video images, the obtained video images are input into a smoking feature recognition model to obtain a feature recognition result. The smoking feature record in the target no-smoking area is updated according to the feature recognition result. The score of the suspected smoker is calculated according to the category of the smoking feature and the number of video frames with the smoking feature in the smoking feature record. The suspected smoker with a score greater than the preset threshold is determined as the target smoker, and an alarm message for the occurrence of a smoking behavior in the target no-smoking area is generated. By recognizing various types of smoking features such as cigarette butts, cigarette ends, smoke, smoking gestures, exhaling smoke, and holding cigarettes, as well as continuously calculating the score of each suspected smoker based on the number of video frames with each type of smoking feature, and determining the target smoker through the score, compared with determining the existence of a smoking behavior by recognizing a single smoking feature, the false positive rate of smoking behavior is reduced, the accuracy of smoking behavior recognition is improved, and the smoking behavior occurring in the no-smoking area can be detected and prevented in a timely manner.

[0071] Embodiment 2

[0072] Figure 2 The flowchart of a smoking behavior recognition method provided by Embodiment 2 of the present invention. Embodiment 2 of the present invention is optimized on the basis of Embodiment 1 above, as Figure 2 shown, the smoking behavior recognition method includes:

[0073] S201. Receive multiplexed video data, determine the frame extraction rate of each path of video data, and perform frame extraction on the video data according to the frame extraction rate to obtain video images.

[0074] In this embodiment, the target no-smoking area may be an area where smoking is prohibited. Multiplexed video data can be obtained by collecting video data through cameras in multiple target no-smoking areas. The multiplexed video data is sent to an electronic device such as an edge terminal or a server, so that the edge terminal or the server and other electronic devices receive the multiplexed video data.

[0075] In one embodiment, different non-smoking areas are set with different frame rates according to the environment, flow of people, time, etc. For example, taking the target non-smoking area as an elevator car as an example, a higher frame rate can be set when the flow of people is high during the day, and a lower frame rate can be set when the flow of people is low at night. Taking the warehouse as the target non-smoking area as an example, the frame rate can be increased when the ambient light is dark. Specifically, for each channel of video data, at least two frame rates can be set in advance for the target non-smoking area corresponding to the channel of video data, and each frame rate is associated with at least one of the factors such as the environment, flow of people, and time of the target non-smoking area, so as to determine the frame rate of the channel of video data when the corresponding factors are met, so as to meet the target non-smoking area in different environments, flow of people, and time. The frame rate of the video data of the non-smoking area can be dynamically and flexibly adjusted, avoiding the increase of data processing volume by frame extraction at a higher frame rate when there is little or no people in the non-smoking area, resulting in waste of computing power of the edge terminal or server, which can save computing power and improve computing power utilization.

[0076] After considering factors such as the current environment, flow of people, current time, etc. of the target no-smoking area, the frame rate of the video data of the target no-smoking area is further determined through various factors according to a pre-configured mapping relationship between the factors and the frame rate. The video data of the target no-smoking area is framed at the frame rate to obtain a video image, wherein the mapping relationship between the factors and the frame rate can be a comparison table of different factors and frame rates.

[0077] S202: For each channel of video data, the video image obtained by extracting frames is input into a smoking feature recognition model.

[0078] The smoking feature recognition model of this embodiment may be a neural network model that detects and tracks people and recognizes smoking features. The smoking feature recognition model may be trained by the following steps:

[0079] S1. Obtain a sample image, where the sample image is an image of a person smoking, and annotate the sample image with a human body detection frame and categories of various smoking features;

[0080] S2. Initialize a smoking feature recognition model. For example, load any YOLO neural network in the existing YOLO series and initialize parameters.

[0081] S3, extracting sample images and inputting them into the smoking feature recognition model for recognition, thereby obtaining a human body detection frame and categories of various smoking features;

[0082] S4. Calculate the human detection loss value using the labeled and recognized human detection bounding boxes (such as calculating the mean squared error loss value), and calculate the smoking feature recognition loss value using the labeled and recognized smoking feature categories (such as calculating the mean squared error loss value), and obtain the total loss value by calculating the sum of the human detection loss value and the smoking feature recognition loss value;

[0083] S5. Determine whether the stop training condition is satisfied. The stop training condition is that the total loss value is less than the first threshold and both the human detection loss value and the smoking feature recognition loss value are less than the second threshold, where the second threshold is less than the first threshold; if so, execute S6, if not, execute S7;

[0084] S6. Determine that the smoking feature recognition model has completed training;

[0085] S7. Adjust the parameters of the smoking feature recognition model using the total loss value (such as performing gradient descent on the model parameters), and return to S3.

[0086] Training the smoking feature recognition model with the sample images of people smoking, and calculating the sum of the human detection loss value and the smoking feature recognition loss value as the total loss value to adjust the model during training can constrain the model to learn the capabilities of human tracking detection and smoking feature recognition. For each video data stream, the video images obtained by frame extraction can be input into the smoking feature recognition model.

[0087] S203. Perform smoking feature recognition in the smoking feature recognition model to obtain the person ID of the suspected smoking person and at least one category of smoking features.

[0088] After the video images are input into the smoking feature recognition model, the smoking feature recognition model performs person detection and tracking and smoking feature recognition to obtain the ID of the detected person and the category of the smoking features of each person, and takes the person ID, smoking features, and category as the feature recognition result.

[0089] The following is an example of the feature recognition result:

[0090] [(ID1, Holding cigarette - 1, Cigarette butt - 2, Cigarette stick - 3), (ID2, Smoke - 4), ……, (IDn, Smoke - 4, Exhaling smoke - 5)];

[0091] Among them, ID is the ID of the detected person, and -1, -2, etc. represent categories, that is, 1 is holding a cigarette, 2 is a cigarette butt, and so on.

[0092] S204. Determine whether the person ID of the suspected smoking person in the feature recognition result has been recorded in the smoking feature record of the target no - smoking area.

[0093] In this embodiment, the smoking feature record is used to record the personnel ID of the suspected smoking personnel detected, the start time when the smoking feature is first detected, and the video frame numbers at which various smoking features are recognized. The smoking feature record can be pre-configured for each target no-smoking area. After frame extraction and smoking feature recognition of the video data of a target no-smoking area to obtain a feature recognition result, for each personnel ID in the feature recognition result, it can be determined whether the personnel ID has been recorded in the smoking feature record. If so, it means that the personnel corresponding to the personnel ID has been detected with smoking features before the current frame extraction, and S205 can be executed. If not, it means that the personnel corresponding to the personnel ID is detected with smoking features for the first time, and S206 can be executed.

[0094] S205. In the smoking feature record, increment by 1 the video frame number at which the target smoking feature of the personnel ID of the suspected smoking personnel appears. The target smoking feature is the smoking feature of the suspected smoking personnel in the feature recognition result.

[0095] If the personnel ID of the suspected smoking personnel in the feature recognition result has been recorded in the smoking feature record, then in the smoking feature record, increment by 1 the video frame number at which the smoking feature in the feature recognition result appears for the suspected smoking personnel.

[0096] Exemplarily, assume that the smoking feature record before the current frame extraction is:

[0097] [(ID1, T: 20250326093020; holding a cigarette: 8 frames; cigarette butt: 10 frames; cigarette stick: 20 frames); (ID2, T: 20250326101720; holding a cigarette: 3 frames)];

[0098] The feature recognition result obtained from the current frame extraction is:

[0099] [(ID1, holding a cigarette - 1, cigarette butt - 2, cigarette stick - 3)];

[0100] From this, it can be determined that it is necessary to increment by 1 the video frame numbers of holding a cigarette, cigarette butt, and cigarette stick in the smoking feature record. The updated smoking feature record is as follows:

[0101] [(ID1, T: 20250326093020; holding a cigarette: 9 frames; cigarette butt: 11 frames; cigarette stick: 21 frames); (ID2, T: 20250326101720; holding a cigarette: 3 frames)].

[0102] S206. Add the personnel ID of the suspected smoking personnel to the smoking feature record, initialize in the smoking feature record the video frame number at which the target smoking feature appears for the personnel ID, and record the current time as the start time T of the suspected smoking personnel.

[0103] If the personnel ID of the suspected smoking person in the feature recognition result is not recorded in the smoking feature record, then add the personnel ID of the suspected smoking person in the feature recognition result to the smoking feature record, and add the smoking features of the suspected smoking person to the smoking feature record, initialize the added smoking features to 1, and record that the start time T is equal to the timestamp of the video image of the extracted frame in the video data.

[0104] Exemplarily, assume that the smoking feature record before the current frame extraction is:

[0105] [(ID1, T: 20250326093020; Holding a cigarette: 8 frames; Cigarette butt: 10 frames; Cigarette stick: 20 frames); (ID2, T: 20250326101720; Holding a cigarette: 3 frames)];

[0106] The feature recognition result obtained from the current frame extraction is:

[0107] [(ID1, Holding a cigarette - 1, Cigarette butt - 2, Cigarette stick - 3), (ID3, Holding a cigarette - 1)];

[0108] From this, it can be determined that the video frame numbers of holding a cigarette, cigarette butt, and cigarette stick of the suspected smoking person ID1 in the smoking feature record are incremented by 1, and a record of the suspected smoking person ID3 is added. The updated smoking feature record is as follows:

[0109] [(ID1, T: 20250326093020; Holding a cigarette: 9 frames; Cigarette butt: 11 frames; Cigarette stick: 21 frames); (ID2, T: 20250326101720; Holding a cigarette: 3 frames); (ID3, T: 20250326101809; Holding a cigarette: 1 frame)].

[0110] S207. Obtain the environmental data of the target no - smoking area, and determine the weights of different types of smoking features based on the current time and the environmental data.

[0111] In this embodiment, the environmental data may be data such as the light intensity and the number of people in the target no-smoking area. An environmental data-time-weight comparison table may be pre-configured for each target no-smoking area. This comparison table includes the weights of different types of smoking characteristics at different environmental data and different times. Those skilled in the art can configure the environmental data-time-weight comparison table according to the situation of the target no-smoking area. In this environmental data-time-weight comparison table, when the light intensity is greater, the weight of the smoking characteristics sensitive to light brightness (such as cigarette butts) is smaller (the cigarette butt characteristics are not obvious in bright light), and vice versa, when the light intensity is smaller, the weight of the smoking characteristics sensitive to light brightness (such as cigarette butts) is greater (cigarette butts are more obvious in the dark). Thus, the weights of various types of smoking characteristics are set according to the environmental data and time periods of different target no-smoking areas to achieve personalized smoking behavior recognition in different target no-smoking areas and improve the accuracy of smoking behavior recognition.

[0112] S208. Calculate the weighted sum using the weight of the smoking characteristic and the number of video frames in which the smoking characteristic appears to obtain the score of the suspected smoking person.

[0113] Exemplarily, the score S of the i-th suspected smoking person as of the current frame extraction can be calculated by the following formula i :

[0114] S i = (F1.E1 + F2.E2 + … F n .E n );

[0115] F1, F2, ……, F n respectively represent the weights of the first to the n-th smoking characteristics, and E1, E2, ……, E n respectively represent the number of video frames in which the first to the n-th smoking characteristics appear.

[0116] S209. Determine the score threshold based on the current time and environmental data.

[0117] Exemplarily, each target no-smoking area can set corresponding score thresholds for different time periods and different environmental data. Exemplarily, a lower score threshold can be set when the current time period belongs to the peak period of the number of people, so as to exclude the influence of complex environments when the people in the target no-smoking area are relatively crowded, improve the sensitivity of smoking behavior recognition, and avoid missing the recognition of smoking behavior in complex environments. On the contrary, a higher score threshold can be set when the current time period belongs to the off-peak period of the number of people, so as to improve the accuracy of recognizing the smoking behavior of individuals. Or in combination with environmental data, such as setting a higher score threshold when the light brightness in the target no-smoking area is higher to accurately recognize smoking behavior in a clearly visible environment, and setting a lower score threshold when the light brightness in the target no-smoking area is lower to recognize smoking behavior through a lower score threshold in a darker environment and avoid misidentifying or missing the recognition of smoking behavior.

[0118] S210. Calculate the target duration by using the current time and the start time of each suspected smoking person in the smoking feature record.

[0119] The current time can be the system time, and the start time is the time when the smoking feature of the suspected smoking person is first recognized. After each frame extraction updates the smoking feature record, the duration from the start time to the current time can be calculated as the target duration, which represents the duration of continuously recognizing the smoking feature of the suspected smoking person.

[0120] S211. For each suspected smoking person, when the target duration is less than the preset duration threshold, determine whether the score of the suspected smoking person is greater than or equal to the score threshold.

[0121] For each suspected smoking person, after calculating the score and target duration of the suspected smoking person, when the target duration is less than the preset duration threshold (such as 60 seconds), it can be determined whether the score of the suspected smoking person is greater than or equal to the score threshold. If so, execute S212 and S213; if not, continue to extract frames to continuously track and detect the smoking feature of the suspected smoking person.

[0122] S212. Determine the suspected smoking person as the target smoking person, delete the record of the target smoking person in the smoking feature record, and set the target smoking person as a non-detection object.

[0123] If the score of the suspected smoking person is greater than or equal to the score threshold within the preset duration, it can be determined that the suspected smoking person is the target smoking person, delete the record of the target smoking person in the smoking feature record, and set the target smoking person as a non-detection object. At the next frame extraction, no smoking feature detection will be performed on the target smoking person, avoiding repeated detection and recognition of the smoking person after determining that the person is smoking.

[0124] S213. Generate an alarm message for the occurrence of smoking behavior in the target no-smoking area.

[0125] In one embodiment, after determining the target smoking person, the address information of the target no-smoking area, the identity information of the target smoking person, and a video segment that intercepts the video image including the frame extraction can be obtained from the video data, and an alarm message including the address information, identity information, and video segment is generated and pushed to the alarm terminal.

[0126] S214. When the target duration is greater than the preset duration threshold, delete the record of the suspected smoking person in the smoking feature record.

[0127] Since the record of the target smoking person is deleted after each frame extraction to identify the smoking feature and update the smoking feature record, that is, the suspected smoking person with a score less than the score threshold in the smoking feature record. If the target duration of the suspected smoking person with a score less than the score threshold is greater than the duration threshold (60 seconds), it means that the score is still less than the score threshold after continuously identifying the smoking feature of the suspected smoking person for 60 seconds. This suspected smoking person is not a smoking person, and the record of this suspected smoking person can be deleted from the smoking feature record, and return to S214 to continue frame extraction of the video data.

[0128] As Figure 3 shown in the flowchart of an example of smoking behavior recognition, the edge terminal receives the video and identifies the people in the video, determines whether the smoking feature is recognized to trigger tracking detection. If so, track and detect the person within 60 seconds and calculate the score. If not, continue to identify the people in the video; after calculating the score of the person, determine whether the score is greater than the threshold. If so, determine that there is a smoking person and issue a smoking alarm for the person. If not, continue to identify the video people until the video is no longer received to end the entire process.

[0129] In this embodiment, the video images obtained by frame extraction of the video data are input into the smoking feature recognition model to obtain the person ID and smoking features of at least one category, and the person ID and smoking features are used to update the number of video frames in which the smoking features of various categories of suspected smoking people appear in the smoking feature record. The weights and score thresholds of various categories of smoking features are determined through the environmental data of the target no-smoking area and the current time. The weighted sum is calculated using the weights of the smoking features and the number of video frames in which the smoking features appear as the score of the suspected smoking person, and the target duration of the suspected smoking person is calculated using the current time and the start time. When the target duration is less than the duration threshold and the score is greater than the score threshold, the suspected smoking person is determined as the target smoking person, and an alarm message is generated. On the one hand, it realizes the continuous calculation of the score of each suspected smoking person by identifying various categories of smoking features such as cigarette butts, cigarette ends, smoke, smoking gestures, exhaling smoke, and holding cigarettes, as well as the number of video frames in which various categories of smoking features appear. The target smoking person is determined through the score, which reduces the misjudgment rate of smoking behavior and improves the accuracy of smoking behavior recognition compared with judging the existence of smoking behavior by identifying a single smoking feature, and can timely discover and prevent smoking behavior occurring in the no-smoking area. On the other hand, the weights and score thresholds of various categories of smoking features can be determined according to the environmental data of the target no-smoking area and the current time, which can perform personalized smoking behavior recognition for different target no-smoking areas and time periods, and improves the flexibility and accuracy of smoking behavior recognition.

[0130] Embodiment III

[0131] Figure 4 is a schematic structural diagram of a smoking behavior recognition device provided in Embodiment III of the present invention. AsFigure 4 As shown, the smoking behavior recognition device includes:

[0132] A video data frame extraction module 401, configured to receive multiple channels of video data, extract frames from each channel of video data to obtain video images, and each channel of video data is data collected for a target no-smoking area;

[0133] A smoking feature recognition module 402, configured to input the video images obtained by frame extraction into a smoking feature recognition model for each channel of video data to obtain a feature recognition result;

[0134] A smoking feature record update module 403, configured to update the smoking feature record of the target no-smoking area according to the feature recognition result, where the smoking feature record is used to record the number of video frames in which various types of smoking features of suspected smoking persons in the target no-smoking area appear;

[0135] A score calculation module 404, configured to calculate the score of the suspected smoking person according to the category of the smoking feature and the number of video frames;

[0136] A target smoking person determination module 405, configured to determine the suspected smoking person with a score greater than a preset threshold as a target smoking person, generate an alarm message indicating that a smoking behavior has occurred in the target no-smoking area, and return to the video data frame extraction module 401.

[0137] Optionally, the video data frame extraction module 401 includes:

[0138] A frame extraction unit, configured to determine the frame extraction rate of each channel of video data, and extract frames from the video data according to the frame extraction rate to obtain video images.

[0139] Optionally, the smoking feature recognition module 402 includes:

[0140] An image input unit, configured to input the video images obtained by frame extraction into a smoking feature recognition model;

[0141] A person and smoking feature recognition unit, configured to perform smoking feature recognition in the smoking feature recognition model to obtain the person ID of the suspected smoking person and at least one category of smoking features.

[0142] Optionally, the feature recognition result includes the person ID of the suspected smoking person and at least one category of smoking features, and the smoking feature record update module 403 includes:

[0143] A judgment unit, configured to judge whether the person ID of the suspected smoking person in the feature recognition result has been recorded in the smoking feature record of the target no-smoking area; if so, execute the first update unit, if not, execute the second update unit;

[0144] A first update unit, configured to increment by 1 the number of video frames in which the target smoking feature of the person ID of the suspected smoking person appears in the smoking feature record, where the target smoking feature is the smoking feature of the suspected smoking person in the feature recognition result;

[0145] A second update unit, configured to add the person ID of the suspected smoking person to the smoking feature record, initialize in the smoking feature record the number of video frames in which the target smoking feature appears for the person ID, and record the current time as the start time of the suspected smoking person.

[0146] Optionally, the score calculation module 404 includes:

[0147] A weight item determination unit, configured to obtain the environmental data of the target no-smoking area, and determine the weights of different categories of smoking features based on the current time and the environmental data;

[0148] A score calculation unit, configured to calculate a weighted sum by using the weights of the smoking features and the number of video frames in which the smoking features appear, to obtain the score of the suspected smoking person.

[0149] Optionally, the target smoking person determination module 405 includes:

[0150] A score threshold determination unit, configured to obtain the environmental data of the target no-smoking area, and determine a score threshold based on the current time and the environmental data;

[0151] A duration calculation unit, configured to calculate a target duration by using the current time and the start time of each suspected smoking person in the smoking feature record;

[0152] A score judgment unit, configured to, for each suspected smoking person, when the target duration is less than or equal to a preset duration threshold, judge whether the score of the suspected smoking person is greater than or equal to the score threshold. If so, execute the smoking person determination unit; if not, execute the video data frame extraction module 401;

[0153] A smoking person determination unit, configured to determine the suspected smoking person as a target smoking person, delete the record of the target smoking person in the smoking feature record, and set the target smoking person as a non-detection object;

[0154] A record deletion unit, configured to, when the target duration is greater than the preset duration threshold, delete the record of the suspected smoking person in the smoking feature record, and return to the video data frame extraction module 401.

[0155] Optionally, the target smoking person determination module 405 includes:

[0156] An information determination unit, configured to obtain the address information of the target no-smoking area, the identity information of the target smoker, and intercept a video segment including the video image from the video data;

[0157] An alarm unit, configured to generate alarm information including the address information, the identity information, and the video segment.

[0158] The smoking behavior recognition device provided by the embodiments of the present invention can execute the smoking behavior recognition method provided by any embodiment of the present invention, and has corresponding function modules and beneficial effects for executing the method.

[0159] Embodiment 4

[0160] Figure 5 FIG. shows a schematic structural diagram of an electronic device 50 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0161] As Figure 5 shown, the electronic device 50 includes at least one processor 51, and a memory communicatively connected to the at least one processor 51, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 into the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other through a bus 54. The input / output (I / O) interface 55 is also connected to the bus 54.

[0162] A plurality of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a magnetic disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0163] The processor 51 may be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as the smoking behavior recognition method.

[0164] In some embodiments, the smoking behavior recognition method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the smoking behavior recognition method described above may be executed. Alternatively, in other embodiments, the processor 51 may be configured to execute the smoking behavior recognition method by any other suitable means (e.g., by means of firmware).

[0165] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the computer programs are executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0168] To provide for interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0169] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0170] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0171] It should be understood that various forms of processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0172] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying smoking behavior, characterized in that, Including: Receiving multiplex video data, and extracting frames from each piece of video data to obtain video images, where each piece of video data is data collected for a target no-smoking area; For each piece of video data, inputting the video image obtained by frame extraction into a smoking feature recognition model to obtain a feature recognition result; Updating the smoking feature record of the target no-smoking area according to the feature recognition result, where the smoking feature record is used to record the number of video frames in which various types of smoking features of suspected smokers in the target no-smoking area appear; Calculating the score of the suspected smoker according to the category of the smoking feature and the number of video frames; Determining the suspected smokers with scores greater than a preset threshold as target smokers, generating an alarm message for a smoking behavior occurring in the target no-smoking area, and returning to the step of inputting the video image obtained by frame extraction into the smoking feature recognition model to obtain a feature recognition result.

2. The smoking behavior recognition method according to claim 1, characterized in that, Extracting frames from each piece of video data to obtain video images, including: Determining the frame extraction rate of each piece of video data, and extracting frames from the video data according to the frame extraction rate to obtain video images.

3. The smoking behavior recognition method according to claim 1, wherein Inputting the video image obtained by frame extraction into the smoking feature recognition model to obtain a feature recognition result, including: Inputting the video image obtained by frame extraction into the smoking feature recognition model; Performing smoking feature recognition in the smoking feature recognition model to obtain the person ID of the suspected smoker and at least one category of smoking features.

4. The smoking behavior recognition method according to any one of claims 1 to 3, characterized in that, The feature recognition result includes the person ID of the suspected smoker and at least one category of smoking features. Updating the smoking feature record of the target no-smoking area according to the feature recognition result, including: Judging whether the person ID of the suspected smoker in the feature recognition result has been recorded in the smoking feature record of the target no-smoking area; If so, adding 1 to the number of video frames in which the target smoking feature of the person ID of the suspected smoker appears in the smoking feature record, where the target smoking feature is the smoking feature of the suspected smoker in the feature recognition result; If not, adding the person ID of the suspected smoker to the smoking feature record, initializing the number of video frames in which the target smoking feature appears for the person ID in the smoking feature record, and recording the current time as the start time of the suspected smoker.

5. The smoking behavior recognition method according to claim 4, characterized in that, Calculating the score of the suspected smoker according to the category of the smoking feature and the number of video frames, including: Obtaining the environmental data of the target no-smoking area, and determining the weights of different categories of smoking features based on the current time and the environmental data; Calculating the weighted sum using the weight of the smoking feature and the number of video frames in which the smoking feature appears to obtain the score of the suspected smoker.

6. The smoking behavior recognition method according to claim 4, characterized in that Determining the suspected smokers with scores greater than a preset threshold as target smokers, including: Obtaining the environmental data of the target no-smoking area, and determining the score threshold based on the current time and the environmental data; Calculating the target duration using the current time and the start time of each suspected smoker in the smoking feature record. For each suspected smoker, when the target duration is less than or equal to a preset duration threshold, determine whether the score of the suspected smoker is greater than or equal to the score threshold; If so, determine the suspected smoker as a target smoker, delete the record of the target smoker in the smoking feature record, and set the target smoker as a non-detection object; If not, return to the step of receiving multi-channel video data and extracting frames from each channel of video data to obtain video images; When the target duration is greater than the preset duration threshold, delete the record of the suspected smoker in the smoking feature record, return to the step of receiving multi-channel video data, and extract frames from each channel of video data to obtain video images.

7. The smoking behavior recognition method according to any one of claims 1-3, characterized in that Generate an alarm message for a smoking behavior occurring in the target no-smoking area, including: Obtain the address information of the target no-smoking area, the identity information of the target smoker, and intercept a video segment including the video image from the video data; Generate an alarm message including the address information, the identity information, and the video segment.

8. A smoking behavior recognition device, characterized in that, Include: A video data frame extraction module, configured to receive multi-channel video data, and extract frames from each channel of video data to obtain video images, and each channel of video data is data collected for a target no-smoking area; A smoking feature recognition module, configured to input the video images obtained by frame extraction into a smoking feature recognition model for each channel of video data to obtain a feature recognition result; A smoking feature record update module, configured to update the smoking feature record of the target no-smoking area according to the feature recognition result, and the smoking feature record is used to record the number of video frames in which various types of smoking features of suspected smokers in the target no-smoking area appear; A score calculation module, configured to calculate the score of the suspected smoker according to the category of the smoking feature and the number of video frames; A target smoker determination module, configured to determine a suspected smoker with a score greater than a preset threshold as a target smoker, generate an alarm message for a smoking behavior occurring in the target no-smoking area, and return to the video data frame extraction module.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the smoking behavior recognition method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for enabling a processor to execute the smoking behavior recognition method according to any one of claims 1-7 when executed.

Images