A smoking behavior recognition method based on computer vision

Abstract

The present application relates to a kind of smoking behavior recognition method based on computer vision, belong to computer vision technical field.First, the input video is detected human key point, defined the feature of judging whether human exists smoking behavior.Then according to the feature of key point, whether there is suspected smoking posture is judged, the video frame in video that meets suspected smoking posture is extracted, reaches the preliminary examination of smoking behavior identification.After preliminary screening suspected smoking posture frame, define smoking region of interest, based on RetinaFace face detection algorithm identifies face and face key point, and utilize face detection result to establish smoking region of interest extraction method.Based on the above extracted smoking region of interest, YOLOv5 is used to detect cigarette in region of interest, to achieve the purpose of smoking behavior recognition.The present application takes into account the advantages of existing smoking recognition method, avoids the problem of low accuracy caused by too small cigarette when single cigarette detection is used to realize smoking recognition.

Description

Technical Field

[0001] This invention relates to a computer vision-based method for recognizing smoking behavior, belonging to the field of computer vision technology. Background Technology

[0002] Currently, methods for cigarette detection can be divided into two categories: non-visual methods and computer vision-based methods. Non-visual methods include smoke detectors, WiFi signal-based cigarette detection, and wearable device-based smoking behavior recognition. Visual methods include cigarette smoke detection, posture and motion recognition, and cigarette target detection.

[0003] Smoke detectors detect smoke by detecting the smoke produced when cigarettes are smoked. However, smoke detectors have several drawbacks: in open areas, smoke easily disperses and dilutes, making it difficult to detect; and the presence of many other particles in the air can easily cause false alarms. Smoke detection based on wearable devices typically involves collecting hand movement data (such as acceleration, direction of movement, and distance) using a personal wristband or similar device. This data is then analyzed to determine if someone is smoking. However, this method is unsuitable for general public smoking detection because it requires a wearable device and is only suitable for laboratory settings. Smoke detection based on Wi-Fi signals also struggles to effectively identify smoking due to the instability and inaccuracy of Wi-Fi signals.

[0004] Smoking behavior recognition based on cigarette smoke features treats cigarette smoke as a foreground target, using foreground segmentation algorithms to separate the foreground from the background, and then identifying it based on features such as the shape, color, and texture of the smoke. However, methods for detecting cigarette smoke suffer from problems such as the easy diffusion of cigarette smoke, low concentration, and indistinct edges. Furthermore, indoor backgrounds are mostly white, similar in color to the smoke produced by smoking, making them difficult to distinguish. Outdoor backgrounds are generally more complex, with more interference factors, making detection more difficult. Smoking recognition methods based on cigarette detection generally aim to identify smoking by detecting cigarette targets in the image. However, since cigarettes are generally very small in images, this method has low detection accuracy, and it can only detect cigarettes, not the person smoking. Methods based on human posture for smoking behavior recognition are prone to confusing smoking with similar actions such as drinking water and eating, also having certain limitations. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a computer vision-based method for recognizing smoking behavior, so as to solve the problem of recognizing people smoking in videos.

[0006] The technical solution of this invention is: a computer vision-based method for recognizing smoking behavior. First, key human features are detected in the input video, and features for determining whether a person is smoking are defined. Then, based on the features of the key points, a judgment is made on whether there is a suspected smoking posture. Video frames matching the suspected smoking posture are extracted to achieve initial screening for smoking behavior recognition. After initially screening out frames with suspected smoking postures, a region of interest (ROI) for smoking is defined. The RetinaFace face detection algorithm is used to identify faces and facial key points, and a method for extracting the ROI is established using the face detection results. Based on the extracted ROI, YOLOv5 is used to detect cigarettes within the ROI, achieving the purpose of smoking behavior recognition.

[0007] The specific steps are as follows: Step 1: Process the time-series video sequence, detect the human skeleton, define smoking posture features, design smoking posture judgment criteria, and filter out frames with suspected smoking postures.

[0008] Step 2: After initially screening out suspected smoking posture frames, define the smoking region of interest, identify faces and facial key points based on face detection algorithms, and extract the smoking region of interest using the face detection results.

[0009] Step 3: Based on the extracted smoking region of interest, use YOLOv5 to detect cigarette targets in the region of interest to achieve smoking behavior recognition.

[0010] Step 1 specifically refers to: Step 1.1: Process the time-series video sequence, extract each frame of the video image and use OpenPose to detect the human skeleton.

[0011] Step 1.2: Define the characteristics of smoking posture. The ratio of the distance between the wrist key point and the nose key point to the distance between the neck key point and the hip key point is defined as the ratio index r for judging whether a person has a suspected smoking posture. The angle between the upper arm and the forearm is defined as the angle index α for judging whether a person has a suspected smoking posture.

[0012] Step 1.3: Design smoking posture judgment criteria to determine whether there is a suspected smoking posture, and extract the frames that meet the suspected smoking posture.

[0013] The definition process of the proportional index r is as follows: Find the distance from the left hip to the key point of the neck. .

[0014] Find the distance from the right hip to the key point of the neck. .

[0015] Find the distance from the left wrist to the key point of the nose. .

[0016] Find the distance from the right wrist to the key point of the nose. .

[0017] in, The coordinates of the key point on the left hip. The coordinates of the key point on the right hip. The coordinates of key points on the left wrist. The coordinates of key points on the right wrist. The coordinates of the key points of the nose, Coordinates of key points on the neck. and average Define the proportion of people who are smoking on the left and right sides of a person, respectively. and .

[0018] The definition process of the angle index α is as follows: The direction vector from the left elbow to the left shoulder is obtained as follows: .

[0019] The direction vector from the left elbow to the left wrist is obtained as follows: .

[0020] The direction vector from the right elbow to the right shoulder is obtained as follows: .

[0021] The direction vector from the right elbow to the right wrist is obtained as follows: .

[0022] in, The coordinates of key points on the left wrist. The coordinates of key points on the right wrist. Here are the coordinates of the key point on the left elbow. The coordinates of key points on the left shoulder are shown. The coordinates are the key points of the right elbow.

[0023] Calculate the angle between the left upper arm and forearm. and the angle between the right upper arm and forearm .

[0024] The specific criteria for determining smoking posture are as follows: Step 1.3.1: When the posture detector calculates that the angle α between the forearm and upper arm is less than... If the degree is less than r0, it is considered to be in a suspected smoking posture.

[0025] Step 1.3.2: When the posture detector calculates that the angle α between the forearm and upper arm is greater than... If the degree or r is greater than r0, the person is considered to be in a non-smoking posture.

[0026] Step 2 specifically refers to: Step 2.1: Use RetinaFace to detect facial landmarks on the suspected smoking pose frames extracted in Step 1, and detect the face bounding box and the coordinates of facial landmarks.

[0027] Step 2.2: Based on the facial key point detection coordinates, the area around the mouth is cropped by expanding the length of the cigarette around the two key points at the corners of the mouth detected by the RetinaFace model.

[0028] The specific steps for setting the area of ​​interest for smoking from the mouth are as follows: Set These are the four vertices of the RetinaFace face detection bounding box. Let the four vertices be the rectangular region of interest for smoking to be captured. and The two points represent the key points at the right and left corners of the mouth detected by RetinaFace, respectively. and , They are all parallel.

[0029] set up Coordinates are , Coordinates are , Coordinates are , and The coordinates are respectively and The average length and width of the face bounding boxes detected by RetinaFace are taken as the scale of the face size. Then, based on the ratio of the face to the cigarette, the average is multiplied by a scaling factor 'a' to obtain the maximum length of the cigarette. , to the straight line , to the straight line ,straight line arrive ,straight line arrive The distance between them is .

[0030] The beneficial effects of this invention are: 1. It can identify smoking behavior in videos in real time, with high robustness and real-time performance.

[0031] 2. It combines the advantages of existing smoking recognition methods, avoiding the problems of low accuracy caused by the small size of cigarettes when using cigarette detection alone, and misjudgment caused by the similarity between human posture and smoking when using human posture alone.

[0032] 3. The suspected smoking posture based on posture detection defines a proportional index, which avoids the problem of difficulty in determining the threshold due to the uncertainty of the scale of the person in the video when defining a distance index to determine whether smoking has occurred. It is effective, reliable, and fast, meeting the requirements for preliminary judgment of suspected smoking posture frames, providing a foundation for subsequent detection, and accelerating the detection speed of the entire method proposed in this invention.

[0033] 4. Cigarette detection based on the region of interest around the mouth narrows down the detection area, which not only improves the accuracy of cigarette detection but also determines whether the smoker is actually smoking.

[0034] 5. It integrates posture features and cigarette detection methods based on regions of interest to determine smoking behavior, making the judgment of smoking behavior more reliable and reducing the false detection rate. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the smoking behavior recognition method of the present invention; Figure 2 This is the result of human key point detection in video frames according to the present invention; Figure 3 These are the keyframe extraction and face detection results of this invention; Figure 4 These are the results of the extraction of the region of interest for smoking and the detection of cigarettes in this invention; Figure 5 This is the result of human skeleton detection according to the present invention; Figure 6 This is a schematic diagram of the suspected smoking posture judgment index of the present invention; Figure 7 This is a schematic diagram of the method for determining the region of interest in smoking according to the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1: As Figure 1-7 As shown, a computer vision-based method for recognizing smoking behavior includes the following steps: Step 1: Process the time-series video sequence, detect the human skeleton, define smoking posture features, design smoking posture judgment criteria, and filter out frames with suspected smoking postures. Figure 2 The results of human skeleton detection are shown for several different frames in the test video, and frames suspected of being in smoking poses are filtered out, such as... Figure 3 As shown in the left figure.

[0038] Step 2: After initially screening out suspected smoking pose frames, define the smoking region of interest, identify faces and facial landmarks based on the RetinaFace face detection algorithm, and extract the smoking region of interest using the face detection results. Figure 3 The results of facial landmark detection for the extracted suspected smoking pose frames are shown.

[0039] Step 3: Based on the extracted smoking region of interest, use YOLOv5 to detect cigarette targets in the region of interest to achieve smoking behavior recognition. Figure 4 The extracted region of interest (ROI) for smoking and the cigarette target detection results based on the ROI are presented.

[0040] Step 1 specifically refers to: Step 1.1: Process the time-series video sequence, extract each frame image from the video, and use OpenPose to detect the human skeleton. The detection results are as follows: Figure 5 As shown.

[0041] Step 1.2: Define the characteristics of smoking posture. The ratio of the distance between the wrist key point and the nose key point to the distance between the neck key point and the hip key point is defined as the ratio index r for judging whether a person has a suspected smoking posture. The angle between the upper arm and the forearm is defined as the angle index α for judging whether a person has a suspected smoking posture.

[0042] Step 1.3: Design smoking posture judgment criteria to determine whether there is a suspected smoking posture, and extract the frames that meet the suspected smoking posture.

[0043] like Figure 5-6 As shown, the definition process of the proportional index r is as follows: Find the distance from the left hip to the key point of the neck. .

[0044] Find the distance from the right hip to the key point of the neck. .

[0045] Find the distance from the left wrist to the key point of the nose. .

[0046] Find the distance from the right wrist to the key point of the nose. .

[0047] in, The coordinates of the key point on the left hip. The coordinates of the key point on the right hip. The coordinates of key points on the left wrist. The coordinates of key points on the right wrist. The coordinates of the key points of the nose, Coordinates of key points on the neck. and average Define the proportion of people who are smoking on the left and right sides of a person, respectively. and .

[0048] like Figure 5-6 As shown, the definition process of the angle index α is as follows: The direction vector from the left elbow to the left shoulder is obtained as follows: .

[0049] The direction vector from the left elbow to the left wrist is obtained as follows: .

[0050] The direction vector from the right elbow to the right shoulder is obtained as follows: .

[0051] The direction vector from the right elbow to the right wrist is obtained as follows: .

[0052] in, The coordinates of key points on the left wrist. The coordinates of key points on the right wrist. Here are the coordinates of the key point on the left elbow. The coordinates of key points on the left shoulder are shown. The coordinates are the key points of the right elbow.

[0053] Calculate the angle between the left upper arm and forearm. and the angle between the right upper arm and forearm .

[0054] The specific criteria for determining smoking posture are as follows: Step 1.3.1: When the posture detector calculates that the angle α between the forearm and upper arm is less than... If the degree is less than r0, it is considered to be in a suspected smoking posture.

[0055] Step 1.3.2: When the posture detector calculates that the angle α between the forearm and upper arm is greater than... If the degree or r is greater than r0, the person is considered to be in a non-smoking posture.

[0056] Step 2 specifically refers to: Step 2.1: Use RetinaFace to detect facial landmarks on the suspected smoking pose frames extracted in Step 1, and detect the face bounding box and the coordinates of facial landmarks.

[0057] Step 2.2: Based on the facial key point detection coordinates, the area around the mouth is cropped by expanding the length of the cigarette around the two key points at the corners of the mouth detected by the RetinaFace model.

[0058] like Figure 7 As shown, the specific steps for setting the region of interest for smoking from the mouth are as follows: [Image of the rectangular area] This represents the bounding box of the face detected by RetinaFace. Points M1 and M2 represent the key points of the right and left corners of the mouth detected by RetinaFace, respectively. This indicates the area of ​​interest around the mouth that needs to be identified. and , They are all parallel.

[0059] set up Coordinates are , Coordinates are , Coordinates are , and The coordinates are respectively and The average length and width of the face bounding boxes detected by RetinaFace are taken as the scale of the face size. Then, based on the ratio of the face to the cigarette, the average is multiplied by a scaling factor 'a' to obtain the maximum length of the cigarette. , to the straight line , to the straight line ,straight line arrive ,straight line arrive The distance between them is .

[0060] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A computer vision-based method for recognizing smoking behavior, characterized in that: Step 1: Process the time-series video sequence, detect the human skeleton, define smoking posture features, design smoking posture judgment criteria, and filter out frames with suspected smoking postures. Step 2: After initially screening out suspected smoking posture frames, define the smoking region of interest, identify faces and facial key points based on face detection algorithms, and extract the smoking region of interest using the face detection results; Step 3: Based on the extracted smoking region of interest, use YOLOv5 to detect cigarette targets in the region of interest to achieve smoking behavior recognition; Step 1 specifically refers to: Step 1.1: Process the time-series video sequence, extract each frame of the video image and use OpenPose to detect the human skeleton; Step 1.2: Define the characteristics of smoking posture. The ratio of the distance between the wrist key point and the nose key point to the distance between the neck key point and the hip key point is defined as the ratio index r for judging whether a person has a suspected smoking posture. The angle between the upper arm and the forearm is defined as the angle index α for judging whether a person has a suspected smoking posture. Step 1.3: Design smoking posture judgment criteria to determine whether there is a suspected smoking posture, and extract the frames that meet the suspected smoking posture criteria; The definition process of the proportional index r is as follows: Find the distance from the left hip to the key point of the neck. ; Find the distance from the right hip to the key point of the neck. ; Find the distance from the left wrist to the key point of the nose. ; Find the distance from the right wrist to the key point of the nose. ; in, The coordinates of the key point on the left hip. The coordinates of the key point on the right hip. The coordinates of key points on the left wrist. The coordinates of key points on the right wrist. The coordinates of the key points of the nose. Coordinates of key points on the neck. and average Define the proportion of people who are smoking on the left and right sides of a person, respectively. and ; The definition process of the angle index α is as follows: The direction vector from the left elbow to the left shoulder is obtained as follows: ; The direction vector from the left elbow to the left wrist is obtained as follows: ; The direction vector from the right elbow to the right shoulder is obtained as follows: ; The direction vector from the right elbow to the right wrist is obtained as follows: ; in, The coordinates of key points on the left wrist. The coordinates of key points on the right wrist. Here are the coordinates of the key point on the left elbow. The coordinates of key points on the left shoulder are shown. The coordinates of the key point on the right elbow; Calculate the angle between the left upper arm and forearm. and the angle between the right upper arm and forearm ; The criteria for judging smoking posture are specifically as follows: Step 1.3.1: When the posture detector calculates that the angle α between the forearm and upper arm is less than... If the degree is less than r0, it is considered to be in a suspected smoking posture; Step 1.3.2: When the posture detector calculates that the angle α between the forearm and upper arm is greater than... If the degree or r is greater than r0, the person is considered to be in a non-smoking posture. Step 2 specifically refers to: Step 2.1: Use RetinaFace to detect facial landmarks on the suspected smoking pose frames extracted in Step 1, and detect the face bounding box and the coordinates of facial landmarks; Step 2.2: Based on the facial key point detection coordinates, the area around the mouth is cropped by expanding the length of the cigarette around the two key points at the corners of the mouth detected by the RetinaFace model. The specific steps for setting the area of ​​interest for smoking from the mouth are as follows: Set These are the four vertices of the RetinaFace face detection bounding box. Let the four vertices be the rectangular region of interest for smoking to be captured. and The two points represent the key points at the right and left corners of the mouth detected by RetinaFace, respectively. and , They are all parallel; set up Coordinates are , Coordinates are , Coordinates are , and The coordinates are respectively and The average length and width of the face bounding boxes detected by RetinaFace are taken as the scale of the face size. Then, based on the ratio of the face to the cigarette, the average is multiplied by a scaling factor 'a' to obtain the maximum length of the cigarette. , to the straight line , to the straight line ,straight line arrive ,straight line arrive The distance between them is .

Images