Smoking behavior detection method, electronic device and readable storage medium

Abstract

The present application discloses a smoking behavior detection method, device and computer-readable storage medium, which includes: obtaining an image to be tested, and inputting the image to be tested into a target detection model with cigarettes as the target to determine the suspected cigarette range; identifying multiple facial key points in the image to be tested, and using the facial key points to determine the mouth range; calculating a matching parameter between the suspected cigarette range and the mouth range, wherein the matching parameter is a target distance between the suspected cigarette range and the mouth range, and / or an intersection area between the mouth range and the suspected cigarette range; if the matching parameter is within a preset range, it is determined that smoking behavior exists in the image to be tested; judging whether smoking behavior exists based on the distance between the cigarette and the human mouth in the image to be tested, fewer factors can interfere with the judgment result, and whether smoking behavior exists can be accurately identified, thereby improving the reliability and accuracy of smoking behavior detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a smoking behavior detection method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Short video platforms are incredibly popular these days, with users uploading a massive number of videos daily. Many videos depict smoking, severely impacting online order and posing a significant threat to vulgarity. To quickly identify and flag these videos within this vast sea of content, related technologies employ a human skeleton keypoint location algorithm to identify human motion within an image and determine whether it represents a smoking action. If so, smoking is detected. However, this detection method is susceptible to interference from other motions, resulting in poor robustness and low accuracy and reliability. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a smoking behavior detection method, electronic device and computer-readable storage medium, which can accurately identify whether smoking behavior exists, thereby improving the reliability and accuracy of smoking behavior detection.

[0004] To solve the above technical problems, in a first aspect, the present application provides a smoking behavior detection method, comprising:

[0005] Acquire an image to be tested, and input the image to be tested into an object detection model with cigarettes as targets to determine the range of suspected cigarettes;

[0006] Identifying a plurality of facial key points in the image to be tested, and determining the range of the mouth using the facial key points;

[0007] Calculating a matching parameter between the suspected cigarette range and the mouth range, wherein the matching parameter is a target distance between the suspected cigarette range and the mouth range, and / or an intersection area between the mouth range and the suspected cigarette range;

[0008] If the matching parameter is within a preset range, it is determined that smoking behavior exists in the image to be tested.

[0009] Optionally, it also includes:

[0010] Acquire a video to be tested, and determine a number of candidate video frames in the video to be tested;

[0011] Accordingly, the step of acquiring the image to be tested includes:

[0012] Selecting one of the candidate video frames as the image to be tested according to a preset order;

[0013] The smoking behavior detection method further includes:

[0014] If it is detected that all the candidate video frames have been detected and the number of abnormal video frames with the smoking behavior is greater than the abnormal threshold, the video to be tested is determined to be an abnormal video.

[0015] Optionally, after inputting the image to be tested into an object detection model with cigarettes as the target, the method further includes:

[0016] If the target detection model output determines the cigarette range, it is determined that smoking behavior exists in the image to be tested.

[0017] Optionally, it also includes:

[0018] Performing gesture semantic recognition processing on the image to be tested to obtain a gesture recognition result;

[0019] Correspondingly, if the matching parameter is within a preset range, determining that smoking behavior exists in the image to be tested includes:

[0020] If the matching parameter is within the preset range and the gesture recognition result is a hit, it is determined that the smoking behavior exists in the image to be tested.

[0021] Optionally, performing gesture semantic recognition processing on the image to be tested to obtain a gesture recognition result includes:

[0022] Deleting the non-skin color area in the image to be tested to obtain the skin color area;

[0023] Performing edge extraction processing on the skin color area to obtain a contour image;

[0024] The contour image is input into a gesture classification model to obtain the gesture recognition result.

[0025] Optionally, deleting the non-skin color area in the image to be tested to obtain the skin color area includes:

[0026] Determine whether the channel pixel value of each pixel in each channel of the image to be tested is within the corresponding channel interval;

[0027] If the pixel values of each channel are all within the channel interval, it is determined that the pixel is within the initial skin color area;

[0028] The initial skin color area is subjected to corrosion processing and dilation processing, and the image in the non-skin color area outside the skin color area is deleted to obtain the skin color area.

[0029] Optionally, inputting the image to be detected into a target detection model with cigarettes as targets to determine the range of suspected cigarettes includes:

[0030] Inputting the image to be tested into an object detection model with cigarettes as targets to obtain coordinates of suspected cigarettes;

[0031] Determine range length and width parameters, and based on the range length and width parameters, determine the suspected cigarette range using the suspected cigarette coordinates.

[0032] Optionally, the training process of the target detection model includes:

[0033] Acquire a plurality of initial training images, and perform data augmentation processing on the initial training images to obtain training images;

[0034] Inputting the training image into the initial model to obtain a training recognition range;

[0035] A range distance between the training recognition range and the label range of the training image is calculated, and the model parameters of the initial model are adjusted using the range distance.

[0036] Optionally, performing data augmentation processing on the initial training image to obtain a training image includes:

[0037] Performing random cropping and / or random scaling on any plurality of the initial training images to obtain a plurality of target initial training images;

[0038] The multiple target initial training images are randomly arranged and spliced to obtain one training image.

[0039] Optionally, the identifying a plurality of facial key points in the image to be tested and determining the mouth range using the facial key points includes:

[0040] Performing facial key point detection on the image to be tested to obtain facial key points;

[0041] Connecting lines between the key points of the face according to a preset rule to obtain a plurality of candidate ranges;

[0042] The mouth key points are determined according to the key point information of the facial key points, and the mouth range is generated using the candidate range corresponding to the mouth key points.

[0043] In a second aspect, the present application further provides an electronic device, including a memory and a processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is configured to execute the computer program to implement the above-mentioned smoking behavior detection method.

[0046] In a third aspect, the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above-mentioned smoking behavior detection method when executed by a processor.

[0047] The smoking behavior detection method provided in the present application obtains an image to be tested, and inputs the image to be tested into a target detection model with cigarettes as the target to determine the suspected cigarette range; identifies multiple facial key points in the image to be tested, and uses the facial key points to determine the mouth range; calculates the matching parameters between the suspected cigarette range and the mouth range, wherein the matching parameters are the target distance between the suspected cigarette range and the mouth range, and / or the intersection area between the mouth range and the suspected cigarette range; if the matching parameters are within a preset range, it is determined that smoking behavior exists in the image to be tested.

[0048] As can be seen, after acquiring the image to be tested, this method uses an object detection model to perform target detection on it. The target detection model uses cigarettes as targets, so the detection result indicates whether an object suspected of being a cigarette exists in the image to be tested. If the detection result indicates a suspected cigarette range, it indicates that an object that may be a cigarette exists in the image to be tested, and further detection is required to determine whether smoking behavior is present. Through facial key point detection, the mouth range of the face in the image to be tested can be determined. Smoking behavior inevitably causes the cigarette to come into contact with or be close to the mouth. Therefore, a matching parameter can be calculated between the suspected cigarette range and the mouth range. This matching parameter can specifically represent the distance between the cigarette and the mouth in the image to be tested, or it can represent the overlap area between the suspected cigarette range and the mouth range, i.e., the intersection area. If the matching parameter is within a preset range, it indicates that the object is very close to the mouth, or is in contact. Alternatively, if the suspected cigarette range and the mouth range overlap significantly, this also indicates that the two are in contact or are in close proximity. Since other similar situations are rare, in this case, smoking behavior can be determined in the image to be tested. This method determines whether there is smoking behavior from the perspective of the distance between the cigarette and the human mouth. There are fewer factors that can interfere with the judgment result, and it can accurately identify whether there is smoking behavior. It improves the reliability and accuracy of smoking behavior detection and solves the problem of low accuracy and reliability of smoking behavior detection in related technologies.

[0049] In addition, the present application also provides an electronic device and a computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0051] Figure 1 A schematic diagram of a hardware composition framework applicable to a smoking behavior detection method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a hardware composition framework applicable to another smoking behavior detection method provided in an embodiment of the present application;

[0053] Figure 3 A flowchart of a smoking behavior detection method provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of another smoking behavior detection method provided in an embodiment of the present application;

[0055] Figure 5 An image to be tested provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a selection range provided in an embodiment of the present application;

[0057] Figure 7 A skin color area provided in an embodiment of the present application;

[0058] Figure 8 A contour image provided in an embodiment of the present application;

[0059] Figure 9 A flowchart of a specific smoking behavior detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] For ease of understanding, the hardware framework used in the solution corresponding to the smoking behavior detection method provided in the embodiment of the present application is first introduced. Figure 1 , Figure 1 A schematic diagram of the hardware architecture applicable to a smoking behavior detection method provided in an embodiment of the present application. The electronic device 100 may include a processor 101 and a memory 102, and may further include one or more of a multimedia component 103, an information input / output (I / O) interface 104, and a communication component 105.

[0062] The processor 101 is used to control the overall operation of the electronic device 100 to complete all or part of the steps in the smoking behavior detection method. The memory 102 is used to store various types of data to support the operation of the electronic device 100. This data may include, for example, instructions for any application or method operating on the electronic device 100, as well as application-related data. The memory 102 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. In this embodiment, the memory 102 stores at least programs and / or data for implementing the following functions:

[0063] Acquire an image to be tested, and input the image to be tested into an object detection model with cigarettes as targets to determine the range of suspected cigarettes;

[0064] Identifying a plurality of facial key points in the image to be tested, and determining the range of the mouth using the facial key points;

[0065] Calculating a matching parameter between the suspected cigarette range and the mouth range, wherein the matching parameter is a target distance between the suspected cigarette range and the mouth range, and / or an intersection area between the mouth range and the suspected cigarette range;

[0066] If the matching parameter is within a preset range, it is determined that smoking behavior exists in the image to be tested.

[0067] The multimedia component 103 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 102 or sent through the communication component 105. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 104 provides an interface between the processor 101 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 105 is used for wired or wireless communication between the electronic device 100 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 105 may include: a Wi-Fi component, a Bluetooth component, an NFC component.

[0068] The electronic device 100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the smoking behavior detection method.

[0069] certainly, Figure 1 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device 100 may include Figure 1 More or fewer components than shown, or combinations of certain components.

[0070] It is understandable that the number of electronic devices is not limited in the embodiments of the present application, and multiple electronic devices may work together to complete the smoking behavior detection method. In a possible implementation, please refer to Figure 2 , Figure 2 This is a schematic diagram of the hardware composition framework applicable to another smoking behavior detection method provided in the embodiment of the present application. Figure 2It can be seen that the hardware composition framework may include: a first electronic device 11 and a second electronic device 12 , which are connected via a network 13 .

[0071] In the embodiment of the present application, the hardware structure of the first electronic device 11 and the second electronic device 12 can refer to Figure 1 In other words, the electronic device 100 in this embodiment has two electronic devices 100, and the two devices exchange data. Furthermore, the embodiment of the present application does not limit the form of the network 13, that is, the network 13 can be a wireless network (such as WIFI, Bluetooth, etc.) or a wired network.

[0072] Among them, the first electronic device 11 and the second electronic device 12 can be the same electronic device, such as the first electronic device 11 and the second electronic device 12 are both servers; or they can be different types of electronic devices, for example, the first electronic device 11 can be a smartphone or other smart terminal, and the second electronic device 12 can be a server. In one possible implementation, a server with strong computing power can be used as the second electronic device 12 to improve data processing efficiency and reliability, thereby improving the processing efficiency of smoking behavior detection. At the same time, a smartphone with low cost and wide application range is used as the first electronic device 11 to realize the interaction between the second electronic device 12 and the user. It can be understood that the interaction process can be: the smartphone obtains the image to be tested, and sends the image to be tested to the server, and the server detects the smoking behavior. The server sends the detection results to the smartphone, and the smartphone displays the detection results.

[0073] Based on the above description, please refer to Figure 3 , Figure 3 A flowchart of a smoking behavior detection method provided in an embodiment of the present application is provided. The method in this embodiment includes:

[0074] S101: Acquire an image to be tested, and input the image to be tested into a target detection model with cigarettes as the target to determine the range of suspected cigarettes.

[0075] The images to be tested refer to images that need to be tested to see if they record smoking behavior. There is no limit on their number and type. The target detection model refers to a model that can identify target objects in images after training and give a prediction box (i.e., the location information of the target object). There is no limit on its structure and type. For example, it can be a convolutional neural network model, specifically a YOLOv5 model, so as to improve the detection accuracy and reliability of target detection. The YOLOv5 model is the fifth-generation algorithm in the YOLO (full name: You Only Look Once) target detection algorithm series. It was proposed in 2020 and has greatly improved in accuracy, sensitivity, and effectiveness compared to the previous four generations.

[0076] In this embodiment, the target detection model targets cigarettes, that is, the target objects it can identify are cigarettes. Therefore, after the image to be tested is input into the target detection model, if the detection result obtained is a hit, it may indicate that a cigarette has been detected, or an object suspected to be a cigarette has been detected. This embodiment is limited only to the case where an object suspected to be a cigarette is detected. For other cases, the corresponding operation can be selected as needed. Specifically, the target detection model targets cigarettes, so it will identify any object suspected to be a cigarette in the image to be tested, and there are many situations as to whether an identified object is a cigarette or how likely it is to be a cigarette. It can be understood that if it is determined that a cigarette is detected in the image to be tested, it can be directly determined that there is smoking behavior in the image to be tested without further detection. Therefore, in one embodiment, after the image to be tested is input into the target detection model targeting cigarettes, the following steps may also be included:

[0077] Step 11: If the target detection model output determines the cigarette range, it is determined that smoking behavior exists in the image to be tested.

[0078] Determining the cigarette range refers to the range of cigarettes in the image to be tested. It can be understood that if the target monitoring model determines that a cigarette has been detected, for example, the confidence level of the detected object suspected of being a cigarette is greater than the confidence threshold, then the suspected object can be directly determined to be a cigarette, and the range of the cigarette can be determined and output. The target detection model can assign a special identifier to its output content to indicate the category of the output content, that is, whether it is a determined cigarette range or a suspected cigarette range. If it is detected that the target detection model outputs a determined cigarette range, it can be directly determined that smoking behavior exists in the image to be tested, and the detection of the image to be tested is completed.

[0079] In another case, the target detection model cannot confirm whether the suspected target it detects is a cigarette. In this case, the suspected cigarette range can be output. The suspected cigarette range also includes the position information of the object suspected to be a cigarette. The suspected cigarette range refers to the position range of the cigarette in the image to be tested, which is usually a rectangle that frames the object. As for the method of determining the suspected cigarette range, in one embodiment, the detection result directly includes specific information of the suspected cigarette range, such as vertex coordinates, etc. In this case, the suspected cigarette range can be obtained by directly reading the detection result. In a second embodiment, the detection result only includes reference information for determining the suspected cigarette range, such as the center position coordinates of the suspected cigarette range. In this case, it is necessary to use the detection result and other preset information to jointly determine the suspected cigarette range.

[0080] Furthermore, if the detection result is a miss, it means that the target detection model did not detect the object suspected to be a cigarette, and therefore does not include the above-mentioned location information. Regarding the judgment of whether the detection result is a hit, in one embodiment, the target detection model can directly output a hit and the corresponding location information to indicate the output of the suspected cigarette range; or directly output a determination and the corresponding location information to indicate the determination of the cigarette range, or directly output a miss. In another embodiment, the target detection model can directly output the location information of the object suspected to be a cigarette and the corresponding confidence level, and when the confidence level is within a credible interval, determine that the detection result is a hit, and the location information is the location information corresponding to the object with the confidence level within the credible interval; when the confidence level is within the determination interval, determine that the detection result is the determination of the cigarette range, and the location information is the location information corresponding to the object with the confidence level within the determination interval. Correspondingly, when the confidence level is not within the credible interval, it means that the object corresponding to the location information cannot be a cigarette, so no location information is output, and only a miss detection result is output.

[0081] S102: Identify multiple facial key points in the image to be tested, and use the facial key points to determine the range of the mouth.

[0082] Since smoking requires the cigarette and the mouth to be coordinated, when an object suspected of being a cigarette is detected, it cannot be directly determined that smoking has been detected. Instead, it is necessary to further detect whether the object suspected of being a cigarette is in contact with or close to the mouth.

[0083] Specifically, facial key point detection is performed on the image to be tested to identify the facial key points therein. The facial key points can be set as needed, for example, including mouth key points, nose key points, contour key points, eye key points, eyebrow key points, etc. The number of each type of key points is not limited. Regarding the method of identifying facial key points, in one embodiment, detection can be based on the Dlib library. The Dlib library is an open source library for machine learning that contains many machine learning algorithms, including a prediction algorithm for the 68 facial key points.

[0084] After obtaining the facial key points, they can be used to determine the mouth range. In one embodiment, key points can be filtered to obtain mouth key points, and the range within which the mouth key points are located can then be determined as the mouth range. In another embodiment, to avoid incorrect mouth region determination due to recognition errors, a preset range can be determined based on the distribution of facial key points. The distribution of facial key points corresponding to a given face, such as density and distance, can, to a certain extent, indicate the size of the facial image in the image to be tested. Once the facial image size is determined, the size of the mouth should also fall within a certain range. Therefore, after filtering the key points to obtain the mouth key points, the mouth key points are used to construct an initial mouth range. It is then determined whether the size of the initial mouth range falls within the aforementioned range, for example, whether the area of the mouth range falls within the aforementioned range. If so, the initial mouth range is determined to be the mouth range. This embodiment does not limit the operation performed when the initial mouth range does not fall within the aforementioned range. For example, key point recognition can be performed again on the entire image to be tested, or on the facial image corresponding to the key points.

[0085] S103: Calculate matching parameters between the suspected cigarette range and the mouth range.

[0086] It should be noted that the matching parameters in this application are the target distance between the suspected cigarette range and the mouth range, and / or the intersection area between the mouth range and the suspected cigarette range. That is, the matching parameters may include only the target distance, only the intersection area, or both the target distance and the intersection area. After the suspected cigarette range and the mouth range are determined, the matching parameters between the suspected cigarette range and the mouth range can be calculated. Taking the target distance as an example, the target distance can represent the distance between the cigarette and the mouth, and thus can indicate whether smoking behavior is detected. In one embodiment, the target distance refers to the minimum distance between any point within the suspected cigarette range and any point within the mouth range. In a second embodiment, the target distance refers to the minimum distance between a specified point within the suspected cigarette range and any point within the mouth range. In a third embodiment, the target distance refers to the minimum distance between a specified point within the mouth range and any point within the suspected cigarette range. It will be understood that the calculation method will vary depending on the specific meaning of the target distance.

[0087] The intersection area refers to the overlapping area between the mouth range and the suspected cigarette range, and its specific calculation method is not limited. For example, when the mouth range and the suspected cigarette range are both rectangular frames, the length and width of the overlapping portion between the two can be determined, and the area can be obtained by multiplying the length and width. In another embodiment, when there is a non-rectangular shape in the mouth range and the suspected cigarette range, a rectangular coordinate system can be constructed, and the corresponding segmented curve or non-segmented curve of each side of the overlapping portion can be determined. For example, a fitting curve can be obtained by fitting, and the intersection area can be obtained by integration. It can be understood that the larger the intersection area, the closer the distance between the mouth and the cigarette, or the stronger the contact, and the more likely smoking behavior will occur.

[0088] It should be noted that since the image under test may contain multiple facial images, there may be multiple corresponding mouth regions. Similarly, the image under test may also contain multiple cigarette images. Therefore, when the target detection model detects a hit, it can be used to determine one or more suspected cigarette regions. Therefore, when calculating the suspected cigarette regions, it is necessary to calculate the matching parameters between each mouth region and each suspected cigarette region separately, and then perform the subsequent steps for each matching parameter separately.

[0089] S104: Determine whether the matching parameter is within a preset range.

[0090] S105: If the matching parameter is within the preset range, it is determined that smoking behavior exists in the image to be tested.

[0091] After calculating the matching parameters, a determination is made as to whether the matching parameters are within a preset range. The preset range refers to the distance range indicating a relatively close distance between the cigarette and the mouth. Therefore, if the matching parameters are within the preset range, it indicates that the cigarette and the mouth in the image under test are relatively close or in contact, and therefore, smoking activity can be detected in the image under test.

[0092] S106: Preset operation.

[0093] This embodiment does not limit the specific content of the preset operation. For example, it may be no operation, or it may be outputting a detection result indicating that no smoking behavior is detected, or it may be reacquiring a new image to be detected.

[0094] Using the smoking behavior detection method provided in the embodiments of the present application, after acquiring a test image, target detection is performed on it using a target detection model. The target detection model uses cigarettes as targets, and the resulting detection result indicates whether an object suspected of being a cigarette exists in the test image. If the detection result indicates a suspected cigarette range, this indicates that an object potentially resembling a cigarette exists in the test image, and further detection is required to determine whether smoking behavior is present. Facial key point detection can be used to determine the mouth range of the face in the test image. Smoking inevitably causes the cigarette to come into contact with or be close to the mouth, so a matching parameter can be calculated between the suspected cigarette range and the mouth range. This matching parameter can specifically represent the distance between the cigarette and the mouth in the test image, or it can represent the overlap area between the suspected cigarette range and the mouth range, i.e., the intersection area. If the matching parameter is within a preset range, it indicates that the object is very close to the mouth, or is in contact. Alternatively, if the suspected cigarette range and the mouth range overlap significantly, this also indicates that the two are in contact or are relatively close. Since other similar situations are rare, in this case, smoking behavior can be determined in the test image. This method determines whether there is smoking behavior from the perspective of the distance between the cigarette and the human mouth. There are fewer factors that can interfere with the judgment result, and it can accurately identify whether there is smoking behavior. It improves the reliability and accuracy of smoking behavior detection and solves the problem of low accuracy and reliability of smoking behavior detection in related technologies.

[0095] Based on the above embodiment, this embodiment specifically describes some of the steps in the above embodiment. In one embodiment, the target detection model can be trained on other electronic devices and directly sent to the device that performs the various steps of this application for application. In another embodiment, the target detection model can be trained on the device itself, and the training process of the target detection model may include the following steps:

[0096] Step 21: Obtain multiple initial training images, and perform data augmentation processing on the initial training images to obtain training images.

[0097] Step 22: Input the training image into the initial model to obtain the training recognition range.

[0098] Step 23: Calculate the range distance between the training recognition range and the label range of the training image, and use the range distance to adjust the model parameters of the initial model.

[0099] The initial training image refers to the basic image used for data augmentation processing. Data augmentation (DataAugmentation) can generate more data from limited data, thereby increasing the number and diversity of training samples (i.e., obtaining some negative sample data), thereby improving the robustness of the model. This application does not limit the specific method of data augmentation, and the applicable data augmentation algorithm can be selected as needed. After obtaining the training image, it can be input into the initial model. In one embodiment, in order to match the training image with the initial model, the training image can be scaled during input. The initial model refers to a model that has not reached convergence, and it is transformed into a target recognition model after iterative training reaches convergence. After the initial model processes the training image, the output result is the training recognition range. The training recognition range refers to the position range obtained after the initial model predicts the position of the cigarette in the training image.

[0100] The label range refers to the label data of the training image, which can also be called the true frame (i.e. GroundTruth), which is specifically the real position range of the cigarette in the training image, or when there is no cigarette in the training image, the label range is an empty range, that is, a range without an identification. After obtaining the training recognition range, the range distance between the remaining label ranges is calculated. Since the level of recognition accuracy can be directly reflected by the distance between the training recognition range and the label range obtained by recognition, the larger the distance, the lower the accuracy, and the smaller the distance, the higher the accuracy. Therefore, after obtaining the range distance, the model parameters of the initial model can be adjusted based on it. The specific adjustment process is not limited. For example, the loss value can be calculated based on the range distance, and the model parameters can be adjusted based on the loss value.

[0101] Furthermore, in order to improve the target recognition model's ability to recognize images of various backgrounds, that is, to improve the target recognition model's adaptability to images under various circumstances, the present application may employ a method based on the Mosaic data enhancement method to perform data enhancement processing. Specifically, the process of performing data enhancement processing on the initial training image to obtain the training image may specifically include the following steps:

[0102] Step 31: Perform random cropping and / or random scaling on any multiple initial training images to obtain multiple target initial training images.

[0103] Step 32: Randomly arrange and splice multiple target initial training images to obtain a training image.

[0104] The Mosaic data augmentation method randomly crops four images and then stitches them together into a single training image. This method creates a training image with rich backgrounds. By using these images to train the object detection model, the model becomes more adaptable to images and can accurately and stably recognize images in a variety of situations.

[0105] In this embodiment, the idea of the Mosaic data augmentation method is adopted, but the number of initial training images involved in the stitching is not limited. Specifically, the present application can obtain several processed target initial training images after random cropping and random scaling of at least two initial training images. By randomly arranging and stitching them together, the training images can be obtained. It should be noted that in addition to random cropping and random scaling, other processing can be performed on the initial training images, and the specific processing method is not limited. For example, the color value of the image can be adjusted.

[0106] Based on the above embodiment, in one implementation, the images to be tested can be individual video frames in a video to be tested. By performing smoking behavior detection on the video frames, the video to be tested can be detected and whether the video to be tested is abnormal can be determined. Specifically, the following steps can be included:

[0107] Step 41: Obtain a video to be tested, and determine a number of candidate video frames in the video to be tested.

[0108] The candidate video frames can be any video frame in the video to be tested. The number of candidate video frames is typically less than the total number of video frames in the video to be tested, thereby improving the speed and efficiency of detecting the video to be tested. This embodiment does not limit the specific method for determining the candidate video frames. In one embodiment, several candidate video frames can be sequentially acquired starting from the first frame of the video to be tested at a preset video frame interval. Alternatively, candidate video frames can be acquired by random sampling.

[0109] Accordingly, the process of acquiring the image to be tested may include the following steps:

[0110] Step 42: Select a candidate video frame as the image to be tested according to a preset order.

[0111] The preset order refers to the order in which the images to be tested are selected from the candidate video frames, and its specific content is not limited. Specifically, the preset order can be the order of ascending frame numbers or the order of descending frame numbers.

[0112] Accordingly, the smoking behavior detection method further includes the following steps:

[0113] Step 43: If all candidate video frames are detected and the number of abnormal video frames with smoking behavior is greater than the abnormal threshold, the video to be tested is determined to be an abnormal video.

[0114] Abnormal video frames refer to candidate video frames that contain smoking behavior. After all candidate video frames of the video to be tested are detected, if the number of abnormal video frames is large, that is, greater than the abnormality threshold, the video to be tested can be determined to be an abnormal video. The abnormality threshold can be a fixed value, that is, the corresponding abnormality threshold is the same for different videos to be tested. The abnormality threshold can also be a dynamic value, that is, the specific size of the abnormality threshold varies based on different videos to be tested. For example, the abnormality threshold can be determined according to a fixed ratio and the number of video frames in the video to be tested.

[0115] After obtaining the image to be tested, in this embodiment, the YOLOv5 model is used as the target detection model. Therefore, the image to be tested is input into the target detection model with cigarettes as the target. The process of obtaining the detection result specifically includes the following steps:

[0116] 1. Input the image to be tested into the backbone network in the target detection model to obtain the feature map.

[0117] 2. Input the feature map into the feature extraction network in the target detection model to obtain feature parameters.

[0118] 3. Input the feature parameters into the prediction network in the target detection model to obtain the detection results.

[0119] When the image to be tested is input into the object detection model, the backbone network is first used to extract its feature map. The backbone network is a convolutional neural network that aggregates and forms image feature maps at different image granularity levels. Typically, the backbone network produces a feature map of size 512*20*20.

[0120] The feature extraction network consists of a series of layers used to extract mixed and combined image features. This network then passes image features to the prediction network, which processes the feature maps to obtain feature parameters. In the YOLOv5 model, the feature extraction network is specifically the PANet (Path Aggregation Network). The specific structure and operation of the PANet network can be found in related technologies and will not be detailed here.

[0121] Among them, the prediction network refers to a network used to generate detection results based on feature parameters. It can predict the category of the image, that is, hit or non-hit. It is also used to generate position information that can represent the location of the cigarette image when the image category is determined to be a hit, and use the position information and category information together to form the detection result.

[0122] When the detection result is a suspected cigarette range, it is necessary to perform facial key point recognition and further determine the mouth range. The process of identifying multiple facial key points in the image to be tested and using the facial key points to determine the mouth range may include the following steps:

[0123] Step 51: Detect facial key points on the image to be tested to obtain facial key points.

[0124] Step 52: Connect the key points of the face according to the preset rules to obtain several candidate ranges.

[0125] Step 53: Determine the mouth key points based on the key point information of the facial key points, and generate the mouth range using the candidate range corresponding to the mouth key points.

[0126] The preset rules refer to the rules for connecting lines based on facial key points. The specific content is not limited. For example, you can set which other facial key points a certain facial key point needs to be connected to, or which other facial key points a certain facial key point should not be connected to. Specifically, after obtaining the facial key points, you need to connect lines between the facial key points. By connecting lines and based on the same type of key points, you can circle the various ranges on the face, that is, obtain several candidate ranges, so that you can subsequently determine the mouth range. For details, please refer to Figure 5 , Figure 5 The embodiment of the present application provides an image to be tested, and the facial key points are detected. After the facial key points are detected and connected, the following is obtained: Figure 6 There are several candidate ranges shown. Figure 6 The selected ranges shown include a left eyebrow range, a right eyebrow range, a right eye range, a nose range, a mouth range, and a contour range.

[0127] Keypoint information refers to information uniquely associated with each keypoint, such as a keypoint serial number or keypoint name. Using this keypoint information, the mouth keypoint can be determined from the facial keypoints, and the mouth range can then be generated using the candidate range corresponding to the mouth keypoint.

[0128] Regarding the method of generating the mouth range, in one embodiment, the candidate range corresponding to the mouth key point can be directly determined as the mouth range. In another embodiment, the mouth range can be generated based on the candidate range corresponding to the mouth key point. For example, when the mouth range is set as a rectangular frame, the maximum coordinates of the candidate range in the four directions of the rectangular frame can be taken, and a rectangular frame can be constructed based on the four maximum coordinates. The range within the rectangular frame is the mouth range.

[0129] After determining the mouth range, it is necessary to determine the suspected cigarette range based on the detection results. Specifically, the image to be tested is input into the target detection model with cigarettes as the target. The process of determining the suspected cigarette range may include the following steps:

[0130] Step 61: Input the image to be tested into the target detection model with cigarettes as the target to obtain the coordinates of the suspected cigarettes.

[0131] Step 62: Determine the range length and width parameters, and based on the range length and width parameters, determine the suspected cigarette range using the suspected cigarette coordinates.

[0132] In this embodiment, the suspected cigarette range is a rectangular frame, and the length and width of the rectangular frame are limited by the range length and width parameters. The range length and width parameters refer to the parameters that set the length and width of the suspected cigarette range, and their specific values are not limited.

[0133] After the image to be tested is passed through the Yuyu target detection model, the coordinates of the suspected cigarette are obtained. The suspected cigarette coordinates refer to the reference coordinates used to determine the range of the suspected cigarette. Specifically, they can be the coordinates of any specified point in the suspected cigarette range, such as the upper left corner of the suspected cigarette range, or the center point of the suspected cigarette range. Under the premise that the range length and width parameters are set, different suspected cigarette ranges can be circled in the image to be tested based on different detection coordinates. The target detection model can determine the suspected cigarette range by identifying the detection coordinates corresponding to the cigarette in the image to be tested and adding them to the detection results.

[0134] Furthermore, in order to further improve the detection accuracy, the present application can also recognize the gestures in the image. Specifically, it can also include the following steps:

[0135] Step 71: Perform gesture semantic recognition processing on the image to be tested to obtain a gesture recognition result.

[0136] Correspondingly, if the matching parameter is within the preset range, it is determined that smoking behavior exists in the image to be tested, including:

[0137] Step 72: If the matching parameter is within the preset range and the gesture recognition result is a hit, it is determined that smoking behavior exists in the image to be tested.

[0138] Gesture semantic recognition refers to the process of identifying the type of gesture in an image. By performing gesture semantic recognition on the image to be tested, it can be determined whether the gesture recorded in the image to be tested is related to smoking behavior, for example, whether there is the behavior of fingers pinching a cigarette. Specifically, the process of performing gesture semantic recognition on the image to be tested and obtaining the gesture recognition result may include the following steps:

[0139] Step 81: Delete the non-skin color area in the image to be tested to obtain the skin color area.

[0140] Step 82: Perform edge extraction on the skin color area to obtain a contour image.

[0141] Step 83: Input the contour image into the gesture classification model to obtain a classification result.

[0142] Because hand gestures are the gestures of the human hand, images recording these gestures must fall within the skin color area. The skin color area refers to the area of the image to be tested that meets the human skin color requirements. By removing the non-skin color area from the image to be tested, leaving only the image within the skin color area, the skin color area can be obtained. Before removing the non-skin color area from the image to be tested, the image to be tested can be subjected to denoising, specifically Gaussian filtering or median filtering, to remove noise.

[0143] This embodiment does not limit the specific process of generating the skin color area. In one embodiment, the process of deleting the non-skin color area in the image to be tested to obtain the skin color area may include the following steps:

[0144] Step 91: Determine whether the channel pixel value of each pixel in each channel of the image to be tested is within the corresponding channel interval.

[0145] Step 92: If the pixel values of each channel are all within the channel interval, it is determined that the pixel is in the initial skin color area.

[0146] Step 93: Perform corrosion and dilation processing on the initial skin color area, and delete the image in the non-skin color area outside the skin color area to obtain the skin color area.

[0147] To determine whether each pixel in the image to be tested represents skin color, specifically, it can be determined whether the channel pixel values corresponding to the pixels in each color channel are within the channel interval. There are multiple color channels, such as three channels R, G, and B, which represent red, green, and blue, respectively. The channel pixel values in each color channel work together to obtain the pixel value corresponding to the pixel. Since a person's skin color varies within a certain range, for a color channel, there is a channel interval, and the colors formed by it and the channel intervals corresponding to other color channels are all skin colors. When the channel pixel values corresponding to a certain pixel are all within the corresponding channel intervals, it is determined that the pixel is in the initial skin color area.

[0148] In fact, steps 92 to 93 can be understood as a binarization process based on skin color, that is, when the pixel value corresponding to a pixel is not skin color, it is set to pure black or pure white.

[0149] After determining the initial skin color area, the burrs on the edges can be removed through corrosion and expansion processing, making the initial skin color area smoother. The area after processing is the skin color area. The skin color area can be obtained by deleting the image outside the skin color area. Figure 7 , Figure 7 A skin color area provided in an embodiment of the present application.

[0150] The specific posture of a hand gesture can be identified by the outline of the hand. Therefore, the edge extraction process can be performed on the skin color area to obtain the outline image of the human hand. This image is then input into the gesture classification model to obtain the corresponding classification result. The classification result can indicate the specific posture category of the gesture or whether the gesture is a smoking gesture. Please refer to Figure 8 , Figure 8 This embodiment of the present application provides a contour image. This embodiment does not limit the specific method of edge extraction processing. For example, Fourier operator extraction can be performed on the contour points, and edge extraction can be performed based on the Fourier operator.

[0151] Please refer to Figure 4 , Figure 4 This is a flow chart of another smoking behavior detection method provided in an embodiment of the present application. The smoking behavior detection process recorded therein includes gesture semantic detection, and uses steps S201 to S204 to replace the above-mentioned step S106. Specifically:

[0152] S201: Perform gesture semantic recognition processing on the image to be tested to obtain a gesture recognition result.

[0153] S202: Determine whether the gesture recognition result is a hit.

[0154] S203: Preset operation.

[0155] S204: Determine whether smoking behavior exists in the image to be tested.

[0156] The preset operation of step S203 may be the same as or different from the preset operation of step S106, which is not limited in this embodiment. It should be noted that the calculation process of the matching parameters and the process of gesture semantic recognition processing can be executed in parallel, or can be performed as follows: Figure 4 As shown, serial execution.

[0157] Based on the above examples, please refer to Figure 9 , Figure 9A flow chart of a specific smoking behavior detection method provided in an embodiment of the present application. After extracting the original image (i.e., the image to be tested) from the video to be tested, target detection is performed using YOLOv5, and the cigarette position in the original image is selected using it. The target position is obtained so that the suspected cigarette range can be determined based on the target position. At the same time, the MTCNN model (Multi-Task Convolutional Neural Network, a convolutional neural network that processes multiple tasks at the same time) is used to locate the key points of the face and determine the key points of the mouth, thereby determining the range of the mouth. The distance or intersection area (i.e., matching parameters) between the suspected cigarette range and the mouth range is calculated, and it is determined whether the distance is appropriate. Appropriate means that the matching parameters are within the preset range. If it is not appropriate, it can be determined that there is no smoking behavior, and the next video frame is re-acquired as the original image. If there is smoking behavior, gesture judgment is further performed.

[0158] Specifically, a gesture semantic segmentation model is used to detect gesture information in the original image and determine whether the gesture in the original image indicates smoking. If so, and the matching parameters are appropriate, it can be determined that smoking behavior is present in the original image and identified as an abnormal video frame. If the number of abnormal video frames in the entire test video reaches 30% or more of the total number of video frames, the test video can be determined to be an abnormal video, that is, a smoking video.

[0159] The computer-readable storage medium provided in the embodiments of the present application is introduced below. The computer-readable storage medium described below and the smoking behavior detection method described above can be referenced to each other.

[0160] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned smoking behavior detection method are implemented.

[0161] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0163] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0165] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms include, comprise, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0166] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A smoking behavior detection method, characterized in that: include: Acquire a video to be tested, and determine a number of candidate video frames in the video to be tested; The number of candidate video frames is less than the total number of video frames in the video to be tested; Selecting one of the candidate video frames as a test image in a preset order, and inputting the test image into a target detection model with cigarettes as the target to determine the range of suspected cigarettes; When the target detection model cannot confirm whether the suspected target it detects is a cigarette, the target detection model outputs the suspected cigarette range; Identifying a plurality of facial key points in the image to be tested, and determining the range of the mouth using the facial key points; Calculating a matching parameter between the suspected cigarette range and the mouth range, wherein the matching parameter is a target distance between the suspected cigarette range and the mouth range, where the target distance refers to the minimum distance between any point within the suspected cigarette range and any point within the mouth range; or, the target distance refers to the minimum distance between a specified point within the suspected cigarette range and any point within the mouth range; or, the target distance refers to the minimum distance between a specified point within the mouth range and any point within the suspected cigarette range; and / or, the intersection area between the mouth range and the suspected cigarette range; If the matching parameter is within a preset range, it is determined that smoking behavior exists in the image to be tested; If it is detected that all the candidate video frames have been detected and the number of abnormal video frames with the smoking behavior is greater than the abnormal threshold, the video to be tested is determined to be an abnormal video; After the image to be tested is input into the target detection model with cigarettes as the target, the method further includes: If the target detection model outputs a determined cigarette range, it is determined that there is smoking behavior in the image to be tested; the target detection model directly outputs the position information of the object suspected to be a cigarette and the corresponding confidence level, and when the confidence level is within a credible interval, the detection result is determined to be a hit, and the position information is the position information corresponding to the object with the confidence level within the credible interval; when the confidence level is within a determined interval, the detection result is determined to be a determined cigarette range, and the position information is the position information corresponding to the object with the confidence level within the determined interval; when the confidence level is not within the credible interval, no position information is output, and a miss detection result is output; Also includes: Performing gesture semantic recognition processing on the image to be tested to obtain a gesture recognition result; Correspondingly, if the matching parameter is within a preset range, determining that smoking behavior exists in the image to be tested includes: If the matching parameter is within the preset range and the gesture recognition result is a hit, it is determined that the smoking behavior exists in the image to be tested; The performing gesture semantic recognition processing on the image to be tested to obtain a gesture recognition result includes: Deleting the non-skin color area in the image to be tested to obtain the skin color area; Performing edge extraction processing on the skin color area to obtain a contour image; Inputting the contour image into a gesture classification model to obtain the gesture recognition result; The step of deleting the non-skin color area in the image to be tested to obtain the skin color area includes: Determine whether the channel pixel value of each pixel in each channel of the image to be tested is within the corresponding channel interval; If the pixel values of each channel are all within the channel interval, it is determined that the pixel is within the initial skin color area; Performing corrosion processing and dilation processing on the initial skin color area, and deleting the image in the non-skin color area outside the skin color area, to obtain the skin color area; The training process of the target detection model includes: Acquire a plurality of initial training images, and perform data augmentation processing on the initial training images to obtain training images; Inputting the training image into the initial model to obtain a training recognition range; Calculating a range distance between the training recognition range and the label range of the training image, and adjusting model parameters of the initial model using the range distance; The performing data enhancement processing on the initial training image to obtain a training image includes: Performing random cropping and / or random scaling on any plurality of the initial training images to obtain a plurality of target initial training images; The multiple target initial training images are randomly arranged and spliced to obtain one training image.

2. The smoking behavior detection method according to claim 1, characterized in that: Inputting the image to be detected into a target detection model with cigarettes as targets to determine the range of suspected cigarettes includes: Inputting the image to be tested into an object detection model with cigarettes as targets to obtain coordinates of suspected cigarettes; Determine range length and width parameters, and based on the range length and width parameters, determine the suspected cigarette range using the suspected cigarette coordinates.

3. The smoking behavior detection method according to claim 1, characterized in that: The identifying of a plurality of facial key points in the image to be tested and determining the range of the mouth using the facial key points includes: Performing facial key point detection on the image to be tested to obtain facial key points; Connecting lines between the key points of the face according to a preset rule to obtain a plurality of candidate ranges; The mouth key points are determined according to the key point information of the facial key points, and the mouth range is generated using the candidate range corresponding to the mouth key points.

4. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the smoking behavior detection method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the smoking behavior detection method according to any one of claims 1 to 3 is implemented.

Images