A smoking behavior detection method, device, equipment and storage medium

Abstract

The application discloses a smoking behavior detection method and device, equipment and storage medium, and relates to the technical field of image recognition. The method comprises the following steps: acquiring a pedestrian area in a to-be-detected image; performing smoking detection on the pedestrian area by using a fusion model obtained by fusing multiple models; the fusion model comprises a pedestrian posture recognition submodel, a cigarette recognition submodel and a pedestrian gesture and mouth shape recognition submodel; and whether there is a smoking behavior in the to-be-detected image is judged according to the recognition result of the submodel in the fusion model. By using the detection mode of multiple model fusion, comprehensive judgment is performed on the image features of the cigarette, the posture and action of the person and the gesture and mouth shape of smoking. Compared with a single detection model, the accuracy of smoking detection is greatly improved, and the object of smoking detection is the pedestrian area in the to-be-detected image, that is, the cigarette detection is performed only in the area where the pedestrian is located, so that background interference is avoided, and the situation of missing report or false report is avoided.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, device, and storage medium for detecting smoking behavior. Background Technology

[0002] Currently, safety requirements for locations such as gas stations and CNG stations are high, prohibiting smoking. This is typically achieved through staff on duty, but labor costs are high. Existing technologies involve installing security cameras in these scenarios and using AI algorithms to monitor the stations in real time. This allows for timely alerts upon detecting dangerous behaviors such as smoking, assisting station management. However, commonly used single-model smoking detection based on computer vision suffers from a high false alarm rate. Furthermore, cigarettes typically occupy a small portion of the image, making their image features indistinct, resulting in a high false negative rate. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for detecting smoking behavior, which can improve the accuracy of smoking detection. The specific solution is as follows:

[0004] Firstly, this application discloses a method for detecting smoking behavior, including:

[0005] Obtain the pedestrian region in the image to be detected;

[0006] Smoking detection in the pedestrian area is performed using a fusion model obtained by multi-model fusion; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model.

[0007] Based on the recognition results corresponding to the sub-models within the fusion model, it is determined whether smoking behavior exists in the image to be detected.

[0008] Optionally, the smoking behavior detection method further includes:

[0009] Charging gun detection is performed in the pedestrian area using a charging gun recognition model; wherein, the image to be detected is a monitoring image of a target charging station near a gas station.

[0010] Accordingly, determining whether smoking behavior exists in the image to be detected based on the recognition results of each sub-model within the fusion model includes:

[0011] Based on the recognition results of each sub-model within the fusion model and the charging gun detection results of the charging gun recognition model, it is determined whether smoking behavior exists in the image to be detected.

[0012] Optionally, determining whether smoking behavior exists in the image to be detected based on the recognition results of each sub-model within the fusion model and the charging gun detection results of the charging gun recognition model includes:

[0013] If the charging gun detection result corresponding to the charging gun recognition model indicates the presence of a charging gun, then the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model is corrected according to the charging gun detection result, and the presence of smoking behavior in the image to be detected is determined according to the corrected result.

[0014] Optionally, the step of correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result, includes:

[0015] Determine whether the charging gun intersects with the hand of a pedestrian in the pedestrian area;

[0016] If they intersect, the hand smoking action recognition results identified by the pedestrian gesture and mouth shape recognition sub-model are deleted. Based on the recognition results of the pedestrian posture recognition sub-model, the cigarette recognition sub-model, and the mouth smoking action recognition results corresponding to the pedestrian gesture and mouth shape recognition sub-model, it is determined whether there is smoking behavior in the image to be detected.

[0017] Optionally, the step of correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result, includes:

[0018] If the recognition result of the pedestrian gesture and mouth shape recognition sub-model is that the hand is smoking, then the length and width ratio of the object in the hand is determined according to the hand detection box recognized by the pedestrian gesture and mouth shape recognition sub-model.

[0019] Based on the preset target aspect ratio and the aspect ratio, it is determined whether the object in the hand is a charging gun. Based on the determination result, the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model is corrected. Then, based on the corrected result, it is determined whether there is smoking behavior in the image to be detected.

[0020] Optionally, determining whether smoking behavior exists in the image to be detected based on the recognition results corresponding to the sub-models within the fusion model includes:

[0021] If the cigarette recognition sub-model recognizes a cigarette, and the pedestrian gesture and mouth shape recognition sub-model recognizes a hand smoking action or a head smoking action, then it is determined whether the cigarette intersects with the hand or whether the cigarette intersects with the head.

[0022] If they intersect, it is determined that smoking behavior exists in the image to be detected.

[0023] Optionally, the step of using a fusion model obtained after multi-model fusion to detect smoking in the pedestrian area includes:

[0024] The pedestrian posture recognition sub-model is used to perform overall skeletal point motion recognition on pedestrians in the pedestrian area in order to identify whether the pedestrians are smoking.

[0025] The cigarette recognition sub-model is used to identify cigarettes in the pedestrian area in order to determine whether cigarettes are present in the pedestrian area.

[0026] The pedestrian gesture and mouth shape recognition sub-model is used to recognize the hand smoking action and head smoking action of pedestrians in the pedestrian area in order to identify whether the pedestrian is smoking.

[0027] Secondly, this application discloses a smoking behavior detection device, comprising:

[0028] The pedestrian region acquisition module is used to acquire pedestrian regions in the image to be detected;

[0029] The smoking detection module is used to detect smoking in the pedestrian area using a fusion model obtained by fusing multiple models; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model.

[0030] The smoking behavior determination module is used to determine whether smoking behavior exists in the image to be detected based on the recognition results corresponding to the sub-models within the fusion model.

[0031] Thirdly, this application discloses an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] A processor is used to execute the computer program to implement the aforementioned smoking behavior detection method.

[0034] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned smoking behavior detection method.

[0035] In this application, a pedestrian region is obtained from an image to be detected; a fusion model obtained by multi-model fusion is used to detect smoking in the pedestrian region; the fusion model includes a pedestrian pose recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model; based on the recognition results corresponding to the sub-models within the fusion model, it is determined whether smoking behavior exists in the image to be detected. It is evident that by using a multi-model fusion detection method, which comprehensively judges from the image features of cigarettes, human poses and movements, and the gestures and mouth shapes of smokers, compared to a single detection model, the detection accuracy of smoking is greatly improved.

[0036] The accuracy of the test is improved, and the target of the smoking detection is the pedestrian area in the image to be detected. That is, cigarette detection is only performed in the area where pedestrians are located to avoid background interference and avoid missed or false alarms. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This application provides a flowchart of a smoking behavior detection method;

[0039] Figure 2 This application provides a schematic diagram of a specific dimension for detecting smoking behavior.

[0040] Figure 3 An image representing a specific smoking behavior;

[0041] Figure 4 This is an image representing another specific smoking behavior;

[0042] Figure 5 This application provides a specific schematic diagram of a smoking warning system.

[0043] Figure 6 A specific AIbox workflow diagram is provided for this application;

[0044] Figure 7 A flowchart of a specific smoking behavior detection method provided in this application;

[0045] Figure 8 Flowchart of another specific smoking behavior detection method provided in this application;

[0046] Figure 9 This is an image of a specific pedestrian holding a charging gun;

[0047] Figure 10 This is a schematic diagram of a smoking behavior detection device provided in this application;

[0048] Figure 11 This application provides a structural diagram of an electronic device. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In existing technologies, commonly used single-model smoking detection based on computer vision suffers from a high false positive rate. Furthermore, cigarettes typically occupy a small portion of the overall image, making their image features inconspicuous, resulting in a high false negative rate. To overcome these technical problems, this application proposes a smoking behavior detection method.

[0051] It can improve the accuracy of smoking detection.

[0052] This application discloses a method for detecting smoking behavior. See [link to relevant documentation]. Figure 1 As shown, the method may include the following steps:

[0053] Step S11: Obtain the pedestrian region in the image to be detected.

[0054] In this embodiment, the pedestrian region in the image to be detected is first obtained. Specifically, the real-time scene image captured by the camera is used as the image to be detected. Then, pedestrians are identified from the image, and the area where the pedestrians are located is defined as the pedestrian region. A preset pedestrian recognition model can be used for this purpose. This preset pedestrian recognition model can be a model obtained by removing a preset number of network layers from the YOLOv5s model. It is understood that pedestrians occupy a large proportion of the image, so reducing network layers can improve inference speed without reducing accuracy. It is also understood that since smoking pedestrians are dependent on human presence, smoking detection only needs to be performed on the area where the pedestrian is located.

[0055] Step S12: Use the fusion model obtained by multi-model fusion to detect smoking in the pedestrian area; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model.

[0056] In this embodiment, after obtaining the pedestrian area, a fusion model obtained through multi-model fusion is used to detect smoking in the pedestrian area, i.e., smoking detection. The aforementioned fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model. It can be seen that by using a multi-model fusion detection method, the presence of smoking behavior is determined comprehensively from three different dimensions: the image features of cigarettes, the person's smoking posture and actions, and the smoking gestures and mouth shapes. Compared to a single detection model, this significantly improves the accuracy of smoking detection. It is understandable that any single detection method may result in false positives. For example, directly judging actions may lead to misidentification of actions similar to touching the head or face, similar to smoking; directly detecting cigarettes may lead to misidentification of mask straps, collars, glasses, etc. Therefore, a model fusion approach is adopted, using cigarette detection, skeletal point action recognition, and head and hand action recognition to comprehensively determine whether smoking behavior exists.

[0057] This embodiment can be applied to places such as gas stations, CNG stations, and charging stations. Because the background of these places is too complex, it is easy to misidentify tree branches, white strips on clothes, car keys, or mask straps as cigarettes, resulting in a high false alarm rate. Therefore, using existing solutions to inspect such places will reduce the accuracy of detection. In this embodiment, a fusion model obtained by multi-model fusion is used for multi-dimensional detection to improve the accuracy of detection.

[0058] In this embodiment, the step of using a fusion model derived from multi-model fusion to detect smoking in the pedestrian area may include: using the pedestrian pose recognition sub-model to perform overall skeletal point motion recognition on pedestrians in the pedestrian area to identify whether the pedestrians are smoking; using the cigarette recognition sub-model to identify cigarettes in the pedestrian area to identify whether cigarettes are present in the pedestrian area; and using the pedestrian gesture and mouth shape recognition sub-model to recognize hand and head smoking actions on pedestrians in the pedestrian area to identify whether the pedestrians are smoking. Figure 2 As shown, the pedestrian posture recognition sub-model is used to recognize the smoking action of the whole person's posture and movement. The cigarette recognition sub-model is used to detect cigarette image features within the detected person range. The pedestrian gesture and mouth shape recognition sub-model is used to detect the person's head and hands, and to judge the mouth shape and gesture of the head and hands.

[0059] Step S13: Based on the recognition results corresponding to the sub-models within the fusion model, determine whether there is smoking behavior in the image to be detected.

[0060] In this embodiment, the recognition results corresponding to the sub-models within the fusion model are obtained, namely the recognition results corresponding to the pedestrian posture recognition sub-model, the cigarette recognition sub-model, and the pedestrian gesture and mouth shape recognition sub-model, respectively. Then, based on these recognition results, it is determined whether smoking behavior exists in the image to be detected. Specifically, a voting method can be used; when two or more dimensions represent smoking, it is considered smoking. For example, if the pedestrian posture recognition sub-model determines that smoking behavior exists, and the cigarette recognition sub-model identifies a cigarette, then it can be determined whether smoking behavior exists in the detection image. After determining that smoking behavior exists, a smoking alarm prompt, along with the smoking time and duration, is sent to the management terminal.

[0061] In this embodiment, determining whether smoking behavior exists in the image to be detected based on the recognition results corresponding to the sub-models within the fusion model may include: if the cigarette recognition sub-model recognizes a cigarette, and the pedestrian gesture and mouth shape recognition sub-model recognizes a hand smoking action or a head smoking action, then determining whether the cigarette intersects with the hand, or whether the cigarette intersects with the head; if they intersect, then determining that smoking behavior exists in the image to be detected. Figure 3 and Figure 4 As shown, because cigarettes occupy too small a portion of the image, they are often misidentified. To reduce misidentification, considering that cigarettes must intersect with the hand or mouth during smoking, we can further judge the detected cigarettes. Only when the cigarette intersects with the head or hand can it be considered a real cigarette.

[0062] The aforementioned smoking behavior detection algorithm can be deployed at the edge of a cloud-edge architecture, such as within an AIbox. AIbox's platform-wide big data computing capabilities can significantly reduce hardware investment in security cameras with high computational demands, such as those for detecting smoking, thus reducing equipment costs. At the edge, an NPU (Neural Processing Unit) can be used.

[0063] The Processing Unit (embedded neural network processor) inference further improves recognition speed. Specifically, the site security wide-angle camera takes a picture, and Aibox uses a smoking algorithm to detect whether smoking behavior exists. If so, an alert is pushed to the administrator and the next round of detection is initiated; if not, the next round of detection is directly initiated. For example... Figure 5 The smoking warning shown is an example. Figure 6 The AIbox workflow shown involves acquiring images from the terminal, performing recognition using AIbox, and then reporting the recognition results to the cloud so that administrators can retrieve the results from the cloud.

[0064] Figure 7This is a flowchart of a specific smoking behavior detection method. After Aibox acquires an image, it determines whether a pedestrian is present. If so, the detected person is expanded by pixels in all directions and cropped to obtain the pedestrian region. Then, it determines whether a cigarette is present in the pedestrian region. If so, a classification model is used to further determine whether the cigarette is genuine, avoiding interference. Simultaneously, the head and hands in the pedestrian region are identified, and a classification model is used to further determine whether smoking actions are present. If a cigarette is present, a smoking action is performed, and the hand or head intersects with the cigarette, smoking behavior can be determined. The above classification model can be the EfficientNetV2 model, suitable for high-speed inference.

[0065] As can be seen from the above, this embodiment acquires the pedestrian region in the image to be detected; a fusion model obtained by multi-model fusion is used to detect smoking in the pedestrian region; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model; based on the recognition results corresponding to the sub-models within the fusion model, it is determined whether smoking behavior exists in the image to be detected. It is evident that by using a multi-model fusion detection method, a comprehensive judgment is made based on the image features of cigarettes, human posture and actions, and the gestures and mouth shapes of smokers. Compared to a single detection model, this significantly improves the accuracy of smoking detection. Furthermore, the object of smoking detection is the pedestrian region in the image to be detected, meaning cigarette detection is only performed within the area where pedestrians are located, avoiding background interference and preventing missed or false alarms.

[0066] This application discloses a specific method for detecting smoking behavior. (See also...) Figure 8 As shown, the method may include the following steps:

[0067] Step S21: Obtain the pedestrian area in the image to be detected; the image to be detected is a monitoring image of a target charging station near a gas station.

[0068] In this embodiment, the image to be detected is specifically a monitoring image of a target charging station near a gas station, that is, a smoking detection is performed on a charging station near a gas station.

[0069] Step S22: Use the fusion model obtained by multi-model fusion to perform smoking detection in the pedestrian area.

[0070] The fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model.

[0071] Step S23: Use the charging gun recognition model to detect charging guns in the pedestrian area.

[0072] In this embodiment, while performing smoking detection, a charging gun recognition model is used to detect charging guns in the pedestrian area.

[0073] Step S24: Based on the recognition results of each sub-model in the fusion model and the charging gun detection results of the charging gun recognition model, determine whether there is smoking behavior in the image to be detected.

[0074] In this embodiment, the final determination of whether smoking behavior exists in the image to be detected is based on the recognition results of each sub-model within the fusion model and the charging gun detection results of the charging gun recognition model. It is understandable that, unlike traditional industrial parks or parking lots, charging stations have a large number of charging piles and charging guns, and the action of a person holding a charging gun can easily be misidentified as smoking. Figure 9 As shown, combining the results of charging gun detection with the identification of smoking avoids misjudging pedestrians as smokers simply because they are holding a charging gun. Furthermore, using security cameras to assist management in identifying smoking at charging stations near gas stations and refueling stations can significantly save labor costs while ensuring the safety of these stations.

[0075] In this embodiment, determining whether smoking behavior exists in the image to be detected based on the recognition results of each sub-model within the fusion model and the charging gun detection results of the charging gun recognition model can include: if the charging gun detection result corresponding to the charging gun recognition model indicates the presence of a charging gun, then correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result. Since a charging gun can be misjudged when held in the hand, the specific method involves using the charging gun detection result to correct the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model.

[0076] In this embodiment, the step of correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result, may include: determining whether the charging gun intersects with the hand of a pedestrian in the pedestrian area; if they intersect, deleting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model, and determining whether smoking behavior exists in the image to be detected based on the recognition results of the pedestrian posture recognition sub-model, the recognition results of the cigarette recognition sub-model, and the mouth smoking action recognition result corresponding to the pedestrian gesture and mouth shape recognition sub-model. That is, in a specific real-time mode, by adding a charging gun detection model, when a charging gun is detected in the hand, the smoking judgment is no longer based on the smoking gesture, but only on the recognition results of the pedestrian posture recognition sub-model and the cigarette recognition sub-model.

[0077] The model's recognition results, along with the recognition results of smoking actions in the mouth corresponding to the pedestrian gesture and mouth shape recognition sub-model, effectively reduce false alarms caused by charging guns at charging stations.

[0078] In this embodiment, the step of correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result, may include: if the recognition result of the pedestrian gesture and mouth shape recognition sub-model indicates that there is a hand smoking action, then determining the aspect ratio of the object in the hand based on the hand detection box identified by the pedestrian gesture and mouth shape recognition sub-model; determining whether the object in the hand is a charging gun based on a preset target aspect ratio and the aspect ratio, correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the judgment result, and then determining whether smoking behavior exists in the image to be detected based on the corrected result. The above method, which detects a charging gun in the hand, no longer relies on smoking gestures for smoking judgment, and cannot address situations where a pedestrian is holding a charging gun in one hand and a cigarette in the other. To further improve the accuracy of the judgment and avoid misjudgment, in another specific real-time method, the difference between the aspect ratio of the charging gun and the cigarette is used to determine the type of object in the pedestrian's hand, filtering out features that do not belong to cigarettes and eliminating interference from the charging gun. Of course, in addition to these, you can also distinguish them based on the different sizes of the hand and the charging gun, or the different sizes of the hand and the cigarette, or any other characteristics that can distinguish the charging gun and the cigarette.

[0079] The specific process of step S22 can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.

[0080] As can be seen from the above, this embodiment acquires the pedestrian area in the image to be detected; the image to be detected is a monitoring image of a target charging station near a gas station; a fusion model obtained by multi-model fusion is used to detect smoking in the pedestrian area; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model; a charging gun recognition model is used to detect charging guns in the pedestrian area; based on the recognition results of each sub-model in the fusion model and the charging gun detection results of the charging gun recognition model, it is determined whether smoking behavior exists in the image to be detected. Therefore, for the monitoring image of the target charging station, the smoking judgment is made by combining the recognition results of the original fusion model with the recognition results of the charging gun recognition model, avoiding misjudgment of smoking due to pedestrians holding charging guns.

[0081] Accordingly, this application also discloses a smoking behavior detection device, see [link to relevant documentation]. Figure 10 As shown, the device includes:

[0082] Pedestrian region acquisition module 11 is used to acquire pedestrian regions in the image to be detected;

[0083] The smoking detection module 12 is used to detect smoking in the pedestrian area using a fusion model obtained by multi-model fusion; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model.

[0084] The smoking behavior judgment module 13 is used to determine whether smoking behavior exists in the image to be detected based on the recognition results corresponding to the sub-models within the fusion model.

[0085] As can be seen from the above, this embodiment acquires the pedestrian region in the image to be detected; a fusion model obtained by multi-model fusion is used to detect smoking in the pedestrian region; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model; based on the recognition results corresponding to the sub-models within the fusion model, it is determined whether smoking behavior exists in the image to be detected. It is evident that by using a multi-model fusion detection method, a comprehensive judgment is made based on the image features of cigarettes, human posture and actions, and the gestures and mouth shapes of smokers. Compared to a single detection model, this significantly improves the accuracy of smoking detection. Furthermore, the object of smoking detection is the pedestrian region in the image to be detected, meaning cigarette detection is only performed within the area where pedestrians are located, avoiding background interference and preventing missed or false alarms.

[0086] In some specific embodiments, the smoking behavior detection device may specifically include:

[0087] A charging gun detection unit is used to detect charging guns in the pedestrian area using a charging gun recognition model; wherein the image to be detected is a monitoring image of a target charging station near a gas station.

[0088] Accordingly, the smoking behavior judgment module 13 may specifically include:

[0089] The smoking behavior determination unit is used to determine whether smoking behavior exists in the image to be detected based on the recognition results of each sub-model in the fusion model and the charging gun detection results of the charging gun recognition model.

[0090] In some specific embodiments, the determination of smoking behavior may specifically include:

[0091] The correction unit is used to correct the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model according to the charging gun detection result if the charging gun detection result corresponding to the charging gun recognition model is that a charging gun exists, and to determine whether there is smoking behavior in the image to be detected according to the corrected result.

[0092] In some specific embodiments, the correction unit may specifically include:

[0093] An intersection determination unit is used to determine whether the charging gun intersects with the hand of a pedestrian in the pedestrian area;

[0094] The judgment unit is used to delete the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model if they intersect, and to determine whether there is smoking behavior in the image to be detected based on the recognition results of the pedestrian posture recognition sub-model, the recognition results of the cigarette recognition sub-model, and the mouth smoking action recognition results corresponding to the pedestrian gesture and mouth shape recognition sub-model.

[0095] In some specific embodiments, the correction unit may specifically include:

[0096] The aspect ratio determination unit is used to determine the aspect ratio of the object in the hand based on the hand detection box identified by the pedestrian gesture and mouth shape recognition sub-model if the recognition result of the pedestrian gesture and mouth shape recognition sub-model is that the hand is smoking.

[0097] The judgment unit is used to determine whether the object in the hand is a charging gun based on the preset target aspect ratio and the aspect ratio, and to correct the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the judgment result, and then to determine whether there is smoking behavior in the image to be detected based on the corrected result.

[0098] In some specific embodiments, the smoking behavior determination module 13 may specifically include:

[0099] The intersection judgment unit is used to determine whether the cigarette intersects with the hand or the head if the cigarette recognition sub-model recognizes a cigarette and the pedestrian gesture and mouth shape recognition sub-model recognizes a hand smoking action or a head smoking action.

[0100] A smoking behavior determination unit is used to determine that smoking behavior exists in the image to be detected if the images intersect.

[0101] In some specific embodiments, the smoking detection module 12 may specifically include:

[0102] The motion recognition unit is used to perform overall skeletal point motion recognition on pedestrians in the pedestrian area using the pedestrian posture recognition sub-model, so as to identify whether the pedestrians are smoking.

[0103] A cigarette recognition unit is used to perform cigarette recognition on the pedestrian area using the cigarette recognition sub-model, so as to identify whether the pedestrian area contains cigarettes;

[0104] The local action recognition unit is used to identify hand smoking actions and head smoking actions of pedestrians in the pedestrian area using the pedestrian gesture and mouth shape recognition sub-model, so as to identify whether the pedestrian is smoking.

[0105] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 11 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0106] Figure 11 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the smoking behavior detection method disclosed in any of the foregoing embodiments.

[0107] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0108] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including the image to be detected, etc. The storage method can be temporary storage or permanent storage.

[0109] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the smoking behavior detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0110] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the steps of the smoking behavior detection method disclosed in any of the foregoing embodiments.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of both. The software module can be placed in random access memory.

[0113] Storage media of any kind, including RAM, memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0114] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] The above provides a detailed description of the smoking behavior detection method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting smoking behavior, characterized in that, include: Obtain the pedestrian region in the image to be detected; Smoking detection is performed in the pedestrian area using a fusion model obtained from multi-model fusion. The fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model. Based on the recognition results corresponding to the sub-models within the fusion model, it is determined whether smoking behavior exists in the image to be detected; The smoking behavior detection method further includes: Charging gun detection is performed in the pedestrian area using a charging gun recognition model; wherein, the image to be detected is a monitoring image of a target charging station near a gas station. Accordingly, determining whether smoking behavior exists in the image to be detected based on the recognition results of each sub-model within the fusion model includes: Based on the recognition results of each sub-model within the fusion model and the charging gun detection results of the charging gun recognition model, it is determined whether smoking behavior exists in the image to be detected; The step of determining whether smoking behavior exists in the image to be detected based on the recognition results of each sub-model within the fusion model and the charging gun detection results of the charging gun recognition model includes: If the charging gun detection result corresponding to the charging gun recognition model indicates the presence of a charging gun, then the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model is corrected according to the charging gun detection result, and the presence of smoking behavior in the image to be detected is determined according to the corrected result.

2. The smoking behavior detection method according to claim 1, characterized in that, The step of correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result, includes: Determine whether the charging gun intersects with the hand of a pedestrian in the pedestrian area; If they intersect, the hand smoking action recognition results identified by the pedestrian gesture and mouth shape recognition sub-model are deleted. Based on the recognition results of the pedestrian posture recognition sub-model, the cigarette recognition sub-model, and the mouth smoking action recognition results corresponding to the pedestrian gesture and mouth shape recognition sub-model, it is determined whether there is smoking behavior in the image to be detected.

3. The smoking behavior detection method according to claim 1, characterized in that, The step of correcting the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model based on the charging gun detection result, and determining whether smoking behavior exists in the image to be detected based on the corrected result, includes: If the recognition result of the pedestrian gesture and mouth shape recognition sub-model is that the hand is smoking, then the length and width ratio of the object in the hand is determined according to the hand detection box recognized by the pedestrian gesture and mouth shape recognition sub-model. Based on the preset target aspect ratio and the aspect ratio, it is determined whether the object in the hand is a charging gun. Based on the determination result, the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model is corrected. Then, based on the corrected result, it is determined whether there is smoking behavior in the image to be detected.

4. The smoking behavior detection method according to claim 1, characterized in that, The step of determining whether smoking behavior exists in the image to be detected based on the recognition results corresponding to the sub-models within the fusion model includes: If the cigarette recognition sub-model recognizes a cigarette, and the pedestrian gesture and mouth shape recognition sub-model recognizes a hand smoking action or a head smoking action, then it is determined whether the cigarette intersects with the hand or whether the cigarette intersects with the head. If they intersect, it is determined that smoking behavior exists in the image to be detected.

5. The smoking behavior detection method according to any one of claims 1 to 4, characterized in that, The step of using a fusion model derived from multi-model fusion to detect smoking in the pedestrian area includes: The pedestrian posture recognition sub-model is used to perform overall skeletal point motion recognition on pedestrians in the pedestrian area in order to identify whether the pedestrians are smoking. The cigarette recognition sub-model is used to identify cigarettes in the pedestrian area in order to determine whether cigarettes are present in the pedestrian area. The pedestrian gesture and mouth shape recognition sub-model is used to recognize the hand smoking action and head smoking action of pedestrians in the pedestrian area in order to identify whether the pedestrian is smoking.

6. A smoking behavior detection device, characterized in that, include: The pedestrian region acquisition module is used to acquire pedestrian regions in the image to be detected; The smoking detection module is used to detect smoking in the pedestrian area using a fusion model obtained by fusing multiple models; the fusion model includes a pedestrian posture recognition sub-model, a cigarette recognition sub-model, and a pedestrian gesture and mouth shape recognition sub-model. The smoking behavior judgment module is used to determine whether smoking behavior exists in the image to be detected based on the recognition results corresponding to the sub-models within the fusion model. The smoking behavior detection device further includes: A charging gun detection unit is used to detect charging guns in the pedestrian area using a charging gun recognition model; wherein the image to be detected is a monitoring image of a target charging station near a gas station. Accordingly, the smoking behavior judgment module 13 may specifically include: The smoking behavior judgment unit is used to determine whether there is smoking behavior in the image to be detected based on the recognition results of each sub-model in the fusion model and the charging gun detection results of the charging gun recognition model. The smoking behavior judgment unit is used to correct the hand smoking action recognition result identified by the pedestrian gesture and mouth shape recognition sub-model according to the charging gun detection result if the charging gun detection result corresponding to the charging gun recognition model is that a charging gun exists, and to determine whether there is smoking behavior in the image to be detected according to the corrected result.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the smoking behavior detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein the computer program, when executed by a processor, implements the smoking behavior detection method as described in any one of claims 1 to 5.

Images