Smoking behavior determination method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for determining smoking behavior, which relates to computer technology. The method comprises: inputting the current frame image in the video stream into a cigarette segmentation model to obtain a cigarette segmentation map, wherein the cigarette segmentation map is used to indicate the presence of a cigarette analog in the current frame image; determining the cigarette butt area from the cigarette segmentation map, wherein the cigarette butt area includes a coke area and an ash area; when the coke area and the ash area meet preset conditions, determining that the cigarette analog is a cigarette in a burning state, so as to determine that the user has a smoking behavior based on the cigarette in a burning state. The present application determines that a cigarette analog is a cigarette only when it is detected that the coke area and the ash area meet preset conditions, thereby improving the detection accuracy of cigarettes in a burning state. The present application solves the problem of misdetecting slender objects as cigarettes in the existing solution, and has the beneficial effect of improving the accuracy of determining the user's smoking behavior.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for determining smoking behavior. Background Art

[0002] For the common smoking behavior in daily life, most public places and work environments have no-smoking signs. However, it is difficult to reasonably control smoking behavior only through human management, which depends entirely on people's self-awareness.

[0003] With the rapid development of computer vision technology and hardware technology, analyzing video images to determine the user's smoking behavior is currently the mainstream research direction. When determining the user's smoking behavior, the existing detection model usually focuses on whether there are slender objects in the user's mouth or hands. However, slender objects include lollipops, straws, pens and other objects that are similar in shape to cigarettes. This detection method is difficult to distinguish slender objects from cigarettes, resulting in a high false detection rate. Summary of the invention

[0004] The present application provides a smoking behavior determination method, device, electronic device and storage medium, which can accurately detect burning cigarettes to improve the accuracy of determining the user's smoking behavior.

[0005] In a first aspect, the present application provides a method for determining smoking behavior, comprising: inputting a current frame image in a video stream into a cigarette segmentation model to obtain a cigarette segmentation map, wherein the cigarette segmentation map is used to indicate the presence of a cigarette analog in the current frame image; determining a cigarette butt area from the cigarette segmentation map, wherein the cigarette butt area includes a coke area and an ash area; when the coke area and the ash area meet preset conditions, determining that the cigarette analog is a cigarette in a burning state, so as to determine that the user has smoking behavior based on the cigarette in a burning state.

[0006] Optionally, determining the cigarette butt area from the cigarette segmentation map includes: performing grayscale processing on the cigarette segmentation map to obtain a cigarette grayscale map; determining a cigarette butt ring area from the cigarette grayscale map, the cigarette butt ring area dividing the cigarette grayscale map into a first grayscale area and a second grayscale area; comparing the grayscale value corresponding to the first grayscale area with the grayscale value corresponding to the second grayscale area, determining that the grayscale area with the smaller grayscale value is the cigarette butt area, and the cigarette butt area is the first grayscale area or the second grayscale area.

[0007] Optionally, determining the cigarette butt ring area from the cigarette grayscale image includes: performing a ring fitting process on the cigarette grayscale image to determine whether the cigarette grayscale image contains a ring area; when the cigarette grayscale image contains the ring area and the grayscale value of the ring area is lower than a first preset value, determining that the ring area is the cigarette butt ring area.

[0008] Optionally, the method further includes: when the annular area is not included in the cigarette grayscale image, or the grayscale value of the annular area is not lower than the first preset value, determining that the cigarette butt ring area is not included in the cigarette grayscale image, and determining that the cigarette analog is not a cigarette in a burning state.

[0009] Optionally, the coke area and the ash area are determined in the following manner: grayscale processing is performed on the cigarette butt area to obtain a cigarette butt grayscale map; the area in the cigarette butt grayscale map where the average grayscale value is lower than a second preset value is determined as the coke area; the area in the cigarette butt grayscale map where the average grayscale value is higher than a third preset value is determined as the ash area.

[0010] Optionally, after the current frame image in the video stream is input into the cigarette segmentation model, it also includes: the cigarette segmentation model determines the image area corresponding to the target object from the current frame image; when the image area corresponding to the target object and the image area corresponding to the current frame image satisfy a preset relationship, determines that the target object is the cigarette analog, wherein the preset relationship is related to the length of the cigarette holder; and outputs a cigarette segmentation map corresponding to the cigarette analog.

[0011] Optionally, after determining that the user has smoking behavior based on the cigarette in a burning state, it also includes: determining at least one target hardware information to be linked; calling at least one target hardware device corresponding to the target hardware information to provide a corresponding linkage service based on at least one target hardware device.

[0012] In a second aspect, the present application provides a smoking behavior determination device, the device comprising: an image input module, used to input a current frame image in a video stream into a cigarette segmentation model to obtain a cigarette segmentation map, wherein the cigarette segmentation map is used to indicate the presence of cigarette analogs in the current frame image.

[0013] The region determination module is used to determine the cigarette butt region from the cigarette segmentation map, wherein the cigarette butt region includes a coke region and an ash region.

[0014] The behavior determination module is used to determine that the cigarette analog is a cigarette in a burning state when the coke area and the ash area meet preset conditions, so as to determine that the user has smoking behavior based on the cigarette in a burning state.

[0015] In a third aspect, the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior determination method described in any embodiment of the present application.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the smoking behavior determination method described in any embodiment of the present application when executed.

[0017] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the smoking behavior determination method described in any embodiment of the present application.

[0018] The smoking behavior determination scheme provided in the embodiment of the present application first obtains a cigarette segmentation map related to cigarette analogs through a cigarette segmentation model; then determines the cigarette butt area from the cigarette segmentation map. In this embodiment, the burning characteristics of cigarettes are taken into account. The cigarette butt area includes a coke area and an ash area, and only when the coke area and the ash area meet the preset conditions, the cigarette analog is determined to be a cigarette, thereby improving the detection accuracy of cigarettes in a burning state. This solves the problem of misdetecting slender objects as cigarettes in the existing scheme, and has the beneficial effect of improving the accuracy of determining the user's smoking behavior.

[0019] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the smoking behavior determination device, or may be packaged separately from the processor of the smoking behavior determination device, which is not limited in this application.

[0020] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flow chart of the smoking behavior determination method provided in the embodiment of the present application.

[0025] Figure 2 It is a structural schematic diagram of a cigarette in a burning state provided in an embodiment of the present application.

[0026] Figure 3a This is another flowchart of the smoking behavior determination method provided in an embodiment of the present application.

[0027] Figure 3b It is another structural schematic diagram of a cigarette in a burning state provided in an embodiment of the present application.

[0028] Figure 4 It is a structural schematic diagram of the smoking behavior determination device provided in an embodiment of the present application.

[0029] Figure 5 It is a structural schematic diagram of an electronic device of a smoking behavior determination method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the present application will be clearly and completely described below in conjunction with the drawings in the present embodiment. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0033] Figure 1 A flowchart of a smoking behavior determination method provided in an embodiment of the present application is provided. This embodiment can be applied to identifying user behaviors captured in a video stream to determine whether the current user behavior is a smoking behavior. The method can be performed by a smoking behavior determination device, which can be implemented in the form of hardware and / or software and integrated in an electronic device that executes the method. Preferably, the electronic device in the embodiment of the present application can be a server, or a computer device, etc.

[0034] Specifically, please refer to Figure 1 The smoking behavior determination method provided in this embodiment can be implemented through the following steps S110~S130.

[0035] S110, inputting the current frame image in the video stream into the cigarette segmentation model to obtain a cigarette segmentation map.

[0036] Video stream, which means the video data received by the server. Among them, the current video data can be divided into multiple types according to the different scenarios of identifying smoking behavior. For example, taking the identification of the smoking behavior of users while driving a vehicle as an example, a high-resolution camera can be installed at an appropriate position inside the vehicle (such as the A-pillar or the steering wheel) to capture the upper body area of ​​the driver. The camera should be configured to continuously record the video stream and capture the possible behavior of the driver, so as to send the video stream captured by the camera to the server, so as to process the received video data at the server to extract at least one frame of image from the video data; it can also identify the smoking behavior of users in public places as an example, and a high-resolution camera can be installed at an appropriate position of a building in a public place to capture the user's behavior, so as to analyze whether the user has smoking behavior, etc.

[0037] In this embodiment, the cigarette segmentation model is obtained based on a deep learning algorithm. Specifically, this embodiment can obtain the cigarette segmentation model in the following manner: first, according to the actual application scenario, a large number of images of different users in the smoking process at multiple angles and multiple postures in the corresponding scenario are collected; then the cigarette contours appearing in each image are accurately annotated, and the cigarette edge contours are annotated so that the model can recognize the cigarette according to the annotated cigarette edge contours; then the annotated cigarette contours are learned through a semantic segmentation model, and optionally, the current semantic segmentation model can be implemented through a Unet model or an Hrnet model; finally, the model is trained using the annotated image data set, and when the training accuracy reaches convergence, the cigarette segmentation model is obtained so that it can obtain the cigarette segmentation result of the current frame image.

[0038] After the server obtains the image from the video stream, the smoking behavior determination scheme provided in this embodiment will be executed for each frame of the image. Specifically, for the current frame image, the current frame image will be input into the cigarette segmentation model to obtain a cigarette segmentation map. Among them, the cigarette segmentation map is used to indicate that there is a cigarette analog in the current frame image. It is easy to understand that when the current frame image is segmented, the cigarette segmentation model will segment out the objects in the image that contain features similar to cigarettes to obtain a cigarette segmentation map. The current feature similar to cigarettes can be a slender feature, and the corresponding object can be a straw, a lollipop stick, and a slender pen. In this embodiment, objects containing features similar to cigarettes are called cigarette analogs. In this embodiment, what is segmented in the cigarette segmentation map is not necessarily an actual cigarette, but a cigarette analog; and then the cigarette segmentation map needs to be executed in subsequent steps, so as to determine in the subsequent steps whether the cigarette analog in the cigarette segmentation map is a cigarette in a burning state, so as to determine whether the user in the current frame image has smoking behavior, etc.

[0039] Optionally, after the current frame image is input into the cigarette segmentation model, if the current frame image does not contain cigarette analogs, indicating that the user in the current frame image does not have smoking behavior, then the next frame image in the video stream is acquired and the next frame image is input into the cigarette segmentation model.

[0040] S120: Determine a cigarette butt region from the cigarette segmentation image, where the cigarette butt region includes a coke region and an ash region.

[0041] A cigarette butt region is determined from the cigarette segmentation image to determine whether the cigarette analog in the cigarette segmentation image has a burning behavior. The cigarette butt region indicates a region corresponding to when the cigarette is burning.

[0042] In this embodiment, the method of determining the cigarette butt area from the cigarette segmentation map can be: first, grayscale processing is performed on the original image corresponding to the obtained cigarette segmentation map to obtain a cigarette grayscale map; during the cigarette burning process, due to the cigarette burning characteristics, the grayscale value of the cigarette butt area in a burning state is generally smaller than the grayscale value of the unburned part. This is because the cigarette butt area in a burning state generally produces gray and black products. When expressed by grayscale values ​​in the grayscale map, the corresponding average grayscale value should be [5, 30], preferably, it can be 10 or 15, etc.; the unburned part is usually white, and the corresponding average grayscale value should be [235, 255], preferably, it can be 230 or 240, etc. Therefore, by comparing the average grayscale values ​​in the cigarette grayscale map, it can be determined that the area with a smaller grayscale value is the cigarette butt area.

[0043] A possible situation is that in the process of analyzing the cigarette analog in the cigarette segmentation map, if the cigarette butt area is determined by using the grayscale value corresponding to the above cigarette grayscale map, if the cigarette analog is a pen, and the pen holder is white and the pen tip is gray-black, then in the process of making a judgment, it is easy to determine the gray-black pen tip as the cigarette butt area that meets the above average grayscale value condition. Therefore, in order to accurately identify the cigarette in the burning state, the solution provided in this embodiment further defines the cigarette butt area to include the coke area and the ash area.

[0044] Please refer to Figure 2 , Figure 2 It is a structural diagram of a cigarette in a burning state provided in an embodiment of the present application. In this embodiment, the specific explanation of the coke area and the ash area is as follows: Due to the difference between the chemical reaction produced by the combustion and the chemical composition in the tobacco, during the smoking process, the cigarette butt will appear black, white and gray after burning, mainly black and grayish white. The black part is mainly coke, which is composed of carbonized substances left after the tobacco is burned. When tobacco burns, the carbon compounds in the tobacco undergo a combustion reaction to produce coke, which remains on the surface of the cigarette butt to form a black coke area, such as Figure 2Area A in the example; the grayish white part is mainly ash, which is composed of grayish white residual substances after tobacco combustion. After tobacco combustion, in addition to generating coke, it also produces some grayish white gas and particulate matter. These substances condense in the air and deposit on the surface of the cigarette butt, forming a grayish white ash area, such as Figure 2 Region B in the example. In the cigarette analog, even if the cigarette butt region can be detected, it is still necessary to further detect whether the cigarette butt region includes the coke region and the ash region to improve the accuracy of determining the cigarette.

[0045] Specifically, when determining the coke area and the ash area, it can be seen from the above description that during the cigarette burning process, the coke area mainly appears black and the ash area mainly appears grayish white, so the area with an average grayscale value close to the grayscale value corresponding to black can be determined as the ash area, and the area with an average grayscale value close to the average of the white grayscale value and the gray grayscale value can be determined as the ash area; optionally, by analyzing the cigarette burning characteristics, it is found that the ash area is close to the unburned cigarette rod position, therefore, the area in the cigarette butt area close to the unburned cigarette rod position can be determined as the ash area, and another part of the cigarette butt area can be determined as the coke area, and the specific method of determining the coke area and the ash area is not limited here.

[0046] S130: When the coke area and the ash area meet preset conditions, determine that the cigarette analog is a cigarette in a burning state, so as to determine that the user has a smoking behavior based on the cigarette in a burning state.

[0047] A possible situation is that after the above step S120, the cigarette butt area can be determined from the cigarette analog in the cigarette segmentation map, and further analysis of the grayscale value in the cigarette butt area can also determine the grayscale value that matches the coke area and the ash area, but it is also impossible to directly determine that the cigarette analog is a burning cigarette. The reason is that for the case where the cigarette analog is a pen, and the pen shaft is white and the pen tip is gray-black, there is also a situation where the corresponding coke area and ash area can be determined from the grayscale value corresponding to the gray-black pen tip. Therefore, in order to further improve the accurate identification of burning cigarettes, the solution provided in this embodiment needs to further determine whether the coke area and the ash area meet the preset conditions on the basis of S120, so that when the preset conditions are met, the cigarette analog can be determined as a cigarette in a burning state, thereby improving the accuracy of cigarette identification.

[0048] Specifically, the above-mentioned preset condition can be a corresponding condition defined by the first area corresponding to the coke area and the second area corresponding to the ash area. In this embodiment, the reason for determining the preset condition as the area condition of the corresponding area is that, during the smoking process, the gray-white ash produced by the burning of the cigarette butt is usually larger than the area of ​​the black coke. The specific reasons are explained as follows: 1. It is related to the coke formation speed. Since the coke formation speed is relatively slow, and the ash formation speed is fast, when smoking, the second area corresponding to the ash area will be larger than the first area corresponding to the coke area. 2. It is related to the combustion characteristics. When the cigarette butt burns, the coke may fall off gradually during the combustion process, while the ash is the residue after combustion, which will remain on the surface of the cigarette butt for a longer time, so the second area corresponding to the ash area will be relatively large. Therefore, the condition provided in this embodiment can be that when the second area corresponding to the ash area is not less than the first area corresponding to the coke area, it can be determined that the cigarette analog is a cigarette in a burning state.

[0049] Preferably, when the cigarette butt is burning, the ratio of the first area corresponding to the coke area to the second area corresponding to the ash area may be between 1:1 and 1:3. This means that the first area may be equal to the second area, or the first area may be 1 / 3 of the second area. The ratio of the first area to the second area in the specific preset conditions is not limited here.

[0050] To summarize, the smoking behavior determination scheme provided in this embodiment can determine that the cigarette is similar to a cigarette in a burning state only when there is a cigarette butt area in the cigarette similarity and the coke area and the ash area in the cigarette butt area meet the above-mentioned preset conditions. Further, it can be determined that the user in the current frame image has smoking behavior, thereby improving the accuracy of smoking behavior recognition.

[0051] The smoking behavior determination method provided in this embodiment first obtains a cigarette segmentation map related to a cigarette analogue through a cigarette segmentation model; then determines the cigarette butt region from the cigarette segmentation map. In this embodiment, the burning characteristics of cigarettes are taken into consideration, and the cigarette butt region includes a coke region and an ash region. Only when the coke region and the ash region meet the preset conditions, the cigarette analogue is determined to be a cigarette, thereby improving the detection accuracy of cigarettes in a burning state. This solves the problem of misdetecting slender objects as cigarettes in the existing solution, and has the beneficial effect of improving the accuracy of determining the user's smoking behavior.

[0052] Figure 3aThis is another flow chart of the method for determining smoking behavior provided by the embodiment of the present application. The embodiment of the present application is optimized on the basis of the above embodiments. Specifically, the optimization is as follows: This embodiment explains in detail the related processes of "determining the cigarette butt area from the cigarette segmentation map" and "inputting the current frame image in the video stream into the cigarette segmentation model to obtain the cigarette segmentation map" in the above embodiments. Figure 3a The smoking behavior determination method of this embodiment can be specifically implemented by the following steps S210 to S280.

[0053] S210: Input the current frame image in the video stream into the cigarette segmentation model.

[0054] S220: The cigarette segmentation model determines the image area corresponding to the target object from the current frame image.

[0055] The target object can be understood as an object similar to a cigarette analog in the current frame image, but in this embodiment, not all objects similar to cigarette analogs are analyzed, and the target object can be all objects of slender shape contained in the user's mouth or held in the user's hand. The actual purpose of this embodiment is to determine whether the cigarette analog is a cigarette in a burning state by executing the corresponding scheme of this embodiment on the cigarette analog. Then, in order to filter part of the images to be detected and reduce the data processing pressure of the cigarette segmentation model, when the cigarette segmentation model recognizes that the current frame image contains the target object, the image area corresponding to the target object will be obtained, and then the subsequent embodiments will determine whether the cigarette segmentation model needs to output the cigarette analog based on the image area corresponding to the target object, thereby improving the efficiency of the model outputting the cigarette segmentation map.

[0056] S230: When the image area corresponding to the target object and the image area corresponding to the current frame image satisfy a preset relationship, determine that the target object is a cigarette analog.

[0057] In this embodiment, if a target object is detected, it is determined whether the cigarette segmentation map needs to be output by determining whether the image area corresponding to the target object and the image area corresponding to the current frame image satisfy a preset relationship. The preset relationship in this embodiment is related to the length of the cigarette holder. The reason for this is that if the target object is a cigarette, then at least the cigarette holder should be included in the current frame image, and the preset relationship in this embodiment should not be less than the value of the area corresponding to the length of the cigarette holder to the image area corresponding to the current frame image.

[0058] For example, the preset relationship is 1%, 2%, or 3%, etc., that is, when it is determined that the image area corresponding to the target object is not less than 3% of the image area corresponding to the current frame image, the target object is determined to be a cigarette analog. If the preset relationship is not satisfied, it indicates that the target object is not a cigarette, and further, it can be determined not to be a cigarette analog, and subsequent operations are not performed on the current frame image, and the operation of acquiring the next frame image in the video stream is performed.

[0059] S240: Output a cigarette segmentation map corresponding to the cigarette analog.

[0060] In this embodiment, the cigarette segmentation map is used to indicate that a cigarette analogue exists in the current frame image.

[0061] S250, grayscale processing is performed on the cigarette segmentation image to obtain a cigarette grayscale image.

[0062] In this embodiment, the cigarette segmentation map indicates the original image containing cigarette analogs obtained from the current frame image, which can be realized based on the cutout technology. Then, in order to further analyze the corresponding features contained in the cigarette segmentation map, in this embodiment, the cigarette segmentation map is gray-scaled to obtain a cigarette grayscale map.

[0063] S260: Determine a cigarette butt ring region from the cigarette grayscale image, where the cigarette butt ring region divides the cigarette grayscale image into a first grayscale region and a second grayscale region.

[0064] Please refer to Figure 3b , Figure 3b is another structural schematic diagram of a cigarette in a burning state provided in an embodiment of the present application. The cigarette butt ring area refers to a circle of obvious black or darker area (such as Figure 3b O area in the example) is mainly caused by the physical and chemical processes in the combustion process. The specific reason is that the part of the cigarette butt close to the cigarette rod is in close contact with the cigarette rod, and the oxygen supply is often insufficient. Tobacco combustion requires oxygen. Insufficient oxygen will lead to incomplete combustion and produce a large amount of carbonized substances, forming a black cigarette butt ring, which is an important feature of the cigarette butt combustion process. In this embodiment, if the cigarette butt ring area can be determined from the cigarette grayscale image, slender objects such as pens can be distinguished from burning cigarettes, and unburned cigarettes can be distinguished from burning cigarettes. At the same time, the cigarette butt ring area can also be used to divide the cigarette into a first grayscale area and a second grayscale area. In the current process, it is only known that the cigarette butt ring area can divide the cigarette into a first grayscale area and a second grayscale area. So whether the first grayscale area belongs to the cigarette butt area or the second grayscale area belongs to the cigarette butt area, it is necessary to perform the subsequent step S270 to make relevant judgments.

[0065] A preferred implementation method, in this embodiment, the method of determining the cigarette butt ring area from the cigarette grayscale image can be implemented by the following steps a) and b):

[0066] a) Performing annular fitting processing on the cigarette grayscale image to determine whether the cigarette grayscale image contains an annular region.

[0067] In this embodiment, before the cigarette grayscale image is subjected to circular fitting processing, the cigarette grayscale image can be preprocessed using morphological operations, such as corrosion and expansion. The current preprocessing can eliminate noise, smooth the image, and fill holes, which is helpful to extract the contour of the cigarette ring; and then use contour detection algorithms, such as edge detection or connected area marking, to detect the regional contour in the image. The cigarette ring is usually a continuous closed contour, and the continuous closed contour is detected; finally, the closed loop contour detected in the cigarette grayscale image is subjected to circular fitting. The cigarette ring is usually a circle or a part of a circle. By fitting the circle, the exact position, size and shape characteristics of the cigarette ring can be further determined. This helps to eliminate the situation of irregular or deformed contours and improve the accuracy of cigarette ring detection. In this embodiment, the circular fitting processing method can be implemented by the Hough transform algorithm.

[0068] Therefore, in this embodiment, it can be determined whether the cigarette grayscale image contains a cigarette butt ring area by determining whether the cigarette grayscale image contains a ring area. Then, in this embodiment, if the ring area can be detected, the following step b needs to be performed, and the ring area is the cigarette butt ring area; if the ring area cannot be detected in the current process, it is determined that the cigarette grayscale image does not contain a cigarette butt ring area, thereby determining that the cigarette analog is not a cigarette in a burning state, and no subsequent operations need to be performed.

[0069] b) When the cigarette grayscale image includes an annular area and the grayscale value of the annular area is lower than a first preset value, the annular area is determined to be a cigarette butt ring area.

[0070] In this embodiment, even if it is determined that the cigarette grayscale image contains a ring area, in order to accurately distinguish the cigarettes, the grayscale value of the ring area needs to be calculated. Figure 3b It can be seen that the cigarette ring area is mainly black. In this embodiment, when it is determined that the cigarette grayscale image contains a ring area and the grayscale value of the ring area is lower than the first preset value, the ring area is determined to be a cigarette ring area. Among them, the current first preset value is related to the grayscale value corresponding to black. In this embodiment, the first preset value can be [0, 10], preferably 3 or 5. The selection of the specific first preset value is not limited here.

[0071] Furthermore, when the cigarette grayscale image contains a ring area, but the grayscale value of the ring area is not lower than the first preset value, this embodiment determines that the cigarette grayscale image does not contain the cigarette butt ring area, and determines that the cigarette analog is not a cigarette in a burning state, and there is no need to perform subsequent operations.

[0072] This embodiment can perform preliminary screening on whether the cigarette analog is a cigarette in a burning state by determining whether the cigarette ring area exists, and execute subsequent operations when the cigarette analog contains the cigarette ring area. Otherwise, it is determined that the cigarette analog contained in the current frame image is not a cigarette, and the relevant operations of inputting the next frame image in the video stream into the cigarette segmentation model are executed, which can speed up the recognition efficiency of the current frame image.

[0073] S270 , comparing the grayscale value corresponding to the first grayscale area with the grayscale value corresponding to the second grayscale area, and determining that the grayscale area with the smaller grayscale value is the cigarette butt area.

[0074] Since the burning cigarette is divided into the first grayscale area and the second grayscale area by determining the cigarette butt ring area, it is not clear which grayscale area represents the cigarette butt area. In this embodiment, by comparing the grayscale value corresponding to the first grayscale area with the grayscale value corresponding to the second grayscale area, the grayscale area with the smaller grayscale value is the cigarette butt area. Figure 3b It can be seen that during the burning process of a cigarette, the cigarette butt area will produce black coke and gray ash, and the corresponding grayscale value is smaller. The cigarette rod area is the unburned part, which is usually white and has a smaller corresponding grayscale value. Therefore, by calculating the grayscale value corresponding to the first grayscale area and the grayscale value corresponding to the second grayscale area, the area with a smaller grayscale value can be determined as the cigarette butt area, that is, Figure 3b The M area shown; the area with a larger gray value is determined as the cigarette rod area, that is, Figure 3b The N region is shown.

[0075] In this embodiment, the cigarette butt area includes a coke area and an ash area. Then the coke area and the ash area are determined in the following manner: grayscale processing is performed on the cigarette butt area to obtain a cigarette butt grayscale map; the area in the cigarette butt grayscale map with an average grayscale value lower than the second preset value is determined as the coke area; the area in the cigarette butt grayscale map with an average grayscale value higher than the third preset value is determined as the ash area. The purpose of this is that while the coke area and the ash area are accurately divided by the second preset value and the third preset value, even if the cigarette butt area contains a similar grayscale value, there are very few cigarette analogs with coke areas and ash areas that meet the second preset value and the third preset value, thereby improving the accuracy of determining the user's smoking behavior in the subsequent steps.

[0076] Among them, the above-mentioned second preset value can be the area where the pixel grayscale values ​​with an average grayscale value lower than 15 are located in the cigarette butt area M, that is, the coke area A can be determined from the cigarette butt area M; the above-mentioned third preset value can be the area where the pixel grayscale values ​​with an average grayscale value greater than 200 are located in the cigarette butt area M, that is, the ash area B can be determined from the cigarette butt area M.

[0077] S280: When the coke area and the ash area meet preset conditions, determine that the cigarette analog is a cigarette in a burning state, so as to determine that the user has a smoking behavior based on the cigarette in a burning state.

[0078] Through the above steps S210~S280, it can be determined that in this embodiment, only when the cigarette ring area and the cigarette area exist in the cigarette analog at the same time, and the coke area and the ash area in the cigarette area meet the above preset conditions, can the cigarette analog be determined to be a cigarette in a burning state, and further, it can be determined that the user in the current frame image has smoking behavior, thereby improving the accuracy of recognizing smoking behavior.

[0079] In another preferred implementation, the improved solution of this embodiment, after determining that the user has a smoking behavior according to the cigarette in a burning state, further includes: determining at least one target hardware information to be linked; calling the target hardware device corresponding to the at least one target hardware information to provide a corresponding linkage service based on the at least one target hardware device. The purpose of this is to remind the user not to smoke through the linkage service to avoid dangerous accidents.

[0080] In an application scenario, taking the detection of the driver's smoking behavior while driving a vehicle as an example, at least one target hardware information to be linked may be the voice module information and window information in the cockpit. Thus, after determining that the user has smoking behavior, the window can be opened by the user based on the window information to ensure air flow. At the same time, a voice reminder can be provided to the user based on the voice module information to remind the user of abnormal behavior and to pay attention to safe driving, thereby reducing the incidence of driving accidents.

[0081] Another application example, the linkage service of this embodiment can further track the location of cigarettes when a burning cigarette is detected, and detect the cigarette butt close to the car window. If a cigarette butt is detected close to the car window and suddenly disappears, it may mean that the burning cigarette has been thrown out of the window. When the act of throwing a burning cigarette out of the window is detected, the fire protection system can be immediately triggered to prevent safety problems such as fires, so as to maintain road traffic order and environmental sanitation.

[0082] The improved smoking behavior determination method of the present embodiment first determines whether the image area corresponding to the target object and the image area corresponding to the current frame image satisfy a preset relationship through a cigarette segmentation model, and performs preliminary screening of cigarette analogs to reduce data processing pressure; secondly, by determining whether the cigarette grayscale image corresponding to the cigarette segmentation image contains a cigarette butt ring area, when the cigarette butt ring area is not included, it can be determined that the cigarette analog is not a cigarette in a burning state, so as to ensure the accuracy of determining the smoking behavior in subsequent steps; finally, when the cigarette butt ring area exists, by determining that the coke area and the ash area in the cigarette butt area satisfy the preset conditions, it can be determined that the cigarette analog is a cigarette in a burning state, and then it can be determined that the user in the current frame image has smoking behavior, thereby improving the accuracy of recognizing smoking behavior.

[0083] Figure 4 is a schematic diagram of a smoking behavior determination device provided in an embodiment of the present application, and the device is suitable for executing the smoking behavior determination method provided in an embodiment of the present application. Figure 4 As shown, the device may specifically include: an image input module 410 , a region determination module 420 and a behavior determination module 430 .

[0084] The image input module 410 is used to input the current frame image in the video stream into the cigarette segmentation model to obtain a cigarette segmentation map, wherein the cigarette segmentation map is used to indicate the presence of cigarette analogs in the current frame image.

[0085] The region determination module 420 is used to determine the cigarette butt region from the cigarette segmentation image, where the cigarette butt region includes a coke region and an ash region.

[0086] The behavior determination module 430 is used to determine that the cigarette analog is a cigarette in a burning state when the coke area and the ash area meet preset conditions, so as to determine that the user has a smoking behavior based on the cigarette in a burning state.

[0087] The improved smoking behavior determination device of this embodiment first obtains a cigarette segmentation map related to the cigarette analog through a cigarette segmentation model; then determines the cigarette butt area from the cigarette segmentation map. In this embodiment, the burning characteristics of cigarettes are taken into account. The cigarette butt area includes a coke area and an ash area, and only when the coke area and the ash area meet the preset conditions, the cigarette analog is determined to be a cigarette, thereby improving the detection accuracy of cigarettes in a burning state. This solves the problem of misdetecting slender objects as cigarettes in the existing solution, and has the beneficial effect of improving the accuracy of determining the user's smoking behavior.

[0088] In one embodiment, the region determination module 420 includes a grayscale processing unit and a region determination unit.

[0089] The grayscale processing unit is used to perform grayscale processing on the cigarette segmentation image to obtain a cigarette grayscale image.

[0090] The area determination unit is used to determine the cigarette butt ring area from the cigarette grayscale image, and the cigarette butt ring area divides the cigarette grayscale image into a first grayscale area and a second grayscale area; it is also used to compare the grayscale value corresponding to the first grayscale area with the grayscale value corresponding to the second grayscale area, and determine that the grayscale area with a smaller grayscale value is the cigarette butt area, and the cigarette butt area is the first grayscale area or the second grayscale area.

[0091] In one embodiment, the region determination unit includes a fitting processing subunit and a region determination subunit.

[0092] The fitting processing subunit is used to perform annular fitting processing on the cigarette grayscale image to determine whether the cigarette grayscale image contains an annular area.

[0093] The area determination subunit is used to determine that the annular area is the cigarette butt ring area when the cigarette grayscale image contains the annular area and the grayscale value of the annular area is lower than a preset value.

[0094] In one embodiment, the area determination subunit is further used to determine that the cigarette grayscale image does not include the cigarette butt ring area, and to determine that the cigarette analog is not a cigarette in a burning state when the cigarette grayscale image does not include the annular area or the grayscale value of the annular area is not lower than the first preset value.

[0095] In one embodiment, the device further includes: a grayscale processing module.

[0096] The grayscale processing module is used to perform grayscale processing on the cigarette butt area to obtain a cigarette butt grayscale image.

[0097] The area determination module is specifically used to determine the area in the cigarette butt grayscale image where the average grayscale value is lower than the second preset value as the coke area; and determine the area in the cigarette butt grayscale image where the average grayscale value is higher than the third preset value as the ash area.

[0098] In one embodiment, the device further comprises: an area determination module, a similar object determination module and a segmentation map output module.

[0099] The area determination module is used by the cigarette segmentation model to determine the image area corresponding to the target object from the current frame image.

[0100] The analogue determination module is used to determine that the target object is the cigarette analogue when the image area corresponding to the target object and the image area corresponding to the current frame image satisfy a preset relationship, wherein the preset relationship is related to the length of the cigarette holder.

[0101] The segmentation map output module is used to output the cigarette segmentation map corresponding to the cigarette analog.

[0102] In one embodiment, the apparatus further includes an information determination module and a device calling module.

[0103] The information determination module is used to determine at least one target hardware information to be linked.

[0104] The device calling module is used to call at least one target hardware device corresponding to the target hardware information to provide a corresponding linkage service based on at least one target hardware device.

[0105] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0106] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior determination method described in any embodiment of the present application.

[0107] An embodiment of the present application further provides a computer-readable medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the smoking behavior determination method described in any embodiment of the present application when executed.

[0108] Reference below Figure 5 , Figure 5 1 is a schematic diagram of the structure of an electronic device for a method for determining smoking behavior provided in an embodiment of the present application, and shows a schematic diagram of the structure of a computer system 500 suitable for implementing the electronic device of the embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0110] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0111] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0112] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, etc., or any suitable combination of the above.

[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The modules and / or units involved in the embodiments described in the present application may be implemented by software or hardware. The modules and / or units described may also be set in a processor. For example, it may be described as: a processor includes an image input module, a region determination module, and a behavior determination module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0115] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: inputting the current frame image in the video stream into the cigarette segmentation model to obtain a cigarette segmentation map, and the cigarette segmentation map is used to indicate that there is a cigarette analog in the current frame image; determining a cigarette butt area from the cigarette segmentation map, and the cigarette butt area includes a coke area and an ash area; when the coke area and the ash area meet preset conditions, determining that the cigarette analog is a cigarette in a burning state, so as to determine that the user has a smoking behavior based on the cigarette in a burning state.

[0116] According to the technical solution of this embodiment, the burning characteristics of cigarettes are taken into consideration, and a cigarette segmentation map related to cigarette analogs is obtained through a cigarette segmentation model; then the cigarette butt area is determined from the cigarette segmentation map, and the cigarette butt area includes a coke area and an ash area, and only when the coke area and the ash area are detected to meet the preset conditions, the cigarette analog is determined to be a cigarette, thereby improving the detection accuracy of cigarettes in a burning state. This solves the problem of misdetecting slender objects as cigarettes in the existing solution, and has the beneficial effect of improving the accuracy of determining the user's smoking behavior.

[0117] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for determining smoking behavior, characterized in that: include: Inputting a current frame image in a video stream into a cigarette segmentation model to obtain a cigarette segmentation map, wherein the cigarette segmentation map is used to indicate the presence of cigarette analogs in the current frame image; Determine a cigarette butt region from the cigarette segmentation image, wherein the cigarette butt region includes a coke region and an ash region; When the coke area and the ash area meet preset conditions, determining that the cigarette analog is a cigarette in a burning state, so as to determine that the user has a smoking behavior based on the cigarette in a burning state; Wherein, the preset condition is that the second area corresponding to the ash region is not less than the first area corresponding to the coke region.

2. The method for determining smoking behavior according to claim 1, characterized in that: The step of determining the cigarette butt region from the cigarette segmentation map comprises: Performing grayscale processing on the cigarette segmentation image to obtain a cigarette grayscale image; Determine a cigarette butt ring region from the cigarette grayscale image, wherein the cigarette butt ring region divides the cigarette grayscale image into a first grayscale region and a second grayscale region; Compare the grayscale value corresponding to the first grayscale area with the grayscale value corresponding to the second grayscale area, determine that the grayscale area with the smaller grayscale value is the cigarette butt area, and the cigarette butt area is the first grayscale area or the second grayscale area.

3. The method for determining smoking behavior according to claim 2, characterized in that: Determining the cigarette butt ring area from the cigarette grayscale image includes: Performing a ring fitting process on the cigarette grayscale image to determine whether the cigarette grayscale image contains a ring area; When the cigarette grayscale image contains the annular area and the grayscale value of the annular area is lower than a first preset value, the annular area is determined to be the cigarette butt ring area.

4. The method for determining smoking behavior according to claim 3, characterized in that: The method further comprises: When the cigarette grayscale image does not include the annular area, or the grayscale value of the annular area is not lower than the first preset value, it is determined that the cigarette grayscale image does not include the cigarette butt ring area, and it is determined that the cigarette analog is not a cigarette in a burning state.

5. The method for determining smoking behavior according to claim 1, characterized in that: The coke region and the ash region are determined as follows: Performing grayscale processing on the cigarette butt region to obtain a cigarette butt grayscale image; Determine the area in the cigarette butt grayscale image where the average grayscale value is lower than a second preset value as the coke area; The area in the cigarette butt grayscale image where the average grayscale value is higher than a third preset value is determined as the ash area.

6. The method for determining smoking behavior according to claim 1, characterized in that: After the current frame image in the video stream is input into the cigarette segmentation model, it also includes: The cigarette segmentation model determines the image area corresponding to the target object from the current frame image; When the image area corresponding to the target object and the image area corresponding to the current frame image satisfy a preset relationship, determining that the target object is the cigarette analog, wherein the preset relationship is related to the length of the cigarette holder; Outputting a cigarette segmentation map corresponding to the cigarette analog.

7. The method for determining smoking behavior according to claim 1, characterized in that: After determining that the user has smoking behavior based on the burning cigarette, the method further includes: Determine at least one target hardware information to be linked; At least one target hardware device corresponding to the target hardware information is called to provide a corresponding linkage service based on the at least one target hardware device.

8. A smoking behavior determination device, characterized in that: include: An image input module, used for inputting a current frame image in a video stream into a cigarette segmentation model to obtain a cigarette segmentation map, wherein the cigarette segmentation map is used for indicating the presence of cigarette analogs in the current frame image; A region determination module, used for determining a cigarette butt region from the cigarette segmentation map, wherein the cigarette butt region includes a coke region and an ash region; a behavior determination module, configured to determine that the cigarette analog is a cigarette in a burning state when the coke area and the ash area meet preset conditions, so as to determine that the user has a smoking behavior based on the cigarette in a burning state; Wherein, the preset condition is that the second area corresponding to the ash region is not less than the first area corresponding to the coke region.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the smoking behavior determination method according to any one of claims 1 to 7 is implemented.

Images