Smoking behavior determination method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for determining smoking behavior, which relates to computer technology. The method comprises: acquiring a plurality of frames of continuous exhalation images, each exhalation image comprising a chest key point and a hand key point; determining a hand angle change sequence and a chest angle change sequence respectively according to each chest key point and each hand key point; calculating sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether the user has smoking behavior according to the sequence features. The present application determines the sequence features by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence. In essence, the application analyzes the movement direction of the chest key points and the hand key points during the user's exhalation process, thereby determining the user's smoking behavior. The implementation process of the present solution can improve the accuracy of the determined smoking behavior.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for determining smoking behavior. Background Art

[0002] With the improvement of vehicle intelligence, monitoring user behavior while driving and providing in-vehicle linkage services based on the behavior monitoring results is an effective way to improve user driving experience.

[0003] Existing behavior monitoring systems mainly rely on identifying relevant behavior features to determine whether a user is performing specific activities while driving. Taking the identification of a user's smoking behavior as an example, existing solutions generally use visual technology to identify relevant physical features of a burning cigarette, such as the appearance features of the cigarette, the smoke features, and the features of the cigarette butt, so as to determine that the user is smoking when the relevant physical features are present.

[0004] The above-mentioned scheme for determining smoking behavior has the problem that when the relevant physical features are not obvious, the final determination of the smoking behavior is inaccurate due to inaccurate feature recognition. Summary of the invention

[0005] The present application provides a smoking behavior determination method, device, electronic device and storage medium, which can improve the accuracy of determining smoking behavior.

[0006] In a first aspect, the present application provides a method for determining smoking behavior, comprising: acquiring a plurality of frames of continuous exhalation images, each of the exhalation images comprising a chest key point and a hand key point; determining a hand angle change sequence and a chest angle change sequence respectively according to each of the chest key points and each of the hand key points, the hand angle change sequence being obtained by determining an angle change situation related to the hand key point in each of the exhalation images; the chest angle change sequence being obtained by determining an angle change situation related to the chest key point in each of the exhalation images; calculating sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether a user has smoking behavior according to the sequence features, the sequence features being obtained by analyzing a change direction corresponding to the hand angle change sequence and a change direction corresponding to the chest angle change sequence.

[0007] Optionally, the acquiring of several frames of continuous exhalation images includes: for at least one frame of image in the video stream, acquiring the hand key points and the chest key points in each frame of image; when the hand key points and the chest key points in the current frame of image satisfy a preset relationship, determining the several frames of continuous exhalation images from continuous images after the current frame of image according to breathing characteristics.

[0008] Optionally, the respiratory characteristics include an exhalation cycle and a direction in which the chest key points change; and determining the several frames of continuous exhalation images from the continuous images after the current frame image based on the respiratory characteristics includes: when it is detected that the exhalation cycle exceeds a preset cycle and the direction in which the chest key points corresponding to the continuous images after the current frame image change is a preset direction, determining that the continuous images corresponding to the exhalation cycle are the several frames of continuous exhalation images.

[0009] Optionally, the hand key point and the chest key point in the current frame image satisfy a preset relationship, including: when the distance between the hand key point and the chest key point is less than a preset distance, and the angle formed by the hand key point and the chest key point is less than a preset angle, determining that the current frame image satisfies the preset relationship.

[0010] Optionally, the calculating of sequence features of the hand angle change sequence and the chest angle change sequence includes: performing frequency domain conversion on the hand angle change sequence and the chest angle change sequence respectively based on Fourier transform to obtain a hand angle frequency domain sequence and a chest angle frequency domain sequence; calculating the covariance of the hand angle frequency domain sequence and the chest angle frequency domain sequence, and obtaining the sequence features according to the covariance.

[0011] Optionally, the sequence feature includes an inconsistency feature; determining whether the user has the smoking behavior based on the sequence feature includes: judging whether the covariance is less than a preset threshold; when the covariance is less than the preset threshold, the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence have the inconsistency feature, then it is determined that the user has the smoking behavior; when the covariance is not less than the preset threshold, the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence do not have the inconsistency feature, then it is determined that the user does not have the smoking behavior.

[0012] Optionally, after determining that the user has smoking behavior, the method further includes: determining at least one target hardware information to be linked; calling at least one target hardware device corresponding to the target hardware information to provide a corresponding linkage service based on the at least one target hardware device.

[0013] In a second aspect, the present application provides a smoking behavior determination device, the device comprising: an image acquisition module, for acquiring a plurality of frames of continuous exhalation images, each of the exhalation images comprising a chest key point and a hand key point.

[0014] A sequence determination module is used to determine a hand angle change sequence and a chest angle change sequence respectively according to each of the chest key points and each of the hand key points, wherein the hand angle change sequence is obtained by determining the angle change related to the hand key point in each of the exhalation images; and the chest angle change sequence is obtained by determining the angle change related to the chest key point in each of the exhalation images.

[0015] The behavior determination module is used to calculate the sequence characteristics of the hand angle change sequence and the chest angle change sequence, and determine whether the user has smoking behavior based on the sequence characteristics, wherein the sequence characteristics are obtained by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence.

[0016] In a third aspect, the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior determination method described in any embodiment of the present application.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the smoking behavior determination method described in any embodiment of the present application when executed.

[0018] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the smoking behavior determination method described in any embodiment of the present application.

[0019] The smoking behavior determination scheme provided in the embodiment of the present application first obtains several frames of continuous exhalation images, each of which includes chest key points and hand key points; then determines the hand angle change sequence and chest angle change sequence respectively according to each chest key point and each hand key point; finally calculates the sequence features of the hand angle change sequence and the chest angle change sequence to determine whether the user has smoking behavior according to the sequence features. This embodiment determines the sequence features by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence. In essence, it analyzes the movement direction of the chest key points and the hand key points during the user's exhalation process, so as to determine the user's smoking behavior. The implementation process of this scheme does not need to rely on the recognition of the physical features of the cigarette in a burning state, and can improve the accuracy of the determined smoking behavior.

[0020] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the smoking behavior determination device, or may be packaged separately from the processor of the smoking behavior determination device, which is not limited in this application.

[0021] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0023] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flow chart of the smoking behavior determination method provided in the embodiment of the present application.

[0026] Figure 2 It is a schematic diagram of the angles related to the hand key points and the chest key points respectively provided in the embodiments of the present application.

[0027] Figure 3 This is another flowchart of the smoking behavior determination method provided in an embodiment of the present application.

[0028] Figure 4 It is a structural schematic diagram of the smoking behavior determination device provided in an embodiment of the present application.

[0029] Figure 5 It is a structural schematic diagram of an electronic device of a smoking behavior determination method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the present application will be clearly and completely described below in conjunction with the drawings in the present embodiment. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0033] Figure 1 A flowchart of a smoking behavior determination method provided in an embodiment of the present application. This embodiment can be applied to determine whether a user has smoking behavior. The method can be performed by a smoking behavior determination device, which can be implemented in the form of hardware and / or software and integrated in an electronic device that executes the method. Preferably, the electronic device in the embodiment of the present application can be a server, or a computer device, etc.

[0034] refer to Figure 1 The smoking behavior determination method of this embodiment includes but is not limited to the following steps:

[0035] S110, acquiring a plurality of frames of continuous exhalation images, each of which includes chest key points and hand key points.

[0036] Several frames of continuous exhalation images represent images generated by the user during the exhalation process. It is easy to know that a continuous video stream contains the user's inhalation process and exhalation process. The key solution of this embodiment is to analyze several frames of continuous exhalation images corresponding to the user's exhalation process to ultimately determine whether the user is currently smoking.

[0037] In this embodiment, each exhalation image includes a chest key point and a hand key point. The chest key point is a point corresponding to a certain position on the user's chest. For example, the key point is determined from the corresponding position of the left chest or the right chest. Preferably, the chest key point in this embodiment indicates the key point corresponding to the left chest; the hand key point is a point corresponding to a certain position on the hand. For example, the key point is determined from the wrist, the back of the hand or the finger joint. Preferably, the hand key point in this embodiment indicates the key point corresponding to the right wrist. The specific position selection of the hand key point and the chest key point is not limited here, as long as the movement of the corresponding key point can be clearly captured by the camera installed in the car, and it is convenient for the analysis of the subsequent steps.

[0038] In this embodiment, it is possible to determine whether the user has a smoking behavior through the related characteristics of the chest key points and the hand key points included in each exhalation image in a plurality of frames of continuous exhalation images. The reason for this is that, through analysis of the inhalation process and the exhalation process corresponding to the user's smoking behavior, it can be known that during the inhalation process, the hand holding the cigarette often does not move near the mouth, so the position of the hand key points shown in the continuous images does not have any change, while the chest key points will show a trend of moving in the first direction (outward) in the continuous images due to the expansion of the chest cavity during the inhalation process; during the exhalation process, the hand holding the cigarette will leave the mouth, that is, the hand key points shown in the continuous images will move to the outside of the body with the movement of the hand, and the chest key points will show a trend of moving in the second direction (inward relative to the above-mentioned first direction) in the continuous images due to the contraction of the chest cavity during the exhalation process.

[0039] In this embodiment, the purpose is to determine whether the user has a smoking behavior by analyzing the movement direction of the chest key points and the hand key points. Since the hand key points do not move during the inhalation process, while the chest key points will move outward due to the expansion of the chest cavity. Then, the movement directions of the hand key points and the chest key points during the inhalation process are neither the same nor opposite, that is, the movement directions of the hand key points and the chest key points do not have a corresponding relationship. Therefore, the continuous images corresponding to the inhalation process are not convenient for determining whether the user has a smoking behavior; and during the exhalation process, the hand key points will move to the outside of the body, and the chest key points will move inward due to the contraction of the chest cavity. Then, this embodiment analyzes that when the user has a smoking behavior, the movement directions of the hand key points and the chest key points during the exhalation process are completely opposite, so that the user can be determined to have a smoking behavior based on the current completely opposite characteristics.

[0040] S120: Determine a hand angle change sequence and a chest angle change sequence according to each chest key point and each hand key point respectively.

[0041] In this embodiment, the hand angle change sequence is obtained by determining the angle change related to the hand key point in each exhalation image. Specifically, the angle change related to the hand key point can be determined by taking the hand key point as the coordinate origin, the vertical direction of the ground as the vertical coordinate, and the horizontal direction parallel to the ground as the horizontal coordinate. Then, the angle related to the hand key point can be the first angle generated by the line connecting the hand key point and the chest key point and the vertical coordinate in each frame of the image. Specifically, refer to Figure 2 , Figure 2 is a schematic diagram of the angles related to the key points of the hand and the key points of the chest provided in the embodiment of the present application. Figure 2 In the figure, point M represents a key point of the hand. In this embodiment, the determined hand angle change sequence is obtained by the change of ∠A in several frames of continuous exhalation images.

[0042] In this embodiment, the chest angle change sequence is obtained by determining the angle change related to the chest key point in each exhalation image. Specifically, the angle change related to the chest key point can be determined by taking the chest key point as the coordinate origin, the vertical direction of the ground as the vertical coordinate, and the horizontal direction parallel to the ground as the horizontal coordinate. Then, the angle related to the chest key point can be the second angle generated by the line connecting the chest key point and the hand key point and the vertical coordinate in each frame of the image. Specifically, refer to Figure 2 , among which, in Figure 2 In the figure, point N represents the key points of the chest. In this embodiment, the determined chest angle change sequence is obtained by the change of ∠B in several frames of continuous exhalation images.

[0043] Specifically, when determining the angle change sequence and the chest angle change sequence, linear fitting can be performed on the first angle and the second angle in several frames of continuous exhalation images, respectively, to obtain corresponding linear equations, so as to use mathematical expressions to respectively express the corresponding hand angle change sequence and chest angle change sequence, and the specific expression method is not limited here.

[0044] S130, calculating sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether the user has smoking behavior according to the sequence features.

[0045] In this embodiment, the sequence features are obtained by analyzing the change directions corresponding to the hand angle change sequence and the change directions corresponding to the chest angle change sequence.

[0046] Then, when the changing direction corresponding to the hand angle changing sequence and the changing direction corresponding to the chest angle changing sequence are completely opposite by analyzing, it can be determined that in the current exhalation process, there are characteristics that the key points of the user's hands move toward the outside of the body and the key points of the chest move toward the inside of the body due to exhalation, and at this time it can be determined that the user has a smoking behavior; if, after analysis, the changing direction corresponding to the hand angle changing sequence and the changing direction corresponding to the chest angle changing sequence do not have completely opposite characteristics, it indicates that the user does not have a smoking behavior in the current frame of continuous exhalation images, and the next several frames of continuous exhalation images can be obtained to execute the smoking behavior determination scheme provided in this embodiment for the next several frames of continuous exhalation images.

[0047] The smoking behavior determination method provided in this embodiment first obtains several frames of continuous exhalation images, each of which includes chest key points and hand key points; then determines the hand angle change sequence and chest angle change sequence respectively according to each chest key point and each hand key point; finally calculates the sequence features of the hand angle change sequence and the chest angle change sequence to determine whether the user has smoking behavior according to the sequence features. This embodiment determines the sequence features by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence. In essence, it analyzes the movement direction of the chest key points and the hand key points during the user's exhalation process, so as to determine the user's smoking behavior. The implementation process of this solution does not need to rely on the recognition of the physical features of the cigarette in a burning state, and can improve the accuracy of the determined smoking behavior.

[0048] Figure 3This is another flow chart of the method for determining smoking behavior provided by the embodiment of the present application. The embodiment of the present application is optimized on the basis of the above embodiments. Specifically, the optimization is as follows: This embodiment explains in detail the optimization process of "obtaining a plurality of frames of continuous exhalation images" and "calculating the sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether the user has smoking behavior according to the sequence features". Specifically, please refer to Figure 3 The smoking behavior determination method of this embodiment is implemented by the following steps S310~S352.

[0049] S310: For at least one frame of image in the video stream, obtain hand key points and chest key points in each frame of image.

[0050] Video stream, which means the video data received by the server. Among them, the current video data can be divided into multiple types according to different scenarios for identifying smoking behavior. For example, taking the identification of a user's smoking behavior while driving a vehicle as an example, a high-resolution camera can be installed at an appropriate location inside the vehicle (such as the A-pillar or the steering wheel) to capture the driver's upper body area. The camera should be configured to continuously record the video stream and capture the possible behavior of the driver, so that the video stream captured by the camera is sent to the server, so that the received video data can be processed on the server to extract at least one frame of image from the video data, so as to be able to analyze whether the user has smoking behavior, etc. based on the extracted at least one frame of image.

[0051] In this embodiment, the method of extracting the hand key points and chest key points in each frame image can be to first define the hand key point as the wrist and the chest key point as the left chest of the driver. In this embodiment, a key point detection model can be constructed based on a deep learning method, so that when a frame of image to be detected is input to the key point detection model, the hand key points and chest key points contained in the current frame image can be output. When the key point detection model cannot output the hand key points and chest key points contained in the current frame image, it indicates that the current frame image does not contain the user's smoking behavior, and no subsequent operations need to be performed.

[0052] Specifically, the method for training the key point detection model in this embodiment can be: first, collect a large number of images containing different drivers and different angles of the drivers; second, accurately annotate these images, annotate the position coordinates of the hand and chest key points, and these annotated data will be used to train the key point detection model; then select a suitable deep learning model for semantic segmentation; finally, use the annotated image data set to train the model so that it can recognize the chest key points and hand key points defined above. The training process should include adjusting the model parameters and the recognition accuracy of the key points. Until the final model output result reaches a certain convergence accuracy, the training of the image segmentation model is completed.

[0053] S311. When the hand key points and the chest key points in the current frame image satisfy a preset relationship, determine a plurality of frames of continuous exhalation images from continuous images after the current frame image according to the breathing characteristics.

[0054] Since the server receives a large number of video streams, in order to reduce the processing pressure on the server, the present embodiment does not need to execute the smoking behavior scheme provided in the present embodiment for each frame image in the video stream. It is only necessary to identify the hand key points and the chest key points in each frame image, so that when it is detected that the hand key points and the chest key points in the current frame image meet the preset relationship, the subsequent operation steps provided in the present embodiment can be executed; accordingly, when it is detected that the hand key points and the chest key points in the current frame image do not meet the preset relationship, it is not necessary to determine that there is no smoking behavior in the current frame image, and there is no need to execute the subsequent steps.

[0055] In a preferred embodiment, the above-mentioned preset relationship may include the distance relationship and angle relationship between the hand key point and the chest key point. The specific triggering method for executing the smoking behavior determination provided in this embodiment is that when the distance between the hand key point and the chest key point is less than the preset distance, and the angle formed by the hand key point and the chest key point is less than the preset angle, it is determined that the current frame image satisfies the preset relationship. Among them, the distance between the current hand key point and the chest key point represents the straight-line distance of the line connecting the hand key point and the chest key point in the current frame image; the angle formed by the hand key point and the chest key point represents the angle between the line connecting the hand key point and the chest key point and the vertical line.

[0056] Specifically, through the analysis of the user's smoking behavior, it can be known that when the user is ready to smoke, he will always hold the cigarette close to his mouth. During this process, the distance between the key point of the hand and the key point of the chest will gradually decrease, and the angle formed between the two key points will also gradually decrease. In this embodiment, the preset distance and the preset angle are set as the conditions for triggering the detection of smoking behavior. If the conditions are met, the current user may start smoking, and each frame image is monitored starting from the current frame image. Similarly, during the monitoring process, if the above preset distance and preset angle are not met, the detection operation of the current frame image is exited.

[0057] Among them, the above-mentioned preset distance can be represented by pixel points in the image, such as 100 pixels, 150 pixels or 200 pixels, etc., preferably 200 pixels; the above-mentioned preset angle can be 40°, 60° or 80°, etc., preferably 60°, and the determination of the specific preset distance and preset angle is not limited here.

[0058] After detecting that the hand key points and chest key points in the current frame image satisfy the preset relationship, this embodiment can determine several frames of continuous exhalation images from the continuous images after the current frame image according to the breathing characteristics. The reason for this is that since the video frame data received by the server is massive and the user is continuously inhaling and exhaling, since the user behavior cannot be determined in the inhalation process in this embodiment, the frame image corresponding to the inhalation process can be further analyzed, so as to reduce the amount of data analysis, reduce the computing pressure of the server, and release the server resources.

[0059] In this embodiment, the breathing characteristics include the exhalation cycle and the direction of change of the key points of the chest. The exhalation cycle can be understood as the exhalation cycle corresponding to the user when smoking generally exceeds the normal exhalation cycle. This information is very important. This embodiment can determine several frames of continuous exhalation images based on the judgment of the exhalation cycle and the direction of change of the key points of the chest.

[0060] Specifically, the above-mentioned determination of several frames of continuous exhalation images from the continuous images after the current frame image according to the respiratory characteristics can be achieved in the following way: when it is detected that the exhalation cycle exceeds the preset cycle, and the change direction of the chest key points corresponding to the continuous images after the current frame image is the preset direction, the continuous images corresponding to the exhalation cycle are determined to be several frames of continuous exhalation images.

[0061] Specifically, this embodiment analyzes images exceeding a preset period one by one, that is, analyzes the change direction of the chest key points in the current frame image and the next frame image. If it is detected that the change direction of the chest key points is moving outward, it indicates that the current continuous frame images may be inhalation behavior. If it is detected that the change direction of the chest key points is moving inward, it indicates that the current continuous frame images may be exhalation behavior. Therefore, in the current breathing process, all frame images corresponding to the key points moving inward continuously are determined as several frames of continuous exhalation images.

[0062] S320: Determine a hand angle change sequence and a chest angle change sequence according to each chest key point and each hand key point respectively.

[0063] The hand angle change sequence is obtained by determining the angle change related to the hand key points in each exhalation image; the chest angle change sequence is obtained by determining the angle change related to the chest key points in each exhalation image.

[0064] Specifically, the method of calculating the angle change sequence of key points in consecutive frames can be implemented by the following formula:

[0065] (1)

[0066] In the above formula (1), Indicates the coordinates of the key points in the current frame image; Indicates the coordinates corresponding to the same key point in the next frame image.

[0067] In this embodiment, the angle change sequences of the hand key points and the chest key points can be obtained by the following method. In order to distinguish the angle change sequences corresponding to the two key points, in this embodiment, represents the hand angle change sequence, Indicates the sequence of chest angle changes, etc.

[0068] S330, performing frequency domain conversion on the hand angle change sequence and the chest angle change sequence respectively based on Fourier transform to obtain a hand angle frequency domain sequence and a chest angle frequency domain sequence.

[0069] In this embodiment, the method of performing frequency domain conversion on the hand angle change sequence and the chest angle change sequence based on Fourier transform may be:

[0070] (2)

[0071] (3)

[0072] The above formula (2) represents the frequency domain sequence of hand angles, Represents the frequency domain series of chest angles.

[0073] In this embodiment, the purpose of converting the hand angle change sequence and the chest angle change sequence into the frequency domain is that the frequency domain analysis can reveal the periodicity and repetitive patterns hidden in the data by converting the time series data into frequency components. For example, when monitoring the user's breathing or smoking behavior, this periodicity may be manifested as a specific breathing frequency or the frequency of the smoking action. In the frequency domain, these periodic activities can be more clearly identified and analyzed; further, for complex signals or signals containing multiple frequency components, frequency domain analysis can help us identify and focus on the most important frequency components and ignore those irrelevant or interfering components. This method can simplify the data analysis process, making it more direct and efficient to identify and interpret the main features in the data; and in the frequency domain, various filtering techniques can be easily applied to enhance specific frequency components in the signal while suppressing noise. For example, if the frequency range of breathing or hand movements is known, a bandpass filter can be designed to retain only these specific frequencies, thereby improving the signal-to-noise ratio; in the frequency domain, various statistical methods can be used to analyze the correlation of frequency components, which is very useful for pattern recognition and behavior prediction. For example, by calculating the covariance or correlation coefficients between different frequency components, statistically significant behavioral patterns can be identified, such as inconsistencies in hand and chest movements during smoking behavior.

[0074] S340, calculating the covariance of the hand angle frequency domain sequence and the chest angle frequency domain sequence, and obtaining sequence features according to the covariance.

[0075] In the current step, calculate and The covariance of the hand key points and the chest key points is used to evaluate the correlation between the hand and the chest in the frequency domain. In the present embodiment, the purpose of using covariance to represent the direction of change corresponding to the hand key points and the direction of change corresponding to the chest key points is that covariance is an important statistical measure, which is used to measure the degree of linear relationship between two variables. Specifically, it reflects how the value of one variable changes when the value of another variable increases. When analyzing the movement direction of the hand key points and the chest key points, the use of covariance can help us determine whether the two movements have the same or opposite change trends, that is, whether they show coordinated or opposite movement patterns in time. When exhaling, accompanied by the chest contraction and the hands moving outward, their movement directions are opposite, so the calculated covariance value can be close to -1 to indicate opposite movement characteristics.

[0076] In this embodiment, the sequence features include an inconsistent feature, which indicates that in several consecutive frames of exhalation images, the movement direction of the hand key point and the movement direction of the chest key point are completely inconsistent, that is, opposite; correspondingly, the consistent feature indicates that in several consecutive frames of exhalation images, the movement direction of the hand key point and the movement direction of the chest key point are completely consistent.

[0077] S350: Determine whether the covariance is less than a preset threshold.

[0078] The closer the calculated covariance is to -1, the more it indicates that the sequence features are completely opposite, that is, the motion features are inconsistent. Then, in this embodiment, after analyzing that in several consecutive frames of exhalation images, the moving direction of the hand key point is completely opposite to the moving direction of the chest key point, then the preset threshold in this embodiment can be set to -0.7, -0.8 or -0.9, preferably -0.8, and the specific preset threshold setting is not limited here.

[0079] When the covariance is less than the preset threshold, that is, yes, step S351 can be executed; when the covariance is not less than the preset threshold, that is, no, step S352 is executed.

[0080] S351: If the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence have inconsistent features, it is determined that the user has smoking behavior.

[0081] S352: If the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence do not have an inconsistent feature, it is determined that the user does not have a smoking behavior.

[0082] A preferred embodiment, after determining that the user has a smoking behavior, further includes: determining at least one target hardware information to be linked; calling the target hardware device corresponding to the at least one target hardware information to provide a corresponding linkage service based on the at least one target hardware device. The purpose of this is to remind the user not to smoke through the linkage service to avoid dangerous accidents.

[0083] In an application scenario, taking the detection of the driver's smoking behavior while driving a vehicle as an example, at least one target hardware information to be linked may be the voice module information and the window information in the cockpit. Thus, after determining that the user has smoking behavior, the window can be opened by the user based on the window information to ensure air flow. At the same time, a voice reminder can be provided to the user based on the voice module information to remind the user of abnormal behavior and to pay attention to safe driving, thereby reducing driving accidents.

[0084] Another application example, the linkage service of this embodiment can further track the location of cigarettes when a burning cigarette is detected, and detect the cigarette butt close to the car window. If a cigarette butt is detected close to the car window and suddenly disappears, it may mean that the burning cigarette has been thrown out of the window. When the act of throwing a burning cigarette out of the window is detected, the fire protection system can be immediately triggered to prevent safety problems such as fires, so as to maintain road traffic order and environmental sanitation.

[0085] The smoking behavior determination method provided in this embodiment determines the inconsistency feature of the sequence features by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence. In essence, during the user's exhalation process, the movement directions of the chest key points and the hand key points are completely opposite, so that the user's smoking behavior can be determined. The implementation process of this solution does not need to rely on the recognition of the physical features of the cigarette in a burning state, and can improve the accuracy of the determined smoking behavior.

[0086] Figure 4 is a structural diagram of a smoking behavior determination device provided in an embodiment of the present application, and the device is suitable for executing the smoking behavior determination method provided in an embodiment of the present application. Figure 4 As shown, the device may specifically include: an image acquisition module 410 , a sequence determination module 420 and a behavior determination module 430 .

[0087] The image acquisition module 410 is used to acquire a plurality of frames of continuous exhalation images, each of which includes a chest key point and a hand key point.

[0088] The sequence determination module 420 is used to determine a hand angle change sequence and a chest angle change sequence respectively according to each of the chest key points and each of the hand key points, wherein the hand angle change sequence is obtained by determining the angle change associated with the hand key point in each of the exhalation images; and the chest angle change sequence is obtained by determining the angle change associated with the chest key point in each of the exhalation images.

[0089] The behavior determination module 430 is used to calculate the sequence characteristics of the hand angle change sequence and the chest angle change sequence, and determine whether the user has smoking behavior based on the sequence characteristics, wherein the sequence characteristics are obtained by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence.

[0090] The smoking behavior determination device provided in this embodiment first obtains several frames of continuous exhalation images, each of which includes chest key points and hand key points; then determines the hand angle change sequence and chest angle change sequence respectively according to each chest key point and each hand key point; finally calculates the sequence features of the hand angle change sequence and the chest angle change sequence to determine whether the user has smoking behavior according to the sequence features. This embodiment determines the sequence features by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence. In essence, it analyzes the movement direction of the chest key points and the hand key points during the user's exhalation process, so as to determine the user's smoking behavior. The implementation process of this solution does not need to rely on the recognition of the physical features of the cigarette in a burning state, and can improve the accuracy of the determined smoking behavior.

[0091] In one embodiment, the image acquisition module 410 includes a key point acquisition unit and an image determination unit.

[0092] The key point acquisition unit is used to acquire the hand key points and the chest key points in each frame of at least one frame of image in the video stream.

[0093] The image determination unit is used to determine the plurality of frames of continuous exhalation images from the continuous images after the current frame image according to the breathing characteristics when the hand key points and the chest key points in the current frame image satisfy a preset relationship.

[0094] In one embodiment, the breathing characteristics include an exhalation cycle and a change direction of the chest key points.

[0095] The image determination unit is specifically used to determine that the continuous images corresponding to the exhalation cycle are the several frames of continuous exhalation images when it is detected that the exhalation cycle exceeds a preset cycle and the change direction of the chest key point corresponding to the continuous images after the current frame image is a preset direction.

[0096] In one embodiment, the image determination unit is specifically used to determine that the current frame image satisfies the preset relationship when the distance between the hand key point and the chest key point is less than a preset distance, and the angle formed by the hand key point and the chest key point is less than a preset angle.

[0097] In one embodiment, the behavior determination module 430 includes a frequency domain conversion unit and a covariance calculation unit.

[0098] Among them, the frequency domain conversion unit is used to perform frequency domain conversion on the hand angle change sequence and the chest angle change sequence respectively based on Fourier transform to obtain a hand angle frequency domain sequence and a chest angle frequency domain sequence.

[0099] A covariance calculation unit is used to calculate the covariance of the hand angle frequency domain sequence and the chest angle frequency domain sequence, and obtain the sequence feature according to the covariance.

[0100] In one embodiment, the sequence feature includes an inconsistent feature. The behavior determination module 430 also includes a threshold determination unit and a behavior determination unit.

[0101] The threshold determination unit is used to determine whether the covariance is less than a preset threshold.

[0102] The behavior determination unit is used to determine that the user has the smoking behavior when the covariance is less than the preset threshold and the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence have the inconsistency feature; and determine that the user does not have the smoking behavior when the covariance is not less than the preset threshold and the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence do not have the inconsistency feature.

[0103] In one embodiment, the device further includes an information acquisition module and a linkage service module.

[0104] The information acquisition module is used to determine at least one target hardware information to be linked.

[0105] The linkage service module is used to call at least one target hardware device corresponding to the target hardware information to provide a corresponding linkage service based on at least one target hardware device.

[0106] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0107] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior determination method described in any embodiment of the present application.

[0108] An embodiment of the present application further provides a computer-readable medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the smoking behavior determination method described in any embodiment of the present application when executed.

[0109] Reference below Figure 5 , Figure 5 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and shows a schematic diagram of the structure of a computer system 500 suitable for implementing the electronic device in an embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0110] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0111] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0112] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0113] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, etc., or any suitable combination of the above.

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules and / or units involved in the embodiments described in the present application may be implemented by software or hardware. The modules and / or units described may also be arranged in a processor, for example, it may be described as: a processor includes an image acquisition module, a sequence determination module, and a behavior determination module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0116] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: acquiring a plurality of frames of continuous exhalation images, each of which includes a chest key point and a hand key point; determining a hand angle change sequence and a chest angle change sequence according to each chest key point and each hand key point, respectively, the hand angle change sequence is obtained by determining the angle change associated with the hand key point in each exhalation image; the chest angle change sequence is obtained by determining the angle change associated with the chest key point in each exhalation image; calculating the sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether the user has smoking behavior according to the sequence features, the sequence features are obtained by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence.

[0117] According to the technical solution of this embodiment, the sequence features are determined by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence. In essence, the movement direction of the chest key points and the hand key points is analyzed during the user's exhalation process, so that the user's smoking behavior can be determined. The implementation process of this solution does not need to rely on the recognition of the physical characteristics of the cigarette in the burning state, which can improve the accuracy of the determined smoking behavior.

[0118] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for determining smoking behavior, characterized in that: include: Acquire a plurality of frames of continuous exhalation images, each of which includes a chest key point and a hand key point; Determine a hand angle change sequence and a chest angle change sequence respectively according to each of the chest key points and each of the hand key points, wherein the hand angle change sequence is obtained by determining the angle change associated with the hand key point in each of the exhalation images; and the chest angle change sequence is obtained by determining the angle change associated with the chest key point in each of the exhalation images; Calculating sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether the user has a smoking behavior according to the sequence features, wherein the sequence features are obtained by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence; Wherein, determining the hand angle change sequence and the chest angle change sequence respectively according to each of the chest key points and each of the hand key points comprises: ； in, Indicates the coordinates of the key points in the current frame image; represents the coordinates corresponding to the same key point in the next frame image; the angle change sequences of the hand key point and the chest key point are obtained in the above manner; The sequence feature includes an inconsistency feature; the calculating the sequence features of the hand angle change sequence and the chest angle change sequence, and determining whether the user has a smoking behavior according to the sequence features, includes: Based on Fourier transform, the hand angle change sequence and the chest angle change sequence are respectively converted into frequency domains to obtain a hand angle frequency domain sequence and a chest angle frequency domain sequence; Calculating the covariance of the hand angle frequency domain sequence and the chest angle frequency domain sequence, and obtaining the sequence feature according to the covariance; Determining whether the covariance is less than a preset threshold; When the covariance is less than the preset threshold, and the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence have the inconsistency feature, it is determined that the user has the smoking behavior; When the covariance is not less than the preset threshold, and the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence do not have the inconsistent feature, it is determined that the user does not have the smoking behavior.

2. The method for determining smoking behavior according to claim 1, characterized in that: The step of acquiring a plurality of frames of continuous exhalation images comprises: For at least one frame of image in the video stream, obtaining the hand key points and the chest key points in each frame of image; When the hand key points and the chest key points in the current frame image satisfy a preset relationship, the plurality of frames of continuous exhalation images are determined from continuous images after the current frame image according to the breathing characteristics.

3. The method for determining smoking behavior according to claim 2, characterized in that: The breathing characteristics include the exhalation cycle and the changing direction of the chest key points; The step of determining the plurality of frames of continuous exhalation images from the continuous images after the current frame image according to the respiratory characteristics comprises: When it is detected that the exhalation cycle exceeds a preset cycle and the change direction of the chest key point corresponding to the continuous images after the current frame image is a preset direction, it is determined that the continuous images corresponding to the exhalation cycle are the several frames of continuous exhalation images.

4. The method for determining smoking behavior according to claim 2, characterized in that: The hand key point and the chest key point in the current frame image satisfy a preset relationship, including: When the distance between the hand key point and the chest key point is less than a preset distance, and the angle formed by the hand key point and the chest key point is less than a preset angle, it is determined that the current frame image satisfies the preset relationship.

5. The method for determining smoking behavior according to claim 1, characterized in that: After determining that the user has smoking behavior, it also includes: Determine at least one target hardware information to be linked; At least one target hardware device corresponding to the target hardware information is called to provide a corresponding linkage service based on the at least one target hardware device.

6. A smoking behavior determination device, characterized in that: include: An image acquisition module, used to acquire a plurality of frames of continuous exhalation images, each of which includes a chest key point and a hand key point; A sequence determination module, used to determine a hand angle change sequence and a chest angle change sequence respectively according to each of the chest key points and each of the hand key points, wherein the hand angle change sequence is obtained by determining the angle change associated with the hand key point in each of the exhalation images; and the chest angle change sequence is obtained by determining the angle change associated with the chest key point in each of the exhalation images; a behavior determination module, used to calculate sequence features of the hand angle change sequence and the chest angle change sequence, and determine whether the user has a smoking behavior according to the sequence features, wherein the sequence features are obtained by analyzing the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence; Wherein, determining the hand angle change sequence and the chest angle change sequence respectively according to each of the chest key points and each of the hand key points comprises: ； in, Indicates the coordinates of the key points in the current frame image; represents the coordinates corresponding to the same key point in the next frame image; the angle change sequences of the hand key point and the chest key point are obtained in the above manner; Wherein, the sequence feature includes an inconsistent feature; the behavior determination module includes a frequency domain conversion unit, a covariance calculation unit, a threshold judgment unit and a behavior determination unit; The frequency domain conversion unit is used to perform frequency domain conversion on the hand angle change sequence and the chest angle change sequence respectively based on Fourier transform to obtain a hand angle frequency domain sequence and a chest angle frequency domain sequence; The covariance calculation unit is used to calculate the covariance of the hand angle frequency domain sequence and the chest angle frequency domain sequence, and obtain the sequence feature according to the covariance; The threshold determination unit is used to determine whether the covariance is less than a preset threshold; The behavior determination unit is used to determine that the user has the smoking behavior when the covariance is less than the preset threshold and the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence have the inconsistency feature; and to determine that the user does not have the smoking behavior when the covariance is not less than the preset threshold and the change direction corresponding to the hand angle change sequence and the change direction corresponding to the chest angle change sequence do not have the inconsistency feature.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the smoking behavior determination method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the smoking behavior determination method according to any one of claims 1 to 5 is implemented.

Images