Domestic gas stove repeated ignition alarm method based on attitude estimation

Abstract

The invention discloses a household gas stove repeated ignition alarm method and device based on attitude estimation, and the method comprises the steps that a video monitoring device collects image data of a stove user in real time through a camera, and transmits the monitored video data to a local server in real time through a network cable in a video stream mode; the local server processes each frame of image by adopting a VGG network and an openpose algorithm to obtain human body posture estimation data; carrying out normalization processing and weighting processing on the human body posture estimation data, constructing a graph model, and carrying out space-time graph convolution; and the behavior type is identified, and if the behavior type is abnormal, the sound-light alarm circuit is driven to carry out repeated ignition alarm. According to the repeated ignition alarm method for the household gas stove, the body parts are associated with individuals in the image by using the partial affinity field according to the positions and directions of the limbs in the coded image, and meanwhile, key point detection and association are performed, so that accurate alarm for relatively fast repeated ignition operation or close range observation of the stove eye during ignition is realized.

Description

Technical Field

[0001] The invention relates to the technical field of household gas stove safety, and in particular to a household gas stove repeated ignition alarm method and device based on posture estimation. Background Art

[0002] If the gas is ignited repeatedly and quickly or the stove eye is observed closely during ignition, there is a risk that the alarm may not be able to sound the alarm in time. Summary of the invention

[0003] The present invention proposes a method and device for alarming repeated ignition of a household gas stove based on posture estimation, so as to solve the problem that traditional household gas alarms cannot give timely alarms for behaviors such as repeated ignition operations that are fast or close observation of the stove eye during ignition.

[0004] The present invention provides the following technical solutions:

[0005] In a first aspect, this specification provides a method for repeatedly igniting a household gas stove based on posture estimation, comprising:

[0006] The video monitoring device collects image data of the stove user in real time through a camera to obtain monitoring video data, and sends the monitoring video data in real time to a local server in the form of a video stream through a network cable;

[0007] The local server processes each frame of the image according to the monitoring video data using a VGG network and an openpose algorithm to obtain human posture estimation data;

[0008] Normalizing and weighting the human body posture estimation data, constructing a graph model, and obtaining a weighted skeleton graph model;

[0009] Performing spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data;

[0010] According to the convolution operation result data, the behavior category is identified and it is determined whether the behavior category is abnormal. If so, the sound and light alarm circuit is driven to perform repeated ignition alarm.

[0011] In a second aspect, the present invention provides a household gas stove repeated ignition alarm device based on posture estimation, comprising:

[0012] An image data acquisition module is used to control the video monitoring device to collect image data of the stove user in real time through a camera to obtain monitoring video data, and send the monitoring video data in real time to a local server in the form of a video stream through a network cable;

[0013] A human posture estimation module is used to control the local server to process each frame of the image according to the monitoring video data using the VGG network and the openpose algorithm to obtain human posture estimation data;

[0014] A skeleton graph model construction module is used to perform normalization and weighting processing on the human body posture estimation data, construct a graph model, and obtain a weighted skeleton graph model;

[0015] A spatiotemporal graph convolution module, used to perform spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data;

[0016] The behavior category recognition module is used to identify the behavior category according to the convolution operation result data, and determine whether the behavior category is abnormal. If so, drive the sound and light alarm circuit to perform repeated ignition alarm.

[0017] The household gas stove repeated ignition alarm method and device based on posture estimation provided by the embodiment of the present invention encodes the position and direction of the limbs in the image, uses partial affinity fields to associate body parts with individuals in the image, and simultaneously performs key point detection and association, so as to obtain high-quality results at a small computing cost, reduce time complexity, and only use a 3*3 convolution kernel. While increasing the number of nonlinear transformations, the network depth is improved, the number of parameters is reduced, and the feature learning ability is stronger. The effect of the neural network is improved to a certain extent, thereby realizing accurate alarm for repeated ignition operations that are fast or close observation of the stove eye during ignition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a system block diagram corresponding to the method for repeatedly igniting an alarm for a household gas stove based on posture estimation in an embodiment of the present invention;

[0019] Figure 2 The present invention is a flowchart of a method for repeatedly igniting a household gas stove based on posture estimation in an embodiment of the present invention.

[0020] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0022] As described herein, the term “including” and various variations thereof may be understood as open-ended terms meaning “including but not limited to,” and the term “one embodiment” may be understood as “at least one embodiment.”

[0023] The inventors found that the traditional household gas alarm is placed within a radius of 1.5 meters from the gas source and about 0.3 meters from the ceiling. When the ignition operation is repeated quickly or the stove eye is observed closely during the ignition, there is a hidden danger that the alarm cannot give an alarm in time. In view of this, in the embodiment of the present invention, by encoding the position and direction of the limbs in the image, the body parts are associated with the individuals in the image using the partial affinity field, and key point detection and association are performed at the same time. When there are multiple ignition operations or there is a tendency to lean over and get close to the stove eye during ignition, the alarm emits a high-decibel prompt sound, which realizes accurate alarm for repeated ignition operations that are fast or the stove eye is observed closely during ignition.

[0024] Embodiment 1

[0025] like Figure 1 As shown, an embodiment of the present invention provides a method for repeatedly igniting a household gas stove based on posture estimation. The method is executed by a processor and includes the following steps:

[0026] Step 1: The video monitoring device collects image data of the stove user in real time through a camera to obtain monitoring video data, and sends the monitoring video data in real time to a local server in the form of a video stream through a network cable.

[0027] Step 2: The local server processes each frame of the image according to the monitoring video data using the VGG network and the openpose algorithm to obtain human posture estimation data.

[0028] In specific implementation, the local server uses the VGG network to extract the features of each frame image in the surveillance video data, uses the openpose algorithm to detect and associate key points of the features of each frame image, uses a 3*3 convolution kernel to perform deep neural network iteration, and fuses the prediction results of the previous stage with the original image features as the input data of the next stage to determine the coordinates of 25 key points of the human body and obtain human posture estimation data.

[0029] Step three: normalize and weight the human body posture estimation data, construct a graph model, and obtain a weighted skeleton graph model.

[0030] In a specific implementation, according to the human body posture estimation data, the key point coordinates are used as an input matrix, and the input matrix is ​​normalized to obtain normalized data; according to the normalized data, the composition rules are determined, a skeleton graph model is constructed, and the skeleton graph model is weighted by edge importance to obtain a weighted skeleton graph model.

[0031] Step 4: Perform spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data.

[0032] In a specific implementation, a multi-stage spatiotemporal graph convolution is performed on the weighted skeleton graph model to increase the joint dimension, reduce the key frame dimension, decompose the action, and obtain the convolution operation result data; and the multi-stage spatiotemporal graph convolution includes 1 graph convolution and 3 time convolutions.

[0033] Step 5: Identify the behavior category based on the convolution operation result data, and determine whether the behavior category is abnormal. If so, drive the sound and light alarm circuit to perform repeated ignition alarm.

[0034] In the specific implementation, the convolution operation result data is average pooled and fully connected, and the behavior category is identified through the Softmax classifier. According to the preset abnormal behavior judgment conditions, it is judged whether the behavior category is abnormal. If so, the sound and light alarm circuit is driven to repeatedly ignite the alarm; and the dangerous abnormal behavior includes multiple ignition operations and bending over to check the stove eye.

[0035] Embodiment 2

[0036] like Figure 1 As shown, the embodiment of the present invention provides a household gas stove repeated ignition and dangerous behavior alarm device based on posture estimation, comprising an image acquisition module, an extraction and matching module and an indication warning module, for implementing the following Figure 2 The household gas stove repeated ignition alarm method based on posture estimation shown in the figure has the following specific steps:

[0037] Step 1: Use the openpose algorithm to obtain the key points of the body and foot bones in real time. This method encodes the position and orientation of the limbs in the image, uses partial affinity fields to associate body parts with individuals in the image, improves accuracy, and performs key point detection and association at the same time, obtaining high-quality results at a small computational cost and reducing time complexity. Finally, 25 key points were obtained.

[0038] Step 2: Collect image data of stove users in real time through the camera. Obtain surveillance video in the form of video stream, estimate human posture for each frame in the video, use VGG network to extract image features, and iterate the extracted features through deep neural network. The prediction results of the previous stage are integrated with the original image features as the input of the next stage, and finally the coordinates of 25 key points of the human body are obtained. The network at this stage only uses 3*3 convolution kernels, which increases the number of nonlinear transformations while improving the network depth, reducing the number of parameters, and having a stronger learning ability for features, which improves the effect of neural network to a certain extent.

[0039] Step 3. Use the obtained posture data to construct a graph model, perform spatiotemporal graph convolution on the graph model, identify the behavior category, and determine whether an abnormality occurs. First, take the key point coordinates as the input matrix and normalize the input matrix. Normalization can improve the training speed of the model. Secondly, use the normalized data to redefine the composition rules, construct a skeleton graph model, and weight the edge importance of the constructed skeleton graph model to distinguish the importance of different trunks. Then, perform spatiotemporal graph convolution on the skeleton graph at different stages. The convolution operation increases the joint dimension, reduces the key frame dimension, and decomposes the action. Each stage contains 1 graph convolution and 3 time convolutions. Finally, the result of the convolution operation is average pooled and fully connected, and the corresponding action classification is obtained through the Softmax classifier, and it is judged whether it is an abnormal behavior based on the definition of abnormal behavior.

[0040] Step 4: When there are dangerous abnormal behaviors such as multiple ignition operations and bending over to check the stove eye, drive the sound and light alarm circuit.

[0041] In one embodiment, a real-time monitoring and early warning system based on a posture estimation algorithm based on deep learning is implemented by the following equipment:

[0042] Video surveillance device: deploy high-definition surveillance cameras to obtain surveillance videos in real time, connect them to the local server via network cables, and send the videos to the server in real time in the form of video streams;

[0043] Server device: The training device function of the human posture estimation model can be centralized in the server, and the algorithm is used to estimate the posture of the image features, obtain the coordinates of 25 key points of the human body, and determine whether the identified behavior contains dangerous actions. If it does, the alarm mechanism is triggered;

[0044] Sound and light alarm device: The alarm device uses a buzzer that is larger and louder than ordinary buzzers. It is very practical for use in the kitchen and can effectively arouse people's vigilance. When there are multiple ignition operations or other dangerous actions, the alarm sounds and the warning light flashes.

[0045] The above embodiment encodes the position and direction of the limbs in the image, uses partial affinity fields to associate body parts with individuals in the image, and simultaneously performs key point detection and association, so as to obtain high-quality results at a small computational cost, reduce time complexity, and use only a 3*3 convolution kernel. While increasing the number of nonlinear transformations, the network depth is improved, the number of parameters is reduced, and the feature learning ability is stronger. To a certain extent, the effect of the neural network is improved, thereby achieving accurate alarm for repeated ignition operations that are fast or close observation of the stove eye during ignition, thereby solving the problem that traditional household gas alarms cannot provide timely alarms for repeated ignition operations that are fast or close observation of the stove eye during ignition.

[0046] Embodiment 3

[0047] Based on the same technical concept, an embodiment of the present invention also provides a household gas stove repeated ignition alarm device based on posture estimation. Since the principle of solving the problem by the above device is similar to the household gas stove repeated ignition alarm method based on posture estimation, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0048] The embodiment of the present invention provides a household gas stove repeated ignition alarm device based on posture estimation, comprising:

[0049] An image data acquisition module is used to control the video monitoring device to collect image data of the stove user in real time through a camera to obtain monitoring video data, and send the monitoring video data in real time to a local server in the form of a video stream through a network cable;

[0050] A human posture estimation module is used to control the local server to process each frame of the image according to the monitoring video data using the VGG network and the openpose algorithm to obtain human posture estimation data;

[0051] A skeleton graph model construction module is used to perform normalization and weighting processing on the human body posture estimation data, construct a graph model, and obtain a weighted skeleton graph model;

[0052] A spatiotemporal graph convolution module, used to perform spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data;

[0053] The behavior category recognition module is used to identify the behavior category according to the convolution operation result data, and determine whether the behavior category is abnormal. If so, drive the sound and light alarm circuit to perform repeated ignition alarm.

[0054] In summary, the embodiment of the present invention applies an artificial intelligence algorithm to household products, by encoding the position and direction of the limbs in the image, using partial affinity fields to associate body parts with individuals in the image, and simultaneously performing key point detection and association, high-quality results are obtained at a small portion of the computing cost, reducing time complexity, and using only a 3*3 convolution kernel. While increasing the number of nonlinear transformations, the network depth is improved, the number of parameters is reduced, and the learning ability for features is stronger, which improves the effect of the neural network to a certain extent. When there are multiple ignition operations or there is a tendency to lean over and get close to the stove eye when igniting, the alarm emits a high-decibel prompt sound, which realizes accurate alarm for repeated ignition operations that are fast or close observation of the stove eye during ignition, and alarms for dangerous behaviors in the process of using the gas stove, reminding users to pay attention to safety, protecting life and property safety to a certain extent, and solving the problem that traditional household gas alarms cannot promptly alarm for repeated ignition operations that are fast or close observation of the stove eye during ignition.

[0055] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0056] Those skilled in the art know that, in addition to implementing the system and its various subsystems, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various subsystems, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, etc. by performing logic programming on the method steps. Therefore, the system and its various subsystems, modules, and units provided by the present invention can be regarded as structures within hardware components or as software modules for implementing the method.

[0057] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for repeatedly igniting a household gas stove based on attitude estimation, characterized in that: include: The video monitoring device collects image data of the stove user in real time through a camera to obtain monitoring video data, and sends the monitoring video data in real time to a local server in the form of a video stream through a network cable; The local server processes each frame of the image according to the monitoring video data using a VGG network and an openpose algorithm to obtain human posture estimation data; Normalizing and weighting the human body posture estimation data, constructing a graph model, and obtaining a weighted skeleton graph model; Performing spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data; According to the convolution operation result data, the behavior category is identified, and it is determined whether the behavior category is abnormal. If so, the sound and light alarm circuit is driven to perform repeated ignition alarm.

2. The method according to claim 1, characterized in that The local server processes each frame of the image according to the monitoring video data using the VGG network and the openpose algorithm to obtain human posture estimation data, specifically including: The local server uses the VGG network to extract the features of each frame of the surveillance video data, uses the openpose algorithm to detect and associate key points of the features of each frame of the image, uses a 3*3 convolution kernel to iterate the deep neural network, and fuses the prediction results of the previous stage with the original image features as the input data of the next stage to determine the coordinates of 25 key points of the human body and obtain human posture estimation data.

3. The method according to claim 1, characterized in that The human body posture estimation data is normalized and weighted to construct a graph model to obtain a weighted skeleton graph model, which specifically includes: According to the human body posture estimation data, the key point coordinates are used as an input matrix, and the input matrix is ​​normalized to obtain normalized data; According to the normalized data, a composition rule is determined, a skeleton graph model is constructed, and edge importance weighting processing is performed on the skeleton graph model to obtain a weighted skeleton graph model.

4. The method according to claim 1, characterized in that Performing spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data, specifically including: Performing multi-stage spatiotemporal graph convolution on the weighted skeleton graph model, increasing joint dimensions, reducing key frame dimensions, decomposing actions, and obtaining convolution operation result data; and The multi-stage spatiotemporal graph convolution includes 1 graph convolution and 3 temporal convolutions.

5. The method according to claim 1, characterized in that According to the convolution operation result data, the behavior category is identified, and it is determined whether the behavior category is abnormal. If so, the sound and light alarm circuit is driven to perform repeated ignition alarm, specifically including: The convolution operation result data is average pooled and fully connected, and the behavior category is identified through a Softmax classifier. According to a preset abnormal behavior judgment condition, it is determined whether the behavior category is abnormal. If so, the sound and light alarm circuit is driven to repeatedly ignite the alarm; and The dangerous abnormal behaviors include multiple ignition operations and bending over to check the stove eye.

6. A device for executing the method for repeatedly igniting a household gas stove based on posture estimation as claimed in any one of claims 1 to 5, characterized in that: include: An image data acquisition module is used to control the video monitoring device to collect image data of the stove user in real time through a camera to obtain monitoring video data, and send the monitoring video data in real time to a local server in the form of a video stream through a network cable; A human posture estimation module is used to control the local server to process each frame of the image according to the monitoring video data using the VGG network and the openpose algorithm to obtain human posture estimation data; A skeleton graph model construction module is used to perform normalization and weighting processing on the human body posture estimation data, construct a graph model, and obtain a weighted skeleton graph model; A spatiotemporal graph convolution module, used to perform spatiotemporal graph convolution on the weighted skeleton graph model to obtain convolution operation result data; The behavior category recognition module is used to identify the behavior category according to the convolution operation result data, and determine whether the behavior category is abnormal. If so, drive the sound and light alarm circuit to perform repeated ignition alarm.

7. The device according to claim 6, characterized in that The human posture estimation module is specifically used to control the local server to use the VGG network to extract the features of each frame image in the monitoring video data, use the openpose algorithm to detect and associate key points of the features of each frame image, use a 3*3 convolution kernel to perform deep neural network iteration, and fuse the prediction results of the previous stage with the original image features as the input data of the next stage, determine the coordinates of 25 key points of the human body, and obtain human posture estimation data.

8. The device according to claim 6, characterized in that The skeleton graph model construction module is specifically used to use the key point coordinates as the input matrix according to the human body posture estimation data, normalize the input matrix to obtain normalized data, determine the composition rules according to the normalized data, construct the skeleton graph model, and perform edge importance weighting processing on the skeleton graph model to obtain a weighted skeleton graph model.

9. The device according to claim 6, characterized in that The spatiotemporal graph convolution module is specifically used to perform multi-stage spatiotemporal graph convolution on the weighted skeleton graph model, increase joint dimensions, reduce key frame dimensions, decompose actions, and obtain convolution operation result data; and The multi-stage spatiotemporal graph convolution includes 1 graph convolution stage and 3 temporal convolution stages.

10. The device according to claim 6, characterized in that The behavior category recognition module is specifically used to perform average pooling and full connection on the convolution operation result data, identify the behavior category through the Softmax classifier, and judge whether the behavior category is abnormal according to the preset abnormal behavior judgment condition. If so, drive the sound and light alarm circuit to perform repeated ignition alarm; as well as The dangerous abnormal behaviors include multiple ignition operations and bending over to check the stove eye.

Images