Range hood control method and device based on oil pouring action recognition and electronic equipment

Abstract

The invention discloses a range hood control method and device based on oil pouring action recognition and electronic equipment, and relates to the technical field of household appliance control, and the range hood control method based on oil pouring action recognition comprises the steps that a target image sequence which is collected by a camera arranged on a range hood and comprises a kitchen range area is obtained; performing preprocessing operation on each image frame in the target image sequence to obtain a to-be-recognized image sequence, and inputting the to-be-recognized image sequence into the action recognition model to obtain an action recognition result; and under the condition that the action recognition result indicates that the action of the user is the oil pouring action, the range hood is controlled according to the current operation state of the range hood. The range hood control method based on oil pouring action recognition is used for recognizing the gesture action of the user, and after the oil pouring action of the user is recognized, the range hood is started or the air volume level of the range hood is increased in time, and excessive oil smoke in a room is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of household appliance control, and in particular to a range hood control method, device and electronic equipment based on oil pouring action recognition. Background Art

[0002] As a kitchen fume removal device, range hoods have become popular in thousands of households. During the cooking process, users can directly touch the corresponding buttons on the control panel to control the range hood's power on and off, air volume, and lighting.

[0003] In the related art, a large amount of oil smoke is generated when users cook. If the users forget to turn on the range hood or the air volume level of the range hood is low, the indoor area may be filled with oil smoke, which not only brings a bad user experience but also affects the user's health.

[0004] Based on this, there is an urgent need for a range hood control method that can recognize user actions. After recognizing the user's oil pouring action, the range hood air volume level is turned on / increased in time to avoid excessive oil smoke in the room. Summary of the invention

[0005] The purpose of this application is to provide a range hood control method, device and electronic device based on oil pouring action recognition, which is used to recognize the user's gesture actions, and after recognizing the user's oil pouring action, promptly turn on / increase the air volume level of the range hood to avoid excessive oil smoke in the room.

[0006] The present application provides a range hood control method based on oil pouring action recognition, comprising:

[0007] A target image sequence including a stove area captured by a camera arranged on the range hood is obtained; the target image sequence includes: a first preset number of image frames; a preprocessing operation is performed on each image frame in the target image sequence to obtain an image sequence to be identified, and the image sequence to be identified is input into an action recognition model to obtain an action recognition result; when the action recognition result indicates that the user's action is an oil pouring action, the range hood is controlled according to the current operating state of the range hood; wherein the preprocessing operation is used to identify and separate the image of the stove area from the image frame; and the action recognition model is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN.

[0008] Optionally, when the action recognition result indicates that the user's action is pouring oil, the range hood is controlled according to the current operating state of the range hood, including: when the range hood is in a stopped state, controlling the range hood to start and operate at a preset air volume level; or, when the range hood is in an operating state, increasing the air volume level of the range hood.

[0009] Optionally, before inputting the target image sequence into the action recognition model to obtain the action recognition result, the method also includes: acquiring multiple first image sequences including the oil pouring action; each first image sequence includes the first preset number of image frames, and there are a second preset number of repeated image frames between the two first image sequences collected for the same action; performing stove position segmentation on each image frame in the multiple first image sequences to obtain multiple second image sequences.

[0010] Optionally, performing stove position segmentation on each image frame in the multiple first image sequences to obtain multiple second image sequences includes: performing grayscale processing on the target image frame to obtain a grayscale image; and converting the grayscale image into a binary image through binary image transformation; performing image denoising processing on the binary image to obtain an image to be detected, and performing stove contour detection on the image to be detected to determine the stove position; performing image segmentation on the image to be detected based on the stove position to obtain a stove image frame containing only a stove area; wherein the target image frame is any image frame among the multiple image frames contained in the multiple first image sequences; and the stove image frame is an image frame obtained after performing stove position segmentation on the target image frame.

[0011] Optionally, after performing stove position segmentation on each image frame in the multiple first image sequences to obtain a plurality of second image sequences, the method further includes: adjusting each image frame in the multiple second image sequences to an image frame of a preset size, and performing normalization processing on each adjusted image frame based on the mean and variance of the Imagenet dataset to obtain a plurality of image sequences to be trained.

[0012] Optionally, after performing normalization processing on each adjusted image frame based on the mean and variance of the Imagenet data set to obtain multiple image sequences to be trained, the method further includes: using the multiple image sequences to be trained as training samples of the model to be trained to train the model to be trained to obtain the action recognition model; wherein, one image sequence to be trained is one training sample; the model to be trained is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN; the model to be trained includes: a target convolution kernel obtained by fusing a 3*3*3 convolution kernel and a 5*5*5 convolution kernel; the target convolution kernel is used to extract action features with smaller action amplitudes and action features with larger action amplitudes; the model to be trained uses a linear rectification function ReLu activation function for nonlinear transformation; and the model to be trained is optimized using cross entropy loss during the training process.

[0013] The present application also provides a range hood control device based on oil pouring action recognition, comprising:

[0014] An acquisition module is used to acquire a target image sequence including a stove area captured by a camera provided on the range hood; the target image sequence includes: a first preset number of image frames; an image processing module is used to perform a preprocessing operation on each image frame in the target image sequence to obtain a sequence of images to be identified; an action recognition module is used to input the sequence of images to be identified into an action recognition model to obtain an action recognition result; a control module is used to control the range hood according to the current operating state of the range hood when the action recognition result indicates that the user's action is an oil pouring action; wherein the preprocessing operation is used to identify and separate an image of a stove area from an image frame; and the action recognition model is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN.

[0015] Optionally, the control module is specifically used to control the range hood to start and operate at a preset air volume level when the range hood is in a stopped state; the control module is also specifically used to increase the air volume level of the range hood when the range hood is in an operating state.

[0016] Optionally, the acquisition module is further used to acquire multiple first image sequences including the oil pouring action; each first image sequence includes the first preset number of image frames, and there are a second preset number of repeated image frames in the two first image sequences collected for the same action; the image processing module is further used to perform stove position segmentation on each image frame in the multiple first image sequences to obtain multiple second image sequences.

[0017] Optionally, the image processing module is specifically used to perform grayscale processing on the target image frame to obtain a grayscale image; and convert the grayscale image into a binary image through binary image transformation; the image processing module is also specifically used to perform image denoising processing on the binary image to obtain an image to be detected, and perform stove contour detection on the image to be detected to determine the stove position; the image processing module is also specifically used to perform image segmentation on the image to be detected based on the stove position to obtain a stove image frame containing only the stove area; wherein the target image frame is any image frame among the multiple image frames included in the multiple first image sequences; and the stove image frame is an image frame obtained after the target image frame is segmented by the stove position.

[0018] Optionally, the image processing module is further used to adjust each image frame in the multiple second image sequences to an image frame of a preset size, and perform normalization processing on each adjusted image frame based on the mean and variance of the Imagenet dataset to obtain multiple image sequences to be trained.

[0019] Optionally, the device further comprises: a model training module; the model training module is used to train the model to be trained by using the multiple image sequences to be trained as training samples of the model to be trained to obtain the action recognition model;

[0020] Among them, a sequence of images to be trained is a training sample; the model to be trained is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN; the model to be trained includes: a target convolution kernel obtained by fusing a 3*3*3 convolution kernel and a 5*5*5 convolution kernel; the target convolution kernel is used to extract motion features with smaller motion amplitudes and motion features with larger motion amplitudes; the model to be trained uses a linear rectification function ReLu activation function for nonlinear transformation; and the model to be trained uses cross entropy loss for optimization during the training process.

[0021] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of implementing any of the above-mentioned range hood control methods based on oil pouring action recognition through the computer program.

[0022] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program, when executed at runtime, implements the steps of any of the above-mentioned range hood control methods based on oil pouring action recognition.

[0023] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned range hood control methods based on oil pouring action recognition.

[0024] The range hood control method, device and electronic device based on the recognition of pouring oil action provided by the present application first obtain a target image sequence containing the stove area collected by the camera set on the range hood; the target image sequence includes: a first preset number of image frames; then, perform a preprocessing operation on each image frame in the target image sequence to obtain an image sequence to be recognized, and input the image sequence to be recognized into the action recognition model to obtain an action recognition result; finally, when the action recognition result indicates that the user's action is pouring oil, the range hood is controlled according to the current operating state of the range hood. In this way, after recognizing the user's pouring oil action, the air volume level of the range hood can be turned on / increased in time to avoid excessive oil smoke in the room. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is a schematic diagram of a hardware environment of an interaction method of a smart device according to an embodiment of the present application;

[0028] Figure 2 This is one of the flow charts of the range hood control method based on oil pouring action recognition provided by the present application;

[0029] Figure 3 This is the second flow chart of the range hood control method based on oil pouring action recognition provided by the present application;

[0030] Figure 4 It is a flowchart of performing preprocessing operations on image frames provided by the present application;

[0031] Figure 5 It is a structural diagram of the 3D-CNN network model provided by this application;

[0032] Figure 6 It is a structural schematic diagram of a range hood control device based on oil pouring action recognition provided by the present application;

[0033] Figure 7 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] According to one aspect of the embodiment of the present application, a range hood control method based on oil pouring action recognition is provided. The range hood control method based on oil pouring action recognition is widely used in smart home (Smart Home), smart home, smart home equipment ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the range hood control method based on oil pouring action recognition can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0037] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0038] In the related art, the range hood starts after the stove is turned on in the smoke and stove linkage scenario. However, users often do not need to start the range hood (immediately) after turning on the stove, and the range hood makes a lot of noise, so users are annoyed when starting the range hood in advance. In the infrared smoke sensing scenario in the related art, the range hood speed is adjusted by measuring the size of the oil smoke. This method can only be controlled when the oil smoke has been generated, and cannot adjust the range hood before the oil smoke is generated. Users usually want to start the range hood or increase the speed of the range hood before the oil smoke is generated.

[0039] The range hood control method in the related art cannot predict before the oil fume is about to be generated, and can only detect it after the smoke is generated, resulting in smoke in the kitchen. The range hood automatic control technology based on smoke sensing senses the oil fume after it is generated to control the opening of the range hood, but this method cannot predict before the oil fume is generated. The range hood linkage starting method in the related art starts the range hood immediately after the stove is turned on, but in some cases, the range hood does not need to be started when the stove is turned on, and the range hood makes a loud noise, and starting it early may cause annoyance to users.

[0040] In view of the problem of intelligent start-up of range hoods in related technologies, this application proposes a range hood control method based on oil pouring action recognition. The camera is used to obtain images, and an artificial intelligence algorithm is constructed to recognize the oil pouring action of a person. After the oil pouring action is recognized, the range hood is started or the air volume of the range hood is increased, solving the problem of users starting the range hood at the right time.

[0041] The range hood control method based on oil pouring action recognition provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0042] like Figure 2As shown, an embodiment of the present application provides a range hood control method based on oil pouring action recognition, which may include the following steps 201 to 203:

[0043] Step 201: Acquire a target image sequence including a stove area captured by a camera arranged on the range hood.

[0044] Wherein, the target image sequence includes: a first preset number of image frames.

[0045] For example, since the technical solution in this application needs to identify human body movements, and since movements are sequential, the user's pouring oil movement can be identified through the collected image sequence. The image frames in the above target image sequence are extracted from the video collected by the camera.

[0046] Specifically, in the embodiment of the present application, according to the statistics of the oil pouring action, every 10 frames of images can contain a complete oil pouring action, so the first preset number can be 10. That is, in the embodiment of the present application, the collected 10 frames of images can be used as an image sequence to identify the oil pouring action.

[0047] Step 202: perform a preprocessing operation on each image frame in the target image sequence to obtain a sequence of images to be recognized, and input the sequence of images to be recognized into an action recognition model to obtain an action recognition result.

[0048] Among them, the preprocessing operation is used to identify and separate the image of the stove area from the image frame; the action recognition model is a model constructed based on the three-dimensional convolutional neural network network 3D-CNN.

[0049] For example, since different installation positions of the stove will result in different viewing angles, in order to improve the recognition accuracy, it is necessary to pre-process the image frames in the target image sequence to recognize and separate the image of the stove area from the image frames.

[0050] For example, after performing preprocessing operations on the image frames in the target image sequence, they can be input into the action recognition model to recognize the pouring oil action. That is, the action recognition model can recognize whether the user's action is a pouring oil action or a non-pouring oil action.

[0051] For example, since the recognition of the pouring oil action is composed of a sequence of images, traditional convolutional neural networks (CNN) can only process a single image. In order to recognize the action in the video, this application uses a three-dimensional convolutional neural network network 3D-CNN to recognize the pouring oil action.

[0052] It should be noted that the above-mentioned action recognition model can be a model set in the range hood, and can also be a model set in the remote server.

[0053] Step 203: When the action recognition result indicates that the user's action is an oil pouring action, the range hood is controlled according to the current operating state of the range hood.

[0054] For example, when the action recognition model recognizes that the action in the image sequence to be recognized is an action of pouring oil, the range hood can be controlled according to the current operating state of the range hood.

[0055] Specifically, the above step 203 may include the following step 203a or step 203b:

[0056] Step 203a: When the range hood is in a stopped state, control the range hood to start and operate at a preset air volume level.

[0057] Step 203b: when the range hood is in operation, increase the air volume level of the range hood.

[0058] For example, Figure 3 As shown, the technical solution in the embodiment of the present application first obtains video information through a camera, extracts image data through an image processing method, performs data preprocessing such as normalization on the data, inputs the image frame into a trained deep learning network, outputs whether there is an oil pouring action, and finally controls the start-up of the range hood according to the output result. When the range hood is not started, the range hood is controlled to start. When it is already started, the fan speed of the range hood is increased to achieve "wind before smoke", and finally achieve high-precision oil fume extraction.

[0059] Optionally, in an embodiment of the present application, the above-mentioned action recognition model can be trained through the following steps.

[0060] For example, before the above step 202, the range hood control method based on oil pouring action recognition provided by the embodiment of the present application may further include the following steps 301 and 302:

[0061] Step 301: Acquire a plurality of first image sequences including an oil pouring action.

[0062] Each of the first image sequences includes the first preset number of image frames, and two first image sequences captured for the same action have a second preset number of repeated image frames.

[0063] Exemplarily, the actions corresponding to the first image sequences in the above-mentioned multiple first image sequences may be the same action or different actions, that is, one or more first image sequences may be extracted from a video based on the same user gesture action.

[0064] For example, in order to improve the recognition accuracy of the pouring oil action, when collecting samples (i.e., the above-mentioned multiple first image sequences), the embodiment of the present application has a second preset number of repeated image frames between two first image sequences collected for the same user gesture action. The second preset number is less than the first preset number.

[0065] In a possible implementation, the second preset number may be 5, that is, there are 5 identical image frames between two first image sequences captured for the same user gesture action.

[0066] Step 302: Segment each image frame in the plurality of first image sequences according to the position of the cooker to obtain a plurality of second image sequences.

[0067] For example, since the different installation positions of each stove result in different viewing angles, in order to improve the recognition accuracy, after acquiring the above-mentioned first image sequence, the stove position is first located in the image frame. In the embodiment of the present application, the image contour detection and image contour segmentation methods are used to detect the stove position.

[0068] Specifically, the above step 302 may include the following steps 302a1 to 302a3:

[0069] Step 302a1, grayscale the target image frame to obtain a grayscale image; and convert the grayscale image into a binary image through binary image transformation.

[0070] Step 302a2: Perform image noise reduction processing on the binary image to obtain an image to be detected, and perform stove contour detection on the image to be detected to determine the stove position.

[0071] Step 302a3: segment the image to be detected based on the stove position to obtain a stove image frame containing only the stove area.

[0072] The target image frame is any one of the multiple image frames included in the multiple first image sequences; and the stove image frame is an image frame obtained by segmenting the target image frame according to the stove position.

[0073] Exemplarily, for any target image frame in the first image sequence, as Figure 4As shown, firstly, the image frame needs to be binary converted, and then the stove outline in the image frame is detected. After the outline detection is completed, the stove position is identified, and the image is cut according to the identified stove, and only the image of the stove part is retained to reduce the influence of other redundant information images on the recognition result.

[0074] It should be noted that the preprocessing operation in the step of performing the preprocessing operation on each image frame in the target image sequence in step 102 can also be performed according to the steps in steps 302a1 to 302a3. That is, the target image frame can also be any image frame in the target image sequence.

[0075] For example, after the above step 302, the range hood control method based on oil pouring action recognition provided by the embodiment of the present application may further include the following step 303:

[0076] Step 303: adjust each image frame in the plurality of second image sequences to an image frame of a preset size, and perform normalization processing on each adjusted image frame based on the mean and variance of the Imagenet data set to obtain a plurality of image sequences to be trained.

[0077] For example, after the stove area in the image frame is segmented, the image needs to be resized uniformly. In the embodiment of the present application, the image is adjusted to a 480*640 image. At the same time, in order to reduce the influence of light and the like on the recognition result, the image is normalized. The embodiment of the present application uses the standard mean and variance of the Imagenet data set for normalization. The normalization formula includes the following:

[0078] mean=(0.458,0.456,0.406) (Formula 1)

[0079] std=(0.229,0.224,0.225) (Formula 2)

[0080] new_image=(org_image-mean) / std (Formula 3)

[0081] Among them, org_iamge is the original image, and new_image is the normalized image.

[0082] For example, after the above step 303, the range hood control method based on oil pouring action recognition provided by the embodiment of the present application may further include the following step 304:

[0083] Step 304: train the model to be trained by using the multiple image sequences to be trained as training samples of the model to be trained to obtain the action recognition model.

[0084] Among them, a sequence of images to be trained is a training sample; the model to be trained is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN; the model to be trained includes: a target convolution kernel obtained by fusing a 3*3*3 convolution kernel and a 5*5*5 convolution kernel; the target convolution kernel is used to extract motion features with smaller motion amplitudes and motion features with larger motion amplitudes; the model to be trained uses a linear rectification function ReLu activation function for nonlinear transformation; and the model to be trained uses cross entropy loss for optimization during the training process.

[0085] For example, Figure 5 As shown, it is a schematic diagram of the network structure of the model to be trained in the embodiment of the present application. In the first layer, 10 image frames are used as a group as input. Since the pouring oil action feature is the main action in the image, in terms of convolution kernel selection, in order to increase the robustness of convolution, 3*3*3 and 5*5*5 convolution kernels are used to merge, which not only extracts small action features, but also extracts large motion features, and also reduces the data information loss caused by convolution. At the same time, in order to fit nonlinear data, linear rectification function (Linear rectification function, ReLu) is used in the model for nonlinear transformation. The final output of the model is a binary classification output. Since it is a classification problem, the model is optimized using cross entropy loss during training.

[0086] In this way, the range hood can recognize the actions in the collected video through the deep learning model, and turn on the range hood or increase the air volume of the range hood after recognizing the oil pouring action, so as to avoid excessive oil smoke in the room.

[0087] The range hood control method based on the recognition of pouring oil action provided by the embodiment of the present application first obtains a target image sequence containing the stove area collected by the camera set on the range hood; the target image sequence includes: a first preset number of image frames; then, a preprocessing operation is performed on each image frame in the target image sequence to obtain an image sequence to be recognized, and the image sequence to be recognized is input into the action recognition model to obtain an action recognition result; finally, when the action recognition result indicates that the user's action is pouring oil action, the range hood is controlled according to the current operating state of the range hood. In this way, after recognizing the user's pouring oil action, the air volume level of the range hood can be turned on / increased in time to avoid excessive oil smoke in the room.

[0088] It should be noted that the range hood control method based on oil pouring action recognition provided in the embodiment of the present application can be executed by a range hood control device based on oil pouring action recognition, or a control module in the range hood control device based on oil pouring action recognition for executing the range hood control method based on oil pouring action recognition. In the embodiment of the present application, the range hood control device based on oil pouring action recognition executing the range hood control method based on oil pouring action recognition is taken as an example to illustrate the range hood control device based on oil pouring action recognition provided in the embodiment of the present application.

[0089] It should be noted that, in the embodiments of the present application, the range hood control method based on the oil pouring action recognition shown in the above-mentioned method drawings is illustrated by taking one of the drawings in the embodiments of the present application as an example. In specific implementation, the range hood control method based on the oil pouring action recognition shown in the above-mentioned method drawings can also be implemented in combination with any other drawings that can be combined as shown in the above-mentioned embodiments, which will not be described in detail here.

[0090] The range hood control device based on oil pouring action recognition provided by the present application is described below, and the range hood control method based on oil pouring action recognition described above and described below can refer to each other.

[0091] Figure 6 A schematic diagram of the structure of a range hood control device based on oil pouring action recognition provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, specifically including:

[0092] An acquisition module 601 is used to acquire a target image sequence including a stove area captured by a camera provided on the range hood; the target image sequence includes: a first preset number of image frames; an image processing module 602 is used to perform a preprocessing operation on each image frame in the target image sequence to obtain a sequence of images to be identified; an action recognition module 603 is used to input the sequence of images to be identified into an action recognition model to obtain an action recognition result; a control module 604 is used to control the range hood according to the current operating state of the range hood when the action recognition result indicates that the user's action is an oil pouring action; wherein the preprocessing operation is used to identify and separate an image of a stove area from an image frame; and the action recognition model is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN.

[0093] Optionally, the control module 604 is specifically used to control the range hood to start and operate at a preset air volume level when the range hood is in a stopped state; the control module 604 is also specifically used to increase the air volume level of the range hood when the range hood is in an operating state.

[0094] Optionally, the acquisition module 601 is further used to acquire multiple first image sequences including the oil pouring action; each first image sequence includes the first preset number of image frames, and there are a second preset number of repeated image frames in the two first image sequences collected for the same action; the image processing module 602 is further used to perform stove position segmentation on each image frame in the multiple first image sequences to obtain multiple second image sequences.

[0095] Optionally, the image processing module 602 is specifically used to perform grayscale processing on the target image frame to obtain a grayscale image; and convert the grayscale image into a binary image through binary image transformation; the image processing module 602 is also specifically used to perform image denoising processing on the binary image to obtain an image to be detected, and perform stove contour detection on the image to be detected to determine the stove position; the image processing module 602 is also specifically used to perform image segmentation on the image to be detected based on the stove position to obtain a stove image frame containing only the stove area; wherein the target image frame is any image frame of the multiple image frames contained in the multiple first image sequences; and the stove image frame is an image frame obtained after the target image frame is segmented by the stove position.

[0096] Optionally, the image processing module 602 is further used to adjust each image frame in the multiple second image sequences to an image frame of a preset size, and perform normalization processing on each adjusted image frame based on the mean and variance of the Imagenet dataset to obtain multiple image sequences to be trained.

[0097] Optionally, the device further comprises: a model training module; the model training module is used to train the model to be trained by using the multiple image sequences to be trained as training samples of the model to be trained to obtain the action recognition model;

[0098] Among them, a sequence of images to be trained is a training sample; the model to be trained is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN; the model to be trained includes: a target convolution kernel obtained by fusing a 3*3*3 convolution kernel and a 5*5*5 convolution kernel; the target convolution kernel is used to extract motion features with smaller motion amplitudes and motion features with larger motion amplitudes; the model to be trained uses a linear rectification function ReLu activation function for nonlinear transformation; and the model to be trained uses cross entropy loss for optimization during the training process.

[0099] The range hood control device based on the recognition of pouring oil action provided by the present application first obtains a target image sequence containing the stove area collected by the camera set on the range hood; the target image sequence includes: a first preset number of image frames; then, a preprocessing operation is performed on each image frame in the target image sequence to obtain an image sequence to be recognized, and the image sequence to be recognized is input into the action recognition model to obtain an action recognition result; finally, when the action recognition result indicates that the user's action is pouring oil action, the range hood is controlled according to the current operating state of the range hood. In this way, after recognizing the user's pouring oil action, the air volume level of the range hood can be turned on / increased in time to avoid excessive oil smoke in the room.

[0100] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a range hood control method based on the recognition of the pouring oil action, the method comprising: obtaining a target image sequence containing a stove area collected by a camera set on the range hood; the target image sequence comprises: a first preset number of image frames; performing a preprocessing operation on each image frame in the target image sequence to obtain an image sequence to be recognized, and inputting the image sequence to be recognized into an action recognition model to obtain an action recognition result; when the action recognition result indicates that the user's action is an oil pouring action, controlling the range hood according to the current running state of the range hood; wherein the preprocessing operation is used to recognize and separate the image of the stove area from the image frame; and the action recognition model is a model constructed based on a three-dimensional convolutional neural network 3D-CNN.

[0101] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0102] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the range hood control method based on oil pouring action recognition provided by the above-mentioned methods, the method including: obtaining a target image sequence containing a stove area captured by a camera set on the range hood; the target image sequence includes: a first preset number of image frames; performing a preprocessing operation on each image frame in the target image sequence to obtain an image sequence to be recognized, and inputting the image sequence to be recognized into an action recognition model to obtain an action recognition result; when the action recognition result indicates that the user's action is an oil pouring action, controlling the range hood according to the current operating state of the range hood; wherein the preprocessing operation is used to recognize and separate the image of the stove area from the image frame; the action recognition model is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN.

[0103] On the other hand, the present application also provides a computer-readable storage medium, which includes a stored program, wherein the program, when running, executes the range hood control method based on oil pouring action recognition provided by the above-mentioned methods, the method comprising: obtaining a target image sequence containing a stove area captured by a camera set on the range hood; the target image sequence includes: a first preset number of image frames; performing a preprocessing operation on each image frame in the target image sequence to obtain an image sequence to be recognized, and inputting the image sequence to be recognized into an action recognition model to obtain an action recognition result; when the action recognition result indicates that the user's action is an oil pouring action, controlling the range hood according to the current operating state of the range hood; wherein the preprocessing operation is used to identify and separate the image of the stove area from the image frame; the action recognition model is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN.

[0104] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A range hood control method based on oil pouring action recognition, characterized in that: Applied to a range hood, the method comprises: Acquire a target image sequence including a cooking range area captured by a camera arranged on the range hood; the target image sequence includes: a first preset number of image frames; Performing a preprocessing operation on each image frame in the target image sequence to obtain a sequence of images to be recognized, and inputting the sequence of images to be recognized into an action recognition model to obtain an action recognition result; When the action recognition result indicates that the user's action is an oil pouring action, controlling the range hood according to the current operating state of the range hood; Among them, the preprocessing operation is used to identify and separate the image of the stove area from the image frame; the action recognition model is a model constructed based on the three-dimensional convolutional neural network network 3D-CNN.

2. The method according to claim 1, characterized in that When the action recognition result indicates that the action of the user is an oil pouring action, controlling the range hood according to the current operating state of the range hood includes: When the range hood is in a stopped state, controlling the range hood to start and operate at a preset air volume level; or, When the range hood is in operation, the air volume level of the range hood is increased.

3. The method according to claim 1, characterized in that Before inputting the to-be-recognized image sequence into the action recognition model to obtain the action recognition result, the method further comprises: Acquire a plurality of first image sequences including the oil pouring action; each of the first image sequences includes the first preset number of image frames, and two first image sequences acquired for the same action have a second preset number of repeated image frames; Perform stove position segmentation on each image frame in the plurality of first image sequences to obtain a plurality of second image sequences.

4. The method according to claim 3, characterized in that The step of performing stove position segmentation on each image frame in the plurality of first image sequences to obtain a plurality of second image sequences comprises: After graying the target image frame, a grayscale image is obtained; and the grayscale image is converted into a binary image through binary image transformation; Performing image noise reduction processing on the binary image to obtain an image to be detected, and performing cooker contour detection on the image to be detected to determine the position of the cooker; Performing image segmentation on the image to be detected based on the position of the cooker to obtain a cooker image frame containing only the cooker area; The target image frame is any one of the multiple image frames included in the multiple first image sequences; and the stove image frame is an image frame obtained by segmenting the target image frame according to the stove position.

5. The method according to claim 3 or 4, characterized in that: After performing stove position segmentation on each image frame in the plurality of first image sequences to obtain a plurality of second image sequences, the method further comprises: Each image frame in the plurality of second image sequences is adjusted to an image frame of a preset size, and normalization processing is performed on each adjusted image frame based on the mean and variance of the Imagenet data set to obtain a plurality of image sequences to be trained.

6. The method according to claim 5, characterized in that After performing normalization processing on each adjusted image frame based on the mean and variance of the Imagenet data set to obtain a plurality of image sequences to be trained, the method further includes: Using the plurality of to-be-trained image sequences as training samples of the to-be-trained model to train the to-be-trained model, to obtain the action recognition model; Among them, a sequence of images to be trained is a training sample; the model to be trained is a model constructed based on a three-dimensional convolutional neural network network 3D-CNN; the model to be trained includes: a target convolution kernel obtained by fusing a 3*3*3 convolution kernel and a 5*5*5 convolution kernel; the target convolution kernel is used to extract motion features with smaller motion amplitudes and motion features with larger motion amplitudes; the model to be trained uses a linear rectification function ReLu activation function for nonlinear transformation; and the model to be trained uses cross entropy loss for optimization during the training process.

7. A range hood control device based on oil pouring action recognition, characterized in that: The device comprises: An acquisition module, used for acquiring a target image sequence including a cooking range area acquired by a camera arranged on the range hood; the target image sequence comprises: a first preset number of image frames; An image processing module, used for performing a preprocessing operation on each image frame in the target image sequence to obtain an image sequence to be identified; An action recognition module, used for inputting the image sequence to be recognized into an action recognition model to obtain an action recognition result; a control module, configured to control the range hood according to a current operating state of the range hood when the action recognition result indicates that the user's action is an oil pouring action; Among them, the preprocessing operation is used to identify and separate the image of the stove area from the image frame; the action recognition model is a model constructed based on the three-dimensional convolutional neural network network 3D-CNN.

8. The device according to claim 6, characterized in that The control module is specifically used to control the range hood to start and operate at a preset air volume level when the range hood is in a stopped state; the control module is also specifically used to increase the air volume level of the range hood when the range hood is in an operating state.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program, when running, executes the range hood control method based on oil pouring action recognition according to any one of claims 1 to 6.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the range hood control method based on oil pouring action recognition according to any one of claims 1 to 6 through the computer program.

Images