Cooking recognition method and device based on hand behavior

Abstract

The application provides a cooking recognition method and device based on hand behaviors, and relates to the technical field of intelligent cooking.The method comprises the following steps: acquiring a video frame sequence captured by a camera, extracting key points from the video frame sequence, constructing a key point set, and constructing a multi-dimensional feature vector based on the key point set; matching member information corresponding to the multi-dimensional feature vector from a pre-constructed feature database; and generating recommendation information matched with the member information based on the member information.The cooking recognition method and device based on hand behaviors can avoid recognition failure caused by invisible faces in a kitchen scene, as the method does not directly use face detection in the process of recognizing member information.Meanwhile, the method can determine member information only through local features, which improves privacy friendliness to some extent and thus helps to improve user experience.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent cooking, and in particular to a cooking recognition method and device based on hand behavior. Background Technology

[0002] With the popularization of smart home and smart kitchen products, kitchen cameras have been widely used in scenarios such as oil fume monitoring, flame detection, overflow detection, child care, and cooking assistance. Existing kitchen vision systems mainly focus on stove status recognition or hazard event detection, such as whether there is an open flame, whether overflow has occurred, or whether there has been prolonged empty burning.

[0003] In a family setting, different family members often have significantly different cooking frequencies, dietary preferences, and safe operating habits. Being able to identify which specific family member is performing the cooking activity would facilitate more personalized information recommendations. However, traditional technologies for family member identification typically rely on biometric identification methods such as facial recognition and voiceprint recognition.

[0004] However, due to the limitations of the kitchen setting, traditional technologies have certain limitations. For example, the camera's angle may make people's faces invisible, and voiceprint recognition is easily affected by the noise of the range hood, resulting in low recognition efficiency and thus reducing the user experience. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a cooking recognition method and apparatus based on hand behavior to alleviate the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a cooking recognition method based on hand behavior, applied to a range hood, wherein the range hood is equipped with a camera, and the method includes: acquiring a video frame sequence captured by the camera, wherein the video frame sequence is video footage captured from a top-down perspective by a camera installed at the bottom of the range hood; extracting key points from the video frame sequence, constructing a key point set, and constructing a multi-dimensional feature vector based on the key point set; wherein the key points include key points of the user's hands and key points of the arms; matching member information corresponding to the multi-dimensional feature vector from a pre-constructed feature database; wherein the feature database pre-records the mapping relationship between known multi-dimensional feature vectors and member information; and generating recommendation information matching the member information based on the member information.

[0007] In conjunction with the first aspect, the present invention provides a first possible implementation of the first aspect, wherein the step of extracting key points from the video frame sequence and constructing a key point set includes: extracting hand key points and arm key points, and constructing a key point set of arm linkage topology; wherein the arm key points include at least: wrist points and finger joint points; the arm key points include at least: elbow joint points and forearm midline points.

[0008] In conjunction with the first possible implementation of the first aspect, this embodiment of the invention provides a second possible implementation of the first aspect, wherein the step of constructing a multidimensional feature vector based on the set of key points includes: determining morphological features based on the hand key points and arm key points; extracting spatiotemporal features from the continuous video frame sequence to extract spatiotemporal feature vectors characterizing hand action behavior features; and constructing a multidimensional feature vector containing the morphological features and the spatiotemporal feature vectors.

[0009] In conjunction with the first aspect, this embodiment of the invention provides a third possible implementation of the first aspect, wherein the step of determining morphological features based on the key points of the hand and the key points of the arm includes: determining multidimensional morphological features representing the hand based on the key points of the hand and the key points of the arm; wherein the multidimensional morphological features include at least: hand palm-finger ratio vector, palm vector, forearm length feature, and forearm morphological width feature.

[0010] In conjunction with the first aspect, this embodiment of the invention provides a fourth possible implementation of the first aspect, wherein the step of matching the member information corresponding to the multidimensional feature vector from a pre-constructed feature database includes: searching the feature database for a known multidimensional feature vector whose similarity to the multidimensional feature vector is higher than a preset similarity threshold; and determining the member information corresponding to the found known multidimensional feature vector as the member information corresponding to the multidimensional feature vector.

[0011] In conjunction with the first aspect, this invention provides a fifth possible implementation of the first aspect, wherein the method further includes: monitoring the cooking process to obtain monitoring data, wherein the monitoring data includes the user's morphological characteristics and the user's hand movement behavior characteristics during the cooking process; statistically analyzing the monitoring data within a preset time period, and constructing multi-dimensional feature vectors corresponding to multiple cooking subjects based on the monitoring data; obtaining tag information of the cooking subjects, wherein the tag information records the member information of the cooking subjects; and constructing a feature database based on the tag information and the multi-dimensional feature vector corresponding to each cooking subject.

[0012] In conjunction with the fifth possible implementation of the first aspect, this embodiment of the invention provides a sixth possible implementation of the first aspect, wherein the above method further includes: if no member information corresponding to the multidimensional feature vector is matched in the feature database, then the multidimensional feature vector is determined as a candidate feature vector; the number of cooking times of the candidate feature vector is counted; if the number of cooking times reaches a preset threshold, then a new cooking subject is generated, and the new cooking subject is used to update the feature database.

[0013] In conjunction with the first aspect, this embodiment of the invention provides a seventh possible implementation of the first aspect, wherein the step of generating recommendation information matching the member information based on the member information and the cooking behavior includes: searching for recommendation information corresponding to the member information in a pre-built information database, wherein the recommendation information includes at least one of the following: cooking protection suggestions, nutritional diet suggestions, and cooking utensils suggestions; and sending the found recommendation information to the corresponding user terminal so that the recommendation information can be presented through the user terminal.

[0014] In conjunction with the seventh possible implementation of the first aspect, this embodiment of the invention provides an eighth possible implementation of the first aspect, wherein the above method further includes: statistically analyzing cooking data corresponding to each member's information, wherein the cooking data includes cooking frequency and cooking time period, cooking mode, cooking intensity, and cooking operation habits; determining corresponding recommendation information based on the cooking data; generating a correspondence between the member information and the recommendation information, and storing the correspondence in the information database.

[0015] Secondly, embodiments of the present invention also provide a cooking recognition device based on hand behavior, applied to a range hood. The range hood is equipped with a camera. The device includes: an acquisition module, used to acquire a video frame sequence captured by the camera, wherein the video frame sequence is video footage captured from a top-down perspective by a camera installed at the bottom of the range hood; an extraction module, used to extract key points from the video frame sequence, construct a key point set, and construct a multi-dimensional feature vector based on the key point set; wherein the key points include key points of the user's hands and key points of the arms; a matching module, used to match member information corresponding to the multi-dimensional feature vector from a pre-constructed feature database; wherein the feature database pre-records the mapping relationship between known multi-dimensional feature vectors and member information; and a recommendation module, used to generate recommendation information matching the member information based on the member information.

[0016] The embodiments of the present invention bring the following beneficial effects: This invention provides a cooking recognition method and apparatus based on hand behavior, which can acquire video frame sequences captured by a camera; extract key points from the video frame sequences, construct a key point set, and construct a multi-dimensional feature vector based on the key point set; match the member information corresponding to the multi-dimensional feature vector from a pre-constructed feature database, and generate recommendation information matching the member information based on the member information. Since face detection is not directly used in the process of recognizing member information, the recognition failure caused by the invisible face in the kitchen scene can be avoided. At the same time, by using multi-dimensional feature vector matching, member information can be determined only by local features, which also improves privacy to a certain extent and helps to improve the user experience.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a cooking recognition method based on hand behavior, provided as an embodiment of the present invention; Figure 2 A system architecture diagram provided for an embodiment of the present invention; Figure 3 A schematic diagram of a cooking recognition device based on hand behavior provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Typically, identifying the family member cooking allows for personalized nutritional analysis, safety reminders, and product recommendations. Current technologies for family member identification usually rely on biometric methods such as facial recognition and voiceprint recognition. However, due to the limitations of the kitchen environment, facial recognition cameras are often installed below the range hood, taking a downward angle, which can easily obscure the user's face, causing the facial recognition algorithm to fail and making it difficult to meet the reliability requirements for family member identification. Furthermore, the noise generated by the range hood and other equipment during cooking can reduce the efficiency of voiceprint recognition. In addition, both facial and voiceprint recognition involve the collection and processing of sensitive personal information, and the backend processing on cloud servers often carries the risk of data leakage, resulting in a poor user experience.

[0023] Furthermore, existing behavior recognition technologies are primarily geared towards security monitoring or human-computer interaction scenarios, focusing on instantaneous postures or short-term behavioral patterns. They are ill-suited to the statistical analysis needs of long-term, repetitive behaviors in scenarios like kitchen cooking. Particularly for scenarios requiring long-term, cumulative identification of family members, current technologies lack effective adaptive updates and drift suppression mechanisms. Therefore, existing technologies struggle to perform long-term, stable identification and analysis of the cooking behaviors of different family members in a low-intrusion, privacy-friendly manner, even in overhead kitchen scenes.

[0024] Based on this, the cooking recognition method and apparatus based on hand behavior provided in this embodiment of the invention can effectively alleviate the above-mentioned technical problems.

[0025] To facilitate understanding of this embodiment, a cooking recognition method based on hand behavior disclosed in this embodiment of the invention will first be described in detail.

[0026] In one possible implementation, this invention provides a cooking recognition method based on hand gestures. This method is applied to a range hood, which is equipped with a camera that can capture video footage of the cooking area from a top-down perspective. Specifically, in this embodiment, the camera is installed below the range hood and facing the stove area, allowing for top-down capture of the cooking process. Furthermore, the cooking recognition method based on hand gestures provided in this embodiment can run on a local processing unit to achieve a privacy-preserving processing architecture prioritizing the edge. When sufficient computing resources are available, it can also be deployed on a cloud computing platform. When deployed on a cloud computing platform, sensitive data such as user hand gesture characteristics and other user-related data are typically processed locally, with only non-sensitive analysis results uploaded to the cloud computing platform. The specific operating method can be configured according to actual usage, and this invention does not impose any limitations on this.

[0027] Furthermore, such as Figure 1 The flowchart shown illustrates a cooking recognition method based on hand behavior, which includes the following steps: Step S102: Obtain the video frame sequence captured by the camera; In this embodiment of the invention, the video frame sequence is a video image captured from a top-down perspective by a camera installed at the bottom of the range hood. Step S104: Extract key points from the video frame sequence, construct a key point set, and construct a multi-dimensional feature vector based on the key point set; Among them, the key points in the embodiments of the present invention include key points of the user's hand and key points of the arm; In practical use, considering that the camera installed at the bottom of the range hood is usually positioned facing the stove area, it is typically a top-down view. Therefore, the video frame sequence usually includes visible areas such as hands, arms, pots, and stoves. In this embodiment of the invention, the hand and arm areas are extracted for key point extraction, and then a multi-dimensional feature vector is constructed.

[0028] Step S106: Match member information corresponding to multidimensional feature vectors from a pre-built feature database; The feature database contains pre-recorded mapping relationships between known multidimensional feature vectors and member information. Step S108: Generate recommendation information that matches the member information based on the member information.

[0029] This invention provides a cooking recognition method based on hand gestures, which can acquire video frame sequences captured by a camera; extract key points from the video frame sequences, construct a key point set, and construct a multi-dimensional feature vector based on the key point set; match the member information corresponding to the multi-dimensional feature vector from a pre-constructed feature database, and generate recommendation information matching the member information. Since face detection is not directly used in the process of recognizing member information, it can avoid recognition failures caused by the invisibility of faces in kitchen scenarios. At the same time, by using multi-dimensional feature vector matching, member information can be determined only by local features, which also improves privacy to a certain extent and helps to improve the user experience.

[0030] In practical use, since the key points in this embodiment of the invention include key points of the user's hand and arm, it is necessary to extract the key points of the hand and arm separately when extracting key points in step S104. In practical use, a dedicated hand detection unit can be deployed to perform the key point extraction process. Correspondingly, a camera acquisition unit can also be deployed to be responsible for acquiring video data from a camera fixedly installed under the range hood. For ease of understanding, Figure 2 A system architecture diagram is shown, illustrating the overall architecture of a cooking recognition method based on hand gestures. Specifically, as shown... Figure 2 As shown, the overall architecture includes a camera acquisition unit 20, a hand detection unit 21, a feature extraction unit 22, a behavior center management unit 23, an identity mapping unit 24, a behavior analysis unit 25, a decision output unit 26, and a feature database 27.

[0031] Specifically, the camera acquisition unit 20 corresponds to the process in step S102 above. The camera acquisition unit is used to continuously acquire the cooking process from a top-down perspective using the camera, and output the original video frame sequence to provide a data source for subsequent processing.

[0032] Furthermore, the hand detection unit 21 and the feature extraction unit 22 correspond to the process of step S104 above, that is, receiving the video frame sequence output by the camera acquisition unit 20, extracting key points from the video frame sequence, and constructing a key point set. Specifically, this includes extracting hand key points and arm key points, and constructing a key point set for the arm linkage topology. Among them, the arm key points include at least: wrist points and finger joint points; the arm key points include at least: elbow joint points and forearm midline points.

[0033] In practical use, this embodiment of the invention uses a target detection algorithm to detect the hand and arm regions in the aforementioned video frames, and uses semantic segmentation technology to accurately locate the hand contour, outputting pixel-level region information of the hand and arm, providing accurate localization for feature extraction. For example, a lightweight target detection algorithm (such as YOLO) is used to detect the hand and arm regions in the video frames, with a detection confidence threshold set to 0.6. That is, when the confidence exceeds this confidence threshold, the accurately located hand contour is determined to be extracted. Further instance segmentation techniques (such as YOLO-Seg, Mask R-CNN) are used to accurately locate the hand contour, extracting pixel-level regions of the hand and arm. This step provides accurate hand region localization information for subsequent feature extraction, ensuring stable operation even in a kitchen environment with obstructions.

[0034] Furthermore, in the feature extraction unit 22, a hierarchical feature extraction process is performed based on the extracted hand region positioning information, that is, a multi-dimensional feature vector is constructed based on the key point set. Specifically, this includes: determining morphological features based on hand key points and arm key points; performing spatiotemporal feature extraction on a continuous video frame sequence to extract spatiotemporal feature vectors that characterize hand action behavior features; and constructing a multi-dimensional feature vector that includes morphological features and spatiotemporal feature vectors.

[0035] In practical applications, to extract morphological features, keypoint detection algorithms can be used in this embodiment of the invention to detect keypoints of the hand and arm. For example, the coordinates of hand keypoints can be extracted using the OpenPose hand model or MediaPipe Hands, and the coordinates of arm keypoints can be extracted using the OpenPose human pose model or MediaPipe Pose. Specifically, in this embodiment, the hand keypoints typically include 21 points, denoted as P0~P20, including the wrist point (P0) and 20 finger joint points; the arm keypoints include the elbow joint point (P0). ): The center point of the joint where the forearm and upper arm meet, used to define the connection between the forearm and upper arm; the mid-axis point of the forearm ( : Defined as the midpoint of the line connecting the elbow joint and the wrist key point, that is, the point in the middle of the line connecting the forearm and the wrist. This point can be used to calculate the shape width feature of the forearm.

[0036] Furthermore, the morphological features in this embodiment of the invention are multidimensional morphological features representing the hand, determined based on the aforementioned key points of the hand and arm. These multidimensional morphological features focus on extracting scale-invariant features that are unrelated to cooking actions and only related to individual physiological development characteristics. Specifically, the multidimensional morphological features in this embodiment of the invention include at least: (1) Palmar-finger ratio vector; Specifically, the ratio of the length of each finger bone to the length of its corresponding metacarpal bone can be calculated, such as the ratio of the length of the distal phalanx of the index finger to the length of the proximal phalanx of the index finger. This ratio is determined by the development of the finger bones and has a very strong ability to identify the finger's identity. (2) Palm vector; The palm vector, also known as the palm width ratio or palm length ratio, is usually calculated by taking the distance from the base of the index finger (base of the second metacarpal bone) to the base of the little finger (base of the fifth metacarpal bone) as the palm width and the distance from the base of the middle finger to the center point of the wrist as the palm length. The ratio of the palm width to the palm length can distinguish different palm shapes and is not affected by the opening and closing of the hand. (3) Forearm length characteristics; Specifically, the elbow joint point can be used ( ) and wrist point ( ), calculate forearm length From a top-down view of the kitchen, since the camera is positioned below the range hood and facing the stove area, only the forearm and hand areas are usually visible, while the shoulder joints are not. Therefore, the forearm length feature serves as a stable marker of upper limb morphology. This feature has good convergence in top-down projection and can effectively identify different family members. (4) Forearm shape and width characteristics.

[0037] Specifically, the image information of the forearm midline point and its surrounding area can be used to calculate the cross-sectional width or relative proportion of the forearm at that position. This feature can reflect the thickness of an individual's arm or muscle distribution and has a high degree of identity recognition when viewed from above. Furthermore, for the various morphological features obtained above, feature normalization processing can be performed. Specifically, this eliminates pixel differences caused by the distance between the member's standing position and the camera. All linear distances are normalized using the length of the proximal phalanx of the middle finger or the distance from the base of the index finger to the base of the little finger (total metacarpal span) as the reference unit to ensure that the feature vector has scale invariance. Finally, the various morphological features mentioned above constitute the multidimensional morphological features in this embodiment of the invention, which can be represented as follows: The dimensions of this multidimensional morphological feature are typically 18 to 28, and it exhibits high stability over time during different cooking processes, effectively identifying different family members.

[0038] Furthermore, since the cooking process has a certain temporal continuity, in this embodiment of the invention, in addition to constructing the above-mentioned multidimensional morphological features, it is also necessary to further extract the spatiotemporal feature vector representing the characteristics of hand movements. This process is also called the operation behavior feature extraction process. For example, a 3D convolutional neural network or a Video Swin Transformer network can be used to extract spatiotemporal features from a continuous sequence of video frames to capture the dynamic evolution of cooking actions and thus obtain the spatiotemporal feature vector.

[0039] Specifically, the process can include two optional paths: (1) spatiotemporal feature extraction using a 3D convolutional neural network; and (2) long-term modeling based on Video Swin Transformer, which will be explained below.

[0040] (1) Spatiotemporal feature extraction using 3D convolutional neural networks: For example, a 3×3×3 three-dimensional convolution kernel is used to perform convolution operations simultaneously in the spatial dimension (H, W) and the temporal dimension (T). By stacking multiple 3D convolutional layers (such as the 5 convolutional layers in the C3D architecture, with 64, 128, 256, 256, and 512 channels respectively) and 3D pooling layers, low-order spatiotemporal features containing object appearance and short-term motion are extracted from 16 consecutive frames of hand region images (frame rate 10fps, corresponding to a 1.6-second time window). These features can capture the motion trajectory, speed changes, and operation frequency of the hand when interacting with kitchen utensils, effectively distinguishing between behaviors such as "quick chopping" and "slow cutting," which are highly similar in static features but significantly different in temporal dimension. It should be noted that, for the short-duration, high-frequency characteristics of hand movements in the kitchen, eight consecutive frames (corresponding to a 0.8-second time window) can also be used as input to reduce computational latency and capture instantaneous micro-movements (such as sprinkling salt, lighting a fire, etc.). However, a 16-frame configuration can cover a more complete action cycle and provide a more stable feature extraction effect when computational resources allow. The specific number of frames can be set according to actual usage, and this embodiment of the invention does not impose any restrictions on this.

[0041] (2) Long-term modeling based on Video Swin Transformer: A hierarchical visual Transformer structure is adopted, and long-range dependencies between different video blocks are modeled through a 3D ShiftedWindow based Self-Attention mechanism. Specifically, the video frame sequence is divided into non-overlapping 3D patches as token inputs, using a tiny configuration (e.g., 4 stages, each stage containing 2 Swing Transformer blocks, with a window size of 8×7×7). The number of input frames can be selected as 8 or 16 frames (10fps) depending on the scene requirements. Cross-window information interaction is achieved through shifted window operations, thereby obtaining the deep semantic association between "ingredient state changes" and "human posture evolution" during the cooking process, effectively identifying complete cooking sequences with long-term dependencies such as "preparing ingredients - putting them in the pot - taking them out of the pot".

[0042] Furthermore, both of the above paths can output 256-dimensional behavioral feature vectors. Compared with traditional optical flow methods or "2D CNN + RNN" architectures, this method can more directly perform end-to-end modeling of the video volume, avoiding the loss of inter-frame information during transmission, and has stronger robustness in scenes with compact movements and frequent occlusions in a kitchen overhead view.

[0043] Furthermore, the hierarchical feature design extracted through morphological and spatiotemporal features in the embodiments of the present invention can effectively avoid establishing identity recognition based on a single appearance feature or instantaneous behavior, thus forming a temporal consistency model for the question of "who is cooking".

[0044] Furthermore, the aforementioned behavior center management unit 23 corresponds to the procedure for constructing a feature database. Specifically, in this embodiment of the invention, the feature database is constructed through an implicit construction method. This allows the multi-dimensional feature vectors extracted during each cooking process to be mapped to a unified feature space. Based on the long-term aggregation relationship of features in the space, multiple stable behavior centers are adaptively formed. Each behavior center corresponds to a potential cooking subject, and the behavior center gradually stabilizes during the continuous process. Essentially, this is the result of long-term implicit modeling constrained by behavioral consistency. Therefore, without pre-defining the number of family members or identity labels, the natural identification of the cooking subject can be achieved through the natural aggregation of the feature space.

[0045] Specifically, in this embodiment of the invention, monitoring data can be obtained by monitoring the cooking process. This monitoring data includes the user's morphological characteristics and the user's hand gestures during the cooking process. The monitoring data within a preset time period is statistically analyzed, and multi-dimensional feature vectors corresponding to multiple cooking subjects are constructed based on the monitoring data. Tag information for the cooking subjects is obtained, whereby the tag information records member information of the cooking subjects. A feature database is constructed based on the tag information and the multi-dimensional feature vector corresponding to each cooking subject. Figure 2 Feature database 27.

[0046] In practical use, the purpose of statistically analyzing monitoring data over a preset time period is to adaptively form multiple stable behavior centers, each corresponding to a potential cooking entity. That is, in this embodiment of the invention, there is no need to pre-enter the number of family members; multiple stable behavior centers naturally form through long-term feature aggregation. In other words, the number of behavior centers can be obtained through long-term statistical results, and this number represents the number of family members who frequently cook.

[0047] Furthermore, for each cooking subject, the member information of the cooking subject can be confirmed through the weakly supervised process of the aforementioned identity mapping unit 24. Specifically, the user can set the content of the tag information, such as allowing the user to map the behavior center to the member identity of a specific family through one-time confirmation, usage frequency statistics, or time period rules, thereby obtaining tag information representing member information. This mapping process does not participate in the subject modeling itself, but only serves as supplementary information for behavior statistics and result display, and can output tag information representing the identity of family members, providing member information for subsequent behavior analysis units.

[0048] Furthermore, in this embodiment of the invention, in step S106 above, if no member information corresponding to the multidimensional feature vector is matched in the feature database, the multidimensional feature vector is determined as a candidate feature vector; the cooking times of the candidate feature vector are counted; if the cooking times reach a preset threshold, a new cooking subject is generated, and the new cooking subject is used to update the feature database.

[0049] In other words, the feature database in this embodiment of the invention allows for an adaptive update process. This adaptive update process is not a clustering result of a single cooking process, but a dynamic constraint result across cooking processes. That is, a new cooking subject is only determined after the number of cooking times of candidate feature vectors reaches a certain threshold. Specifically, newly collected multidimensional feature vectors are only used to update the feature database when they satisfy the consistency condition with their existing behavioral centers. Furthermore, this method can also achieve data drift suppression in the feature database. For example, if no known multidimensional feature vector with a similarity higher than a preset similarity threshold is found in the feature database, it indicates that the multidimensional feature vector deviates from the existing feature samples of the cooking subject. The unselected feature vectors are temporarily stored without directly updating the feature database. After determining the time window or cumulative stability condition, such as after the number of cooking times of candidate feature vectors reaches a preset threshold, it is determined that the condition for generating a new cooking subject is met, and thus it can be used to update the feature database.

[0050] Furthermore, the feature database in this embodiment of the invention forms a cooking subject through feature aggregation of multiple processes without pre-defining the number of family members or identity labels, thus achieving subject modeling without explicit identity supervision.

[0051] Furthermore, the above Figure 2 The behavior analysis unit 25 corresponds to the process of step S106 in the embodiment of the present invention, that is, to perform statistical analysis of cooking behavior based on the feature database constructed by the behavior center management unit 23 and the identity mapping unit 24.

[0052] Specifically, the behavior analysis unit 25 can acquire the aforementioned feature database to obtain the mapping relationship between known multidimensional feature vectors and member information in the feature database; then, it can search for known multidimensional feature vectors in the feature database whose similarity to the multidimensional feature vectors is higher than a preset similarity threshold; and determine the member information corresponding to the found known multidimensional feature vectors as the member information corresponding to the multidimensional feature vectors.

[0053] In practical use, the similarity described above in the embodiments of the present invention can be characterized by Euclidean distance. For example, the multidimensional feature vector is represented as follows: The known multidimensional feature vectors in the feature database are represented as follows: When multidimensional feature vectors With known multidimensional feature vectors When the Euclidean distance satisfies the consistency condition, for example, the Euclidean distance is expressed as... If the Euclidean distance is small, it indicates that the multidimensional feature vector is relatively close to the known multidimensional feature vector, and the corresponding similarity is high; conversely, a larger Euclidean distance indicates a larger difference between the multidimensional feature vector and the known multidimensional feature vector, and the corresponding similarity is low. A small Euclidean distance can be considered to satisfy the consistency condition, while a large Euclidean distance indicates that the consistency condition is not satisfied. In this case, the multidimensional feature vector can be represented as... As candidate feature vectors.

[0054] Furthermore, the system can analyze the cooking frequency and time distribution of each family member, the typical cooking operation patterns and intensities of different members, and their operating habits in high-temperature and oil-splattering scenarios. The analysis results provide data support for the decision-making output module, essentially building an information database to support the output of recommendation information. Specifically, it can statistically analyze the cooking data corresponding to each member's information, including cooking frequency and time, cooking mode, cooking intensity, and cooking operation habits. Based on the cooking data, it can determine the corresponding recommendation information, generate a correspondence between member information and recommendation information, and store this correspondence in the information database.

[0055] Furthermore, the above Figure 2 The decision output unit 26 corresponds to the process in step S108 above. Specifically, it can search for the recommended information corresponding to the member information in the information database constructed above. In this embodiment of the invention, the recommended information includes at least one of the following: cooking protection suggestions, nutritional diet suggestions, and cooking utensils suggestions. The found recommended information is sent to the corresponding user terminal so that the recommended information can be presented through the user terminal.

[0056] For example, it can generate safety protection recommendations, nutritional diet recommendations, or recommendations for related products for family members. Moreover, the above information database is based on the results of long-term statistics, rather than judgments based on a single identification, thereby improving the reliability of decision outputs.

[0057] Furthermore, this embodiment of the invention also provides a cooking recognition device based on hand gestures, applied to a range hood, wherein the range hood is equipped with a camera, such as... Figure 3 The diagram shows a structural schematic of a cooking recognition device based on hand behavior. The device includes: The acquisition module 30 is used to acquire the video frame sequence captured by the camera, wherein the video frame sequence is a video image captured from a top-down perspective by a camera installed at the bottom of the range hood; Extraction module 32 is used to extract key points from the video frame sequence, construct a key point set, and construct a multi-dimensional feature vector based on the key point set; wherein, the key points include key points of the user's hand and key points of the arm; The matching module 34 is used to match the member information corresponding to the multidimensional feature vector from a pre-built feature database; wherein, the feature database pre-records the mapping relationship between known multidimensional feature vectors and member information; The recommendation module 36 is used to generate recommendation information that matches the member information based on the member information.

[0058] Furthermore, this embodiment of the invention also provides a range hood equipped with a camera, and the controller of the range hood is equipped with the aforementioned cooking recognition device based on hand gestures.

[0059] The cooking recognition device and range hood based on hand behavior provided in this embodiment of the invention have the same technical features as the cooking recognition method based on hand behavior provided in the above embodiments, so they can also solve the same technical problems and achieve the same technical effects.

[0060] Furthermore, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0061] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0062] Furthermore, embodiments of the present invention also provide a schematic diagram of the structure of an electronic device, such as... Figure 4The diagram shows the structure of the electronic device, which includes a processor 41 and a memory 40. The memory 40 stores computer-executable instructions that can be executed by the processor 41, and the processor 41 executes the computer-executable instructions to implement the above-described method.

[0063] exist Figure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43, wherein the processor 41, the communication interface 43, and the memory 40 are connected via the bus 42.

[0064] The memory 40 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 42 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0065] Processor 41 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 41 or by software instructions. Processor 41 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor 41 reads the information in the memory and uses its hardware to complete the aforementioned method.

[0066] The computer program product of the cooking recognition method and device based on hand behavior provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0069] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0071] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cooking recognition method based on hand behavior, characterized in that, Applied to a range hood, wherein the range hood is equipped with a camera, the method includes: Obtain the video frame sequence captured by the camera, wherein the video frame sequence is video footage captured from a top-down perspective by a camera installed at the bottom of the range hood; Key points are extracted from the video frame sequence to construct a key point set, and a multidimensional feature vector is constructed based on the key point set; wherein, the key points include key points of the user's hand and key points of the arm; The member information corresponding to the multidimensional feature vector is matched from a pre-constructed feature database; wherein, the feature database pre-records the mapping relationship between known multidimensional feature vectors and member information; Recommendation information matching the member information is generated based on the member information.

2. The method according to claim 1, characterized in that, The steps of extracting key points from the video frame sequence and constructing a key point set include: Extract key points of the hand and arm to construct a set of key points for the arm linkage topology; wherein the key points of the hand include at least: wrist points and finger joint points; and the key points of the arm include at least: elbow joint points and forearm mid-axis points.

3. The method according to claim 2, characterized in that, The step of constructing a multidimensional feature vector based on the set of key points includes: The morphological features are determined based on the aforementioned key points of the hand and arm; Spatiotemporal feature extraction is performed on the continuous video frame sequence to extract spatiotemporal feature vectors representing hand action behavior features; Construct a multidimensional feature vector that includes the morphological features and the spatiotemporal feature vector.

4. The method according to claim 1, characterized in that, The steps for determining morphological features based on the aforementioned key points of the hand and arm include: Based on the aforementioned key points of the hand and arm, multidimensional morphological features representing the hand are determined; wherein, the multidimensional morphological features include at least: hand palm-finger ratio vector, palm vector, forearm length feature, and forearm shape width feature.

5. The method according to claim 1, characterized in that, The step of matching the member information corresponding to the multidimensional feature vector from a pre-built feature database includes: Search the feature database for known multidimensional feature vectors whose similarity to the multidimensional feature vector is higher than a preset similarity threshold; The member information corresponding to the known multidimensional feature vector is determined as the member information corresponding to the multidimensional feature vector.

6. The method according to claim 1, characterized in that, The method further includes: Monitoring data is obtained by monitoring the cooking process, wherein the monitoring data includes the user's morphological characteristics and the user's hand movement behavior characteristics during the cooking process; The monitoring data within a preset time period is statistically analyzed, and multi-dimensional feature vectors corresponding to multiple cooking subjects are constructed based on the monitoring data. Obtain the tag information of the cooking entity, wherein the tag information records the member information of the cooking entity; A feature database is constructed based on the label information and the multi-dimensional feature vector corresponding to each cooking subject.

7. The method according to claim 6, characterized in that, The method further includes: If no member information corresponding to the multidimensional feature vector is matched in the feature database, then the multidimensional feature vector is determined as a candidate feature vector; The number of times the candidate feature vectors are cooked is counted; If the number of cooking attempts reaches a preset threshold, a new cooking subject is generated, which is used to update the feature database.

8. The method according to claim 1, characterized in that, The step of generating recommendation information that matches the member information based on the member information includes: Search a pre-built information database for recommended information corresponding to the member information, wherein the recommended information includes at least one of the following: cooking protection advice, nutritional diet advice, and cooking utensils advice; The found recommendation information is sent to the corresponding user terminal so that the recommendation information can be presented through the user terminal.

9. The method according to claim 8, characterized in that, The method further includes: The cooking data corresponding to each member's information is statistically analyzed, including cooking frequency and cooking time, cooking mode, cooking intensity, and cooking habits. Based on the cooking data, corresponding recommendation information is determined; Generate the correspondence between the member information and the recommendation information, and store the correspondence in the information database.

10. A cooking recognition device based on hand gestures, characterized in that, A device for use in range hoods, wherein the range hood is equipped with a camera, the device comprising: The acquisition module is used to acquire a video frame sequence captured by the camera, wherein the video frame sequence is a video image captured from a top-down perspective by a camera installed at the bottom of the range hood; An extraction module is used to extract key points from the video frame sequence, construct a key point set, and construct a multi-dimensional feature vector based on the key point set; wherein, the key points include key points of the user's hands and key points of the arms; The matching module is used to match the member information corresponding to the multidimensional feature vector from a pre-built feature database; wherein, the feature database pre-records the mapping relationship between known multidimensional feature vectors and member information; The recommendation module is used to generate recommendation information that matches the member information based on the member information.

Images