Eating behavior detection method and device, readable storage medium and terminal equipment

Abstract

The invention belongs to the technical field of image processing, and particularly relates to an eating behavior detection method and device, a computer readable storage medium and terminal equipment. The method comprises the following steps: acquiring a to-be-detected target image frame; detecting whether a human face exists in the target image frame; if the face exists in the target image frame, detecting whether the target image frame is an eating key behavior frame; wherein the eating key behavior frame is an image frame in a food entry stage; and if the target image frame is the eating key behavior frame, determining that an eating behavior is detected in the target image frame. Through the method, whether the eating behavior is detected or not can be determined by detecting whether the target image frame is the eating key behavior frame or not, so that the detection process is greatly simplified, the influence of various feature factors is effectively reduced, the detection accuracy of the eating behavior can be improved, and the user experience can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method, device, computer-readable storage medium and terminal device for detecting eating behavior. Background Art

[0002] With the development of smart technology, more and more smart terminals are entering the user's field of vision. During the user's work or study, smart terminals can be used for supervision and companionship, and can detect eating behavior in time and remind users to improve their work or study efficiency.

[0003] However, in actual eating scenarios, there are various characteristic factors such as a wide variety of food, multiple eating methods, complex user postures towards the camera, and diverse eating behavior states, which lead to low accuracy of existing eating behavior detection methods and poor user experience. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an eating behavior detection method, apparatus, computer-readable storage medium and terminal device to solve the problem that the existing eating behavior detection method has low accuracy and leads to poor user experience.

[0005] A first aspect of an embodiment of the present application provides an eating behavior detection method, which may include:

[0006] Acquire the target image frame to be detected;

[0007] Detecting whether there is a human face in the target image frame;

[0008] If there is a face in the target image frame, detecting whether the target image frame is a key eating behavior frame; wherein the key eating behavior frame is an image frame at the food intake stage;

[0009] If the target image frame is an eating key behavior frame, it is determined that eating behavior is detected in the target image frame.

[0010] In a specific implementation of the first aspect, detecting whether there is a human face in the target image frame may include:

[0011] Acquire a reference face detection frame; wherein the reference face detection frame is a face detection frame in a previous frame of the target image frame;

[0012] Determine an extended detection frame corresponding to the reference face detection frame; wherein the range of the reference face detection frame is a subset of the range of the extended detection frame;

[0013] Extracting a target sub-image from the target image frame according to the extended detection frame;

[0014] Detect whether there is a face in the target sub-image.

[0015] In a specific implementation of the first aspect, determining the extended detection frame corresponding to the reference face detection frame may include:

[0016] Respectively determining a first center point, a first length, and a first height of the reference face detection frame;

[0017] Determine the second center point, the second length, and the second height of the extended detection frame according to the first center point, the first length, and the first height, respectively; wherein the second center point is consistent with the first center point, the second length is greater than the first length, and the second height is greater than the first height;

[0018] The extended detection frame is determined according to the second center point, the second length, and the second height.

[0019] In a specific implementation of the first aspect, determining the second center point, the second length, and the second height of the extended detection frame according to the first center point, the first length, and the first height, respectively, may include:

[0020] determining the second length according to the first length and a preset length extension ratio; wherein the length extension ratio is greater than 1;

[0021] The second height is determined according to the second height and a preset height expansion ratio; wherein the height expansion ratio is greater than 1.

[0022] In a specific implementation of the first aspect, detecting whether the target image frame is an eating key behavior frame includes:

[0023] Using a preset eating behavior detection model to detect whether the target image frame is an eating key behavior frame;

[0024] The training process of the eating behavior detection model may include:

[0025] Obtaining a preset eating behavior detection training sample set; wherein the eating behavior detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label;

[0026] The sample image of each training sample in the eating behavior detection training sample set is used as input, and the state label of each training sample in the eating behavior detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the eating behavior detection model.

[0027] In a specific implementation of the first aspect, detecting whether a human face exists in the target image frame includes:

[0028] Use a preset face detection model to detect whether there is a face in the target image frame;

[0029] The training process of the face detection model may include:

[0030] Obtaining a preset face detection training sample set; wherein the face detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding face detection frame;

[0031] The sample image of each training sample in the face detection training sample set is used as input, and the face detection frame of each training sample in the face detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the face detection model.

[0032] In a specific implementation of the first aspect, after determining that an eating behavior is detected in the target image frame, the method may further include:

[0033] Execute a prompt operation corresponding to the eating behavior.

[0034] A second aspect of the embodiments of the present application provides an eating behavior detection device, which may include:

[0035] An image frame acquisition module, used to acquire a target image frame to be detected;

[0036] A face detection module, used to detect whether there is a face in the target image frame;

[0037] A behavior detection module is used to detect whether the target image frame is a key behavior frame of eating if there is a face in the target image frame; wherein the key behavior frame of eating is an image frame of the food intake stage;

[0038] The behavior determination module is used to determine that eating behavior is detected in the target image frame if the target image frame is a key eating behavior frame.

[0039] In a specific implementation of the second aspect, the face detection module may include:

[0040] A detection frame acquisition unit, used to acquire a reference face detection frame; wherein the reference face detection frame is a face detection frame in a previous frame of the target image frame;

[0041] A detection frame determination unit, configured to determine an extended detection frame corresponding to the reference face detection frame; wherein the range of the reference face detection frame is a subset of the range of the extended detection frame;

[0042] A sub-image extraction unit, configured to extract a target sub-image from the target image frame according to the extended detection frame;

[0043] A face detection unit is used to detect whether there is a face in the target sub-image.

[0044] In a specific implementation of the second aspect, the detection frame determining unit may include:

[0045] A first determining subunit, used to respectively determine a first center point, a first length and a first height of the reference face detection frame;

[0046] A second determining subunit is used to determine a second center point, a second length, and a second height of the extended detection frame according to the first center point, the first length, and the first height, respectively; wherein the second center point is consistent with the first center point, the second length is greater than the first length, and the second height is greater than the first height;

[0047] A detection frame determination subunit is used to determine the extended detection frame according to the second center point, the second length and the second height.

[0048] In a specific implementation manner of the second aspect, the second determining subunit may include:

[0049] a second length determination subunit, configured to determine the second length according to the first length and a preset length extension ratio; wherein the length extension ratio is greater than 1;

[0050] The second height determination subunit is used to determine the second height according to the second height and a preset height expansion ratio; wherein the height expansion ratio is greater than 1.

[0051] In a specific implementation of the second aspect, the behavior detection module may include:

[0052] A model detection unit, configured to detect whether the target image frame is an eating key behavior frame using a preset eating behavior detection model;

[0053] The model detection unit may include:

[0054] A training set acquisition subunit is used to acquire a preset eating behavior detection training sample set; wherein the eating behavior detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label;

[0055] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the eating behavior detection training sample set as input and the state label of each training sample in the eating behavior detection training sample set as expected output to obtain the eating behavior detection model.

[0056] In a specific implementation of the second aspect, the face detection module may include:

[0057] A model detection unit, used to detect whether there is a face in the target image frame using a preset face detection model;

[0058] The model detection unit may include:

[0059] The training set acquisition subunit is used to acquire a preset face detection training sample set; wherein the face detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding face detection frame;

[0060] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the face detection training sample set as input and the face detection frame of each training sample in the face detection training sample set as expected output to obtain the face detection model.

[0061] In a specific implementation of the second aspect, the eating behavior detection device may further include:

[0062] The operation execution module is used to execute the prompt operation corresponding to the eating behavior.

[0063] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned eating behavior detection methods are implemented.

[0064] A fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned eating behavior detection methods when executing the computer program.

[0065] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any one of the above-mentioned eating behavior detection methods.

[0066] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a target image frame to be detected; detect whether there is a face in the target image frame; if there is a face in the target image frame, detect whether the target image frame is a key eating behavior frame; wherein the key eating behavior frame is an image frame of the food intake stage; if the target image frame is a key eating behavior frame, determine that eating behavior is detected in the target image frame. In the embodiments of the present application, it is possible to determine whether eating behavior is detected by detecting whether the target image frame is a key eating behavior frame, which greatly simplifies the detection process, effectively reduces the influence of multiple characteristic factors, can improve the detection accuracy of eating behavior, and helps improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 This is a flow chart of an embodiment of a method for detecting eating behavior in an embodiment of the present application;

[0069] Figure 2 A schematic flow chart of another face detection embodiment provided by the present application;

[0070] Figure 3 This is a schematic diagram of a face detection frame;

[0071] Figure 4 Schematic diagram of the target subgraph extraction process;

[0072] Figure 5 A schematic diagram of eating behavior;

[0073] Figure 6 This is a structural diagram of an embodiment of an eating behavior detection device in an embodiment of the present application;

[0074] Figure 7 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0077] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0078] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0079] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0080] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0081] With the development of smart technology, more and more smart terminals are entering the user's field of vision. During the user's work or study, smart terminals can be used for supervision and companionship, and can detect eating behavior in time and remind users to improve their work or study efficiency.

[0082] However, in actual eating scenarios, there are various characteristic factors such as a wide variety of food, multiple eating methods, complex user postures towards the camera, and diverse eating behavior states, which lead to low accuracy of existing eating behavior detection methods and poor user experience.

[0083] In view of this, the embodiments of the present application provide an eating behavior detection method, device, computer-readable storage medium and terminal device to solve the problem that the existing eating behavior detection method has low accuracy, resulting in poor user experience. In the embodiments of the present application, whether eating behavior is detected can be determined by detecting whether the target image frame is a key eating behavior frame, which greatly simplifies the detection process, effectively reduces the influence of various characteristic factors, can improve the detection accuracy of eating behavior, and help improve user experience.

[0084] It should be noted that the executor of the method of the present application is a terminal device, specifically, it can be a common computing device such as a learning machine, a tablet computer, a smart desk lamp, a desktop computer, a notebook, a PDA, or other computing devices.

[0085] See also Figure 1 In the embodiment of the present application, an embodiment of a method for detecting eating behavior may include:

[0086] Step S101: Acquire a target image frame to be detected.

[0087] In an embodiment of the present application, one or more image frames may be collected and formed into an image sequence, and then the image frames in the image sequence may be analyzed frame by frame to detect whether there is any eating behavior.

[0088] For the convenience of description, the method of the present application will be described below by taking any image frame in the image sequence (referred to as the target image frame) as an example.

[0089] It is understandable that a pre-stored image sequence can be obtained from a preset memory module or server, and any frame in the image sequence can be used as the target image frame; or, a camera acquisition device pre-installed on the terminal device can be used for real-time acquisition to obtain an image sequence, and any frame therein can be used as the target image frame.

[0090] Step S102: Detect whether there is a face in the target image frame.

[0091] In an embodiment of the present application, a preset face detection model can be used to detect whether there is a face in the target image frame, wherein the face detection model can be a pre-trained artificial intelligence model for face detection.

[0092] It can be understood that the artificial intelligence models used in the embodiments of the present application can all be artificial intelligence models commonly used in the prior art for image detection, including but not limited to convolutional neural network (CNN), recurrent neural network (RNN), YOLO, Fast R-CNN, Faster R-CNN and other common artificial intelligence models in the prior art.

[0093] It is understandable that before using the face detection model to detect whether there is a face in the target image frame, the face detection model can be trained. Specifically, a preset face detection training sample set can be obtained, each of which can include a sample image and a corresponding face detection frame; then, the sample image of each training sample in the face detection training sample set is used as input, and the face detection frame of each training sample in the face detection training sample set is used as the expected output, and the initial artificial intelligence model is trained to obtain the face detection model.

[0094] It should be noted that in order to make the detection results of the face detection model more robust, the face detection training sample set may include a preset number of positive sample images and negative sample images under various application scenarios, where the positive sample images are sample images with faces, and the negative sample images are sample images without faces.

[0095] In a possible embodiment, when training a face detection model, the face detection training sample set may be divided into a training set and a test set according to a preset ratio, and the training set may be used for model training, and the test set may be used for model testing.

[0096] In another possible embodiment, the face detection model can also be updated according to the detection result of the face detection model to improve the detection accuracy of the face detection model. For example, if a face is detected in the target image frame, the target image frame can be used as a positive sample image in the face detection training sample set to update the face detection model; if no face is detected in the target image frame, the target image frame can be used as a negative sample image in the face detection training sample set to update the face detection model.

[0097] It is understandable that, since the acquisition time interval between two adjacent image frames in the image sequence is short, the movement range of the object or person between the two adjacent image frames may be small. Therefore, in another embodiment provided by the present application, the range of face detection in the target image frame can be determined according to the face detection frame detected in the previous frame in the image sequence to improve the efficiency of face detection. Specifically, this embodiment may include the following: Figure 2 Steps shown:

[0098] Step S201: Obtain a reference face detection frame.

[0099] It can be understood that the method of the present application can analyze the image frames in the image sequence frame by frame. Therefore, the image frames to be detected can be detected by the face detection model to detect whether there is a face in the image frame. If there is a face in the image frame, the face detection frame corresponding to the image frame can be output, wherein the face detection frame marks the position of the face in the image frame.

[0100] It is understandable that when there is a face in the previous frame, if there is also a face in the target image frame, the face position in the target image frame is likely to be the same or similar to the face position in the previous frame. Therefore, if there is a face in the previous frame, the face position in the previous frame can be used as a reference for the face position in the target image frame, and the face detection frame in the previous frame can be used as a reference face detection frame for the target image frame.

[0101] Step S202: Determine an extended detection frame corresponding to the reference face detection frame.

[0102] In this embodiment, an extended detection frame corresponding to the reference face detection frame may also be determined, and it may be considered that the extended detection frame marks the possible face position in the target image frame, wherein the range of the reference face detection frame is a subset of the range of the extended detection frame.

[0103] Specifically, the second center point, the second length, and the second height of the extended detection frame can be determined according to the first center point, the first length, and the first height of the reference face detection frame. Figure 3 , the second center point can be consistent with the first center point, the second length can be determined according to the first length and the preset length expansion ratio; the second height can be determined according to the first height and the preset height expansion ratio. Among them, the length expansion ratio and the height expansion ratio can be set to a value greater than 1 according to actual needs, and this embodiment does not specifically limit this. For example, the length expansion ratio can be set to 2, then the first length can be expanded to twice the original length to obtain the second length; the height expansion ratio can also be set to 1.5, then the first height can be expanded to 1.5 times the original height to obtain the second height.

[0104] Step S203: extracting a target sub-image from the target image frame according to the extended detection frame.

[0105] In this embodiment, the target sub-image can be extracted from the target image frame according to the extended detection frame. Specifically, Figure 4 As shown in (a), the target image frame can be captured according to the extended detection frame to obtain Figure 4(b) shows a target sub-image with the same size as the extended detection frame, and it can be considered that the face position is likely to be in the target sub-image.

[0106] Step S204: Detect whether there is a face in the target sub-image.

[0107] In this embodiment, a face detection model can be used to detect whether a face exists in the target sub-image. After obtaining the target sub-image, the target sub-image can be used as an input of the face detection model to detect whether a face exists in the target sub-image. If a face exists in the target sub-image, it can be considered that a face exists in the target image frame; if a face does not exist in the target sub-image, it can be considered that a face does not exist in the target image frame.

[0108] It can be understood that since the size of the target sub-image is smaller than the size of the target image frame, using the target sub-image instead of the target image frame as the input of the face detection model helps to improve the efficiency of face detection.

[0109] In the embodiment of the present application, if a face is detected in the target image frame, a face detection frame can be output and step S103 can be executed; if no face is detected in the target image frame, step S105 can be executed.

[0110] Step S103: Detect whether the target image frame is a key behavior frame of eating.

[0111] In an embodiment of the present application, a preset eating behavior detection model can be used to detect whether the target image frame is a key eating behavior frame, wherein the key eating behavior frame can be an image frame of the food intake stage; the eating behavior detection model can be a pre-trained artificial intelligence model for eating behavior detection.

[0112] It is understandable that the whole process of eating behavior may include multiple stages, and here, the first stage may be a preparation stage, such as Figure 5 (a) shows the stage of preparatory actions such as picking up food, tearing packaging bags and peeling skin; the second stage can be the food intake stage, such as Figure 5 (b), i.e., the stage of the eating action of picking up food and putting it into the mouth; the third stage may be the chewing stage, such as Figure 5 (c) shows the stage of chewing action accompanied by mouth movement after food enters the mouth. The preparatory actions in the first stage are diverse, and the chewing action in the third stage is easily confused with talking, grinding teeth, blowing bubbles, etc. Therefore, in the embodiment of the present application, the second stage can be defined as the key behavior of eating, specifically defined as the food being placed in the mouth and visible, and the image frame containing the second stage can be defined as the key behavior frame of eating.

[0113] It is understandable that before using the eating behavior detection model to detect whether the target image frame is a key eating behavior frame, the eating behavior detection model can be trained. Specifically, a preset eating behavior detection training sample set can be obtained, each of which can include a sample image and a corresponding state label, wherein the state label can include a key frame state label and a non-key frame state label, the key frame state label can indicate that the target image frame is a key eating behavior frame, and the non-eating state label can indicate that the target image frame is not a key eating behavior frame; then, the sample image of each training sample in the eating behavior detection training sample set is used as input, and the state label of each training sample in the eating behavior detection training sample set is used as the expected output, and the initial artificial intelligence model is trained to obtain the eating behavior detection model.

[0114] It should be noted that in order to make the detection results of the eating detection model more robust, the eating detection training sample set may include a preset number of positive sample images and negative sample images under various application scenarios, wherein the positive sample images are sample images of key eating behavior frames (food entry stage of eating behavior), and the negative sample images are sample images that are not key eating behavior frames.

[0115] In a possible embodiment, when training the eating detection model, the eating detection training sample set can be divided into a training set and a test set according to a preset ratio, and the training set is used for model training, and the test set is used for model testing.

[0116] In the embodiment of the present application, if the target image frame is determined to be a key eating behavior frame, step S104 may be executed; if the target image frame is not a key eating behavior frame, step S105 may be executed.

[0117] It is understandable that the eating detection model can also be updated according to the detection results of the eating detection model to improve the detection accuracy of the eating detection model. For example, if the target image frame is a key eating behavior frame, the target image frame can be used as a positive sample image in the eating detection training sample set to update the eating detection model; if the target image frame is not a key eating behavior frame, the target image frame can be used as a negative sample image in the eating detection training sample set to update the eating detection model.

[0118] Step S104: Determine whether eating behavior is detected in the target image frame.

[0119] In the embodiment of the present application, if the target image frame is a key behavior frame of eating, it can be determined that the eating behavior is detected in the target image frame.

[0120] It can be understood that when it is determined that eating behavior is detected in the target image frame, operation prompts corresponding to the eating behavior can be executed, which may include but are not limited to one or more prompt operations such as emitting prompt sound effects or prompt voices and displaying prompt texts to remind the user to concentrate on work.

[0121] In a possible embodiment, when the prompt operation is performed, an accompanying auxiliary perception operation may also be performed. For example, a preset light may be emitted to perform an auxiliary prompt. For another example, a vibration motor may be controlled to vibrate at a preset frequency to perform an auxiliary prompt.

[0122] In another possible embodiment, the user can set the prompt operation on the client that establishes a communication connection with the terminal device. For example, the user can choose to use only a preset light as a prompt operation. For another example, the user can choose to emit a prompt sound effect and display a prompt text as a prompt operation.

[0123] Step S105: Determine whether eating behavior is not detected in the target image frame.

[0124] In the embodiment of the present application, if there is no human face in the target image frame, or the target image frame is not a key eating behavior frame, it can be determined that no eating behavior is detected in the target image frame.

[0125] It can be understood that, in a possible embodiment, after completing the detection of the target image frame, the above method can be repeatedly executed on the next image frame of the target image frame to achieve continuous detection of the user's eating behavior.

[0126] In summary, the embodiment of the present application obtains a target image frame to be detected; detects whether there is a face in the target image frame; if there is a face in the target image frame, detects whether the target image frame is a key eating behavior frame; wherein the key eating behavior frame is an image frame of the food intake stage; if the target image frame is a key eating behavior frame, it is determined that eating behavior is detected in the target image frame. In the embodiment of the present application, whether eating behavior is detected can be determined by detecting whether the target image frame is a key eating behavior frame, which greatly simplifies the detection process, effectively reduces the influence of various characteristic factors, can improve the detection accuracy of eating behavior, and helps to improve user experience.

[0127] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] Corresponding to the eating behavior detection method described in the above embodiment, Figure 6A structural diagram of an embodiment of an eating behavior detection device provided in an embodiment of the present application is shown.

[0129] In this embodiment, an eating behavior detection device may include:

[0130] The image frame acquisition module 601 is used to acquire the target image frame to be detected;

[0131] A face detection module 602 is used to detect whether there is a face in the target image frame;

[0132] The behavior detection module 603 is used to detect whether the target image frame is a key behavior frame of eating if there is a face in the target image frame; wherein the key behavior frame of eating is an image frame of the food intake stage;

[0133] The behavior determination module 604 is configured to determine that eating behavior is detected in the target image frame if the target image frame is a key eating behavior frame.

[0134] In a specific implementation of the embodiment of the present application, the face detection module may include:

[0135] A detection frame acquisition unit, used to acquire a reference face detection frame; wherein the reference face detection frame is a face detection frame in a previous frame of the target image frame;

[0136] A detection frame determination unit, configured to determine an extended detection frame corresponding to the reference face detection frame; wherein the range of the reference face detection frame is a subset of the range of the extended detection frame;

[0137] A sub-image extraction unit, configured to extract a target sub-image from the target image frame according to the extended detection frame;

[0138] A face detection unit is used to detect whether there is a face in the target sub-image.

[0139] In a specific implementation of the embodiment of the present application, the detection frame determining unit may include:

[0140] A first determining subunit, used to respectively determine a first center point, a first length and a first height of the reference face detection frame;

[0141] A second determining subunit is used to determine a second center point, a second length, and a second height of the extended detection frame according to the first center point, the first length, and the first height, respectively; wherein the second center point is consistent with the first center point, the second length is greater than the first length, and the second height is greater than the first height;

[0142] A detection frame determination subunit is used to determine the extended detection frame according to the second center point, the second length and the second height.

[0143] In a specific implementation of the embodiment of the present application, the second determining subunit may include:

[0144] a second length determination subunit, configured to determine the second length according to the first length and a preset length extension ratio; wherein the length extension ratio is greater than 1;

[0145] The second height determination subunit is used to determine the second height according to the second height and a preset height expansion ratio; wherein the height expansion ratio is greater than 1.

[0146] In a specific implementation of the embodiment of the present application, the behavior detection module may include:

[0147] A model detection unit, configured to detect whether the target image frame is an eating key behavior frame using a preset eating behavior detection model;

[0148] The model detection unit may include:

[0149] A training set acquisition subunit is used to acquire a preset eating behavior detection training sample set; wherein the eating behavior detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label;

[0150] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the eating behavior detection training sample set as input and the state label of each training sample in the eating behavior detection training sample set as expected output to obtain the eating behavior detection model.

[0151] In a specific implementation of the embodiment of the present application, the face detection module may include:

[0152] A model detection unit, used to detect whether there is a face in the target image frame using a preset face detection model;

[0153] The model detection unit may include:

[0154] The training set acquisition subunit is used to acquire a preset face detection training sample set; wherein the face detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding face detection frame;

[0155] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the face detection training sample set as input and the face detection frame of each training sample in the face detection training sample set as expected output to obtain the face detection model.

[0156] In a specific implementation of the embodiment of the present application, the eating behavior detection device may further include:

[0157] The operation execution module is used to execute the prompt operation corresponding to the eating behavior.

[0158] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] Figure 7 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0161] like Figure 7 As shown, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned eating behavior detection method embodiments are implemented, for example Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 6 The functions of modules 601 to 604 are shown.

[0162] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 72 in the terminal device 7.

[0163] The terminal device 7 can be a computing device such as a learning machine, a tablet computer, a smart desk lamp, a desktop computer, a notebook, a PDA, etc. It can be understood by those skilled in the art that Figure 7It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 7 may also include input and output devices, network access devices, buses, etc.

[0164] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0165] The memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the terminal device 7. The memory 71 is used to store the computer program and other programs and data required by the terminal device 7. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0166] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0167] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0168] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0169] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0173] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting eating behavior, characterized in that: include: Acquire the target image frame to be detected; Detecting whether there is a human face in the target image frame; If there is a face in the target image frame, detecting whether the target image frame is a key eating behavior frame; wherein the key eating behavior frame is an image frame at the food intake stage; If the target image frame is an eating key behavior frame, it is determined that eating behavior is detected in the target image frame.

2. The eating behavior detection method according to claim 1, characterized in that: The detecting whether there is a human face in the target image frame comprises: Acquire a reference face detection frame; wherein the reference face detection frame is a face detection frame in a previous frame of the target image frame; Determine an extended detection frame corresponding to the reference face detection frame; wherein the range of the reference face detection frame is a subset of the range of the extended detection frame; Extracting a target sub-image from the target image frame according to the extended detection frame; Detect whether there is a face in the target sub-image.

3. The eating behavior detection method according to claim 2, characterized in that: The determining of the extended detection frame corresponding to the reference face detection frame includes: Respectively determining a first center point, a first length, and a first height of the reference face detection frame; Determine the second center point, the second length, and the second height of the extended detection frame according to the first center point, the first length, and the first height, respectively; wherein the second center point is consistent with the first center point, the second length is greater than the first length, and the second height is greater than the first height; The extended detection frame is determined according to the second center point, the second length, and the second height.

4. The eating behavior detection method according to claim 3, characterized in that: The determining the second center point, the second length, and the second height of the extended detection frame respectively according to the first center point, the first length, and the first height comprises: determining the second length according to the first length and a preset length extension ratio; wherein the length extension ratio is greater than 1; The second height is determined according to the second height and a preset height expansion ratio; wherein the height expansion ratio is greater than 1.

5. The eating behavior detection method according to claim 1, characterized in that: The detecting whether the target image frame is a key eating behavior frame includes: Using a preset eating behavior detection model to detect whether the target image frame is an eating key behavior frame; The training process of the eating behavior detection model includes: Obtaining a preset eating behavior detection training sample set; wherein the eating behavior detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label; The sample image of each training sample in the eating behavior detection training sample set is used as input, and the state label of each training sample in the eating behavior detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the eating behavior detection model.

6. The eating behavior detection method according to claim 1, characterized in that: The detecting whether there is a human face in the target image frame comprises: Use a preset face detection model to detect whether there is a face in the target image frame; The training process of the face detection model includes: Obtaining a preset face detection training sample set; wherein the face detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding face detection frame; The sample image of each training sample in the face detection training sample set is used as input, and the face detection frame of each training sample in the face detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the face detection model.

7. The eating behavior detection method according to any one of claims 1 to 6, characterized in that: After determining that eating behavior is detected in the target image frame, the method further includes: Execute a prompt operation corresponding to the eating behavior.

8. An eating behavior detection device, characterized in that: include: An image frame acquisition module, used to acquire a target image frame to be detected; A face detection module, used to detect whether there is a face in the target image frame; A behavior detection module is used to detect whether the target image frame is a key behavior frame of eating if there is a face in the target image frame; wherein the key behavior frame of eating is an image frame of the food intake stage; The behavior determination module is used to determine that eating behavior is detected in the target image frame if the target image frame is a key eating behavior frame.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the eating behavior detection method according to any one of claims 1 to 7 are implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the eating behavior detection method according to any one of claims 1 to 7 are implemented.

Images