Method for Detecting Calorie Intake Information, Electronic Device, and System

Abstract

This application is applicable to the field of terminal technologies, and provides a method for detecting calorie intake information, an electronic device, and a system. This method periodically collects a first image through a wearable device and performs object detection on the first image; when it is detected that the first image contains food to be put into the mouth, the wearable device collects audio data and the motion data of the user's head; determines the number of times the food to be put into the mouth is swallowed and the magnitude of the swallowing sound for each swallowing according to the audio data, and determines the head posture according to the motion data; determines the actual intake of the food to be put into the mouth based on the user's single default swallowing amount, head posture, the number of times the food to be put into the mouth is swallowed, and the magnitude of the swallowing sound for each swallowing; determines the calorie intake information according to the actual intakes of all the foods to be put into the mouth detected and the calories corresponding to the unit portion, thereby being able to improve the accuracy of calorie intake information detection and simplify the manual operation process in the calorie intake information detection process.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a method for detecting calorie intake information, an electronic device, and a system. Background Art

[0002] Currently, with the continuous enhancement of people's health awareness, personal weight management and health management have received increasing attention. Reasonably controlling the calorie intake of each meal is the key to weight management and maintaining health. For this reason, more and more people begin to calculate and record the calorie intake of each meal through calorie management APPs installed on electronic devices such as mobile phones.

[0003] When calculating the calorie intake of each meal for a user, a calorie management APP provided by the related art requires the user to manually input the portions (such as weight, volume, or quantity, etc.) of various foods consumed in each meal, and then calculate the calorie intake of each meal for the user by combining the calories corresponding to the unit portions of various foods stored in the food ingredient library. The operation process is relatively cumbersome, resulting in a poor user experience. Another calorie management APP provided by the related art can automatically recognize the total calories of the food to be consumed in the image based on the image of the food to be consumed taken by the user, and use it as the calorie intake of the user for that meal. However, in actual applications, there may be a situation where some of the food in the image is not consumed by the user, resulting in inaccurate calculation of the calorie intake. Summary of the Invention

[0004] The embodiments of this application provide a method for detecting calorie intake information, an electronic device, and a system, which can not only accurately detect the calorie intake information of each meal for the user, but also simplify the manual operation process in the detection process of calorie intake information, giving the user a better experience.

[0005] In a first aspect, the embodiments of this application provide a method for detecting calorie intake information, including: periodically collecting a first image through a wearable device worn on the head, and performing object detection on the first image; when it is detected that the first image contains food to be put into the mouth, collecting audio data and the movement data of the user's head through the wearable device; the audio data at least includes the swallowing sound of the user swallowing the food to be put into the mouth; determining the number of times the food to be put into the mouth is swallowed and the magnitude of the swallowing sound for each swallowing based on the audio data, and determining the head posture of the user when consuming the food to be put into the mouth based on the movement data; determining the actual intake amount of the food to be put into the mouth based on the user's single default swallowing amount, head posture, number of times swallowed, and the magnitude of the swallowing sound for each swallowing; determining the calorie intake information of the user according to the actual intake amounts of all the foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portions; the calorie intake information includes a first total calorie intake.

[0006] Wearable devices worn on the head may include smart glasses, smart earrings, smart necklaces, and smart headphones.

[0007] The food to be eaten refers to the food that has been picked up by the user through tableware (e.g., chopsticks, spoon or fork, etc.) or hands and is about to be put into the mouth. The food to be eaten is usually a small part of all the food to be ingested. Based on this, if the first image contains food in the tableware or in the hands of the person, it can be determined that the first image contains the food to be eaten.

[0008] When a user is eating, different head postures will result in different amounts of food being swallowed each time. Specifically, the amount of food swallowed each time when the user is tilting his head back is usually greater than the amount of food swallowed each time when the user is not tilting his head back, and the greater the angle at which the user tilts his head back, the greater the amount of food swallowed each time.

[0009] Exemplarily, the user's head posture may be described by the pitch angle of the user's head.

[0010] For example, the number of target peaks appearing in the audio data corresponding to the swallowing sound can be determined as the number of times the food to be ingested is swallowed. The target peak can refer to a peak whose corresponding amplitude is greater than a preset decibel threshold.

[0011] For example, the swallowing sound volume of each swallow of the food to be ingested can be determined according to the amplitude corresponding to each target peak value.

[0012] The user's single default swallowing volume can be based on the various body parameters set by the user in the calorie management app, and usually queries the correspondence between the pre-stored various body parameters and the single default swallowing volume. Among them, the body parameters may include gender, age, height, and weight. The single default swallowing volume of users with different body parameters is different.

[0013] The detection method for calorie intake information provided by the embodiments of the present application periodically acquires a first image by using a wearable device worn on the head, and performs object detection on the first image. When it is detected that the first image contains food to be put into the mouth, the wearable device acquires audio data and the movement data of the user's head, so as to realize the automatic monitoring of the sound and head posture when the user intakes the food to be put into the mouth; by determining the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time according to the audio data, and determining the head posture of the user when the user intakes the food to be put into the mouth according to the movement data of the user's head, and comprehensively considering the user's single default swallowing amount, the head posture of the user when the user intakes the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time, the actual intake amount of the food to be put into the mouth is determined, achieving the accurate detection of the user's actual food intake; thus, the calorie intake information of the user can be accurately calculated according to the actual intake amounts of all the foods to be put into the mouth detected respectively in multiple discontinuous time periods and the calories corresponding to the unit portions. Since the wearable device can automatically monitor the entire eating process of the user and automatically detect the calorie intake information of the user, the manual operation process in the calorie intake information detection process is simplified, providing a better experience for the user.

[0014] In an optional implementation manner of the first aspect, determining the actual intake amount of the food to be put into the mouth based on the user's single default swallowing amount, head posture, number of times swallowed, and the size of the swallowing sound each time includes: determining a first swallowing amount coefficient corresponding to each swallowing according to the size of the swallowing sound each time; determining the texture type of the food to be put into the mouth; the texture type includes liquid food and non-liquid food; determining a second swallowing amount coefficient corresponding to each swallowing according to the texture type and the head posture; and determining the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the number of times swallowed, and the first swallowing amount coefficient and the second swallowing amount coefficient corresponding to each swallowing.

[0015] Wherein, the first swallowing amount coefficient is an adjustment factor for adjusting the actual swallowing amount determined based on the size of the swallowing sound. The first swallowing amount coefficient corresponding to each swallowing of the food to be put into the mouth can be determined by querying the pre-stored corresponding relationship between the size of the swallowing sound and the first swallowing amount coefficient according to the size of the swallowing sound each time the food to be put into the mouth is swallowed.

[0016] The second swallowing amount coefficient is an adjustment factor for adjusting the actual swallowing amount determined based on the head posture. The second swallowing amount coefficient corresponding to each swallowing of the food to be put into the mouth can be determined by querying the pre-stored corresponding relationship between the pitch angle of the head corresponding to the texture type of the food to be put into the mouth and the second swallowing amount coefficient according to the pitch angle of the head when the user intakes the food to be put into the mouth.

[0017] The detection method of calorie intake information provided by this implementation method determines the first swallowing volume coefficient corresponding to each swallowing of the food to be put into the mouth according to the size of the swallowing sound of the food to be put into the mouth each time it is swallowed; and determines the second swallowing volume coefficient corresponding to each swallowing of the food to be put into the mouth according to the head posture of the user when ingesting the food to be put into the mouth and the texture type of the food to be put into the mouth. Thus, based on the first swallowing volume coefficient, the second swallowing volume coefficient, and the user's single default swallowing volume, the actual swallowing volume of the user each time of swallowing food can be accurately determined, which is beneficial to improving the accuracy of calorie intake information detection.

[0018] In an optional implementation manner of the first aspect, determining the actual intake amount of the food to be put into the mouth according to the user's single default swallowing volume, the number of swallows, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing includes: determining the actual intake amount of the food to be put into the mouth according to the following formula:

[0019] ;

[0020] wherein, TY sj is the actual intake amount of the food to be put into the mouth, TY mr is the user's single default swallowing volume, n is the number of swallows, α i is the first swallowing volume coefficient corresponding to the i th swallowing of the food to be put into the mouth, β i is the second swallowing volume coefficient corresponding to the i th swallowing of the food to be put into the mouth.

[0021] In an optional implementation manner of the first aspect, determining the texture type of the food to be put into the mouth includes: determining the texture type of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by target detection and the confidence level corresponding to the first candidate name, and / or according to the audio data.

[0022] Exemplarily, the true name of the food to be put into the mouth can be determined according to the first candidate name of the food to be put into the mouth obtained by target detection of the first image and the confidence level corresponding to the first candidate name; and the texture type of the food to be put into the mouth can be determined according to the true name of the food to be put into the mouth.

[0023] Exemplarily, the texture type of the food to be put into the mouth can be determined according to the audio data collected by the smart glasses. For example, when the audio data does not include chewing sounds, the mobile phone can determine that the food to be put into the mouth is a liquid food; when the audio data includes chewing sounds, the mobile phone can determine that the food to be put into the mouth is a non-liquid food.

[0024] Exemplarily, the texture type of the food to be put into the mouth can be determined by combining the audio data, the first candidate name of the food to be put into the mouth obtained by performing object detection on the first image, and the confidence level corresponding to the first candidate name. For example, first, the real name of the food to be put into the mouth can be determined according to the first candidate name of the food to be put into the mouth obtained by performing object detection on the first image and the confidence level corresponding to the first candidate name, and then the texture type of the food to be put into the mouth can be determined according to the audio data and the real name of the food to be put into the mouth. Exemplarily, when the real name of the food to be put into the mouth belongs to the name corresponding to liquid food and the audio data does not include chewing sounds, the mobile phone can determine that the food to be put into the mouth is liquid food; when the real name of the food to be put into the mouth does not belong to liquid food, or the audio data includes chewing sounds, the mobile phone can determine that the food to be put into the mouth is non-liquid food.

[0025] The detection method of calorie intake information provided by this implementation manner can improve the accuracy of determining the texture type of the food to be put into the mouth by comprehensively determining the texture type of the food to be put into the mouth based on image detection and audio detection.

[0026] In an optional implementation manner of the first aspect, before determining the user's calorie intake information according to the actual intake amounts of all the foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portion, it further includes: determining the real name of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by object detection and the confidence level corresponding to the first candidate name; determining the calories corresponding to the unit portion of the food to be put into the mouth according to the real name of the food to be put into the mouth.

[0027] In an optional implementation manner of the first aspect, determining the user's calorie intake information according to the actual intake amounts of all the foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portion includes: for the foods to be put into the mouth detected in each time period, when there are multiple types of the foods to be put into the mouth, determining the volume proportion of each type of the foods to be put into the mouth according to the outlines of various foods to be put into the mouth obtained by object detection; calculating the calorie intake amount corresponding to the foods to be put into the mouth in the first image according to the actual intake amount, the volume proportion of each type of the foods to be put into the mouth, and the calories corresponding to the unit portion; determining the sum of the calorie intake amounts corresponding to all the foods to be put into the mouth detected in multiple discontinuous time periods as the user's total first calorie intake.

[0028] Exemplarily, the first image can be a depth image. Based on this, the area of the region where each food to be put into the mouth is located can be determined respectively according to the contours of various foods to be put into the mouth obtained by performing object detection on the first image; the volume of each food to be put into the mouth can be determined respectively according to the area of the region where each food to be put into the mouth is located and the depth information; the ratio of the volume of each food to be put into the mouth to the sum of the volumes of all foods to be put into the mouth can be determined respectively as the volume proportion of each food to be put into the mouth.

[0029] Exemplarily, the calorie intake corresponding to the food to be put into the mouth can be calculated according to the actual intake amount of the food to be put into the mouth, the volume proportion of each food to be put into the mouth, and the calorie corresponding to the unit portion by the following formula:

[0030] ;

[0031] where, RL is the calorie intake corresponding to the food to be put into the mouth, TY sj is the actual intake amount of the food to be put into the mouth, m is the number of types of foods to be put into the mouth, μ i is the j th volume proportion of the food to be put into the mouth, Δ RL is the j th calorie corresponding to the unit portion of the food to be put into the mouth.

[0032] For the method for detecting calorie intake information provided in this implementation manner, when the first image contains multiple foods to be put into the mouth, by calculating the calorie intake corresponding to the foods to be put into the mouth in the first image according to the actual intake amount of all foods to be put into the mouth in the first image, the volume proportion of each food to be put into the mouth, and the calorie corresponding to the unit portion, the accuracy of determining the calorie intake corresponding to the foods to be put into the mouth can be improved, and further the accuracy of detecting calorie intake information can be improved.

[0033] In an optional implementation manner of the first aspect, the audio data further includes the chewing sound of the user chewing the food to be put into the mouth; correspondingly, before determining the user's calorie intake information according to the actual intake amount of all foods to be put into the mouth detected respectively in multiple discontinuous time periods and the calorie corresponding to the unit portion, it further includes: determining the second candidate name of the food to be put into the mouth and the confidence corresponding to the second candidate name according to the audio characteristics of the chewing sound included in the audio data; determining the real name of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by object detection, the confidence corresponding to the first candidate name, the second candidate name, and the confidence corresponding to the second candidate name; determining the calorie corresponding to the unit portion of the food to be put into the mouth according to the real name of the food to be put into the mouth.

[0034] The detection method of calorie intake information provided by the embodiments of the present application can comprehensively determine the name of the food to be put into the mouth based on image detection and audio detection, which can improve the accuracy of food name recognition and further improve the accuracy of calorie intake information detection.

[0035] In an optional implementation manner of the first aspect, determining the real name of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by target detection, the confidence level corresponding to the first candidate name, the second candidate name, and the confidence level corresponding to the second candidate name includes: when the first candidate name is the same as the second candidate name, and the confidence levels corresponding to both the first candidate name and the second candidate name are greater than or equal to the preset confidence threshold, determining the first candidate name or the second candidate name as the real name of the food to be put into the mouth; when the first candidate name is different from the second candidate name, the confidence level corresponding to the first candidate name is greater than or equal to the preset confidence threshold, and the confidence level corresponding to the second candidate name is less than the preset confidence threshold, determining the first candidate name as the real name of the food to be put into the mouth; when the first candidate name is different from the second candidate name, the confidence level corresponding to the first candidate name is less than the preset confidence threshold, and the confidence level corresponding to the second candidate name is greater than or equal to the preset confidence threshold, determining the second candidate name as the real name of the food to be put into the mouth.

[0036] In an optional implementation manner of the first aspect, the detection method of calorie intake information further includes: obtaining a second image including all the foods to be ingested and determining the total calories of all the foods to be ingested in the second image; obtaining a third image including the remaining foods and determining the total calories of the remaining foods in the third image; determining the difference between the total calories of all the foods to be ingested and the total calories of the remaining foods as the total amount of the second calorie intake of the user; adjusting the default single swallowing amount of the user based on the number of times of swallowing of various foods to be put into the mouth detected respectively in multiple discontinuous time periods, the first swallowing amount coefficient and the second swallowing amount coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the total amount of the second calorie intake.

[0037] The detection method of calorie intake information provided by this implementation manner can adjust the default single swallowing amount of the user based on the total amount of the second calorie intake of the user obtained by image detection, which can improve the matching degree between the default single swallowing amount of the user and the actual swallowing situation of the user, and is beneficial to improving the accuracy of calorie intake information detection.

[0038] In an alternative implementation of the first aspect, determining the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time it is swallowed based on the audio data includes: extracting the audio data corresponding to the swallowing sound from the audio data; determining the number of target peaks that appear in the audio data corresponding to the swallowing sound as the number of times swallowed; and respectively determining the size of the swallowing sound each time it is swallowed according to the amplitude size corresponding to each target peak.

[0039] In a second aspect, an embodiment of the present application provides an electronic device, and the electronic device is a wearable device for wearing on the head; the electronic device is used to execute the detection method of calorie intake information in any implementation manner of the above first aspect.

[0040] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium includes instructions. When the instructions run on the electronic device, the electronic device is caused to execute the detection method of calorie intake information in any implementation manner of the above first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-executable program product. When the computer-executable program product runs on the electronic device, the electronic device is caused to execute the detection method of calorie intake information in any implementation manner of the above first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a chip system, and the chip system is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions to cause the electronic device to execute the detection method of calorie intake information in any implementation manner of the above first aspect.

[0043] In a sixth aspect, an embodiment of the present application provides a detection system for calorie intake information, including a terminal device and at least one wearable device;

[0044] The wearable device is used to periodically collect a first image and send the first image to the terminal device;

[0045] The terminal device is used to perform target detection on the first image, and in the case of detecting that the first image contains food to be put into the mouth, send an eating monitoring instruction to the wearable device;

[0046] The wearable device is further used to respond to the eating monitoring instruction, start collecting audio data and movement data of the user's head, and send the audio data and the movement data to the terminal device;

[0047] The terminal device is further configured to determine the number of times the to-be-entered food is swallowed and the magnitude of the swallowing sound for each swallowing based on the audio data, and determine the head posture of the user when ingesting the to-be-entered food based on the motion data;

[0048] The terminal device is further configured to determine the actual intake amount of the to-be-entered food based on the user's single default swallowing amount, the head posture, the number of times of swallowing, and the magnitude of the swallowing sound for each swallowing;

[0049] The terminal device is further configured to determine the user's calorie intake information based on the actual intake amounts of all the to-be-entered foods detected in multiple discontinuous time periods and the calories corresponding to the unit portions.

[0050] It can be understood that the beneficial effects of the above second aspect to the sixth aspect can be referred to the relevant descriptions in the above first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of a user interface involved in calculating calorie intake by a calorie management APP provided by the related art;

[0052] Figure 2 Schematic diagram of the structure of a calorie intake information detection system provided by some embodiments of Application Scenario 1 of the present application;

[0053] Figure 3 Schematic diagram of the structure of a terminal device provided by some embodiments of Application Scenario 1 of the present application;

[0054] Figure 4 Architectural diagram of the software system of a terminal device provided by some embodiments of Application Scenario 1 of the present application;

[0055] Figure 5 Schematic diagram of the structure of a wearable device provided by some embodiments of Application Scenario 1 of the present application;

[0056] Figure 6 Flow chart of a calorie intake information detection method provided by some embodiments of Application Scenario 1 of the present application;

[0057] Figure 7 Schematic diagram of a user interface of a calorie management APP provided by an embodiment of the present application;

[0058] Figure 8 Schematic diagram of possible head postures during the user's eating process;

[0059] Figure 9 Timing diagram of a first image collected by the smart glasses;

[0060] Figure 10 Flow chart of the method for detecting calorie intake information provided by some other embodiments of Application Scenario 1 of the present application;

[0061] Figure 11 Schematic diagram of the user interface of a calorie management APP provided by another embodiment of the present application;

[0062] Figure 12 Flow chart of the method for detecting calorie intake information provided by some other embodiments of Application Scenario 1 of the present application;

[0063] Figure 13 Schematic diagram of the structure of a wearable device provided by some other embodiments of Application Scenario 1 of the present application;

[0064] Figure 14 Flow chart of the method for detecting calorie intake information provided by some other embodiments of Application Scenario 1 of the present application;

[0065] Figure 15 Schematic diagram of the structure of a wearable device provided by some embodiments of Application Scenario 2 of the present application;

[0066] Figure 16 Flow chart of the method for detecting calorie intake information provided by some embodiments of Application Scenario 2 of the present application;

[0067] Figure 17 Schematic diagram of the structure of a calorie intake information detection system provided by some embodiments of Application Scenario 3 of the present application;

[0068] Figure 18 Flow chart of the method for detecting calorie intake information provided by some embodiments of Application Scenario 3 of the present application. Detailed implementation manners

[0069] It should be noted that the terms used in the implementation manner part of the embodiments of the present application are only used to explain the specific embodiments of the present application, rather than to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; herein, "and / or" is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more than two, and "at least one", "one or more" mean one, two or more than two.

[0070] Hereinafter, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0071] Reference to "an embodiment" or "some embodiments" etc. described in this specification means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in an embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0072] Calorie intake refers to the calories ingested by the human body through eating food, usually measured in calories (cal) or kilocalories (abbreviated as kcal). Currently, with the continuous enhancement of people's health awareness, personal weight management and health management have received increasing attention. Reasonably controlling the calorie intake of each meal is the key to weight management and maintaining health. For this reason, more and more people begin to calculate and record the calorie intake of each meal through calorie management applications (APPs) installed on electronic devices such as mobile phones or smart watches.

[0073] Exemplarily, when calculating the calorie intake of each meal of a user, a calorie management APP provided by the related art requires the user to manually input the portions (such as weight, volume, or quantity, etc.) of various foods ingested in each meal, and then combines the calories corresponding to the unit portions of various foods stored in the food ingredient library to calculate the total calorie intake of the user for each meal.

[0074] Figure 1 Schematic diagram of the user interface involved when a calorie management APP provided by the related art calculates calorie intake. Exemplarily, as Figure 1 shown, when the user wants to record the calorie intake of breakfast, the user can control the mobile phone to enter the breakfast record interface of the calorie management APP. As Figure 1 shown in (a) of, various food addition controls a1 can be configured in the breakfast record interface A1 of the calorie management APP. For each food ingested by the user for breakfast, the user can click on the addition control of this food in the breakfast record interface A1 to control the mobile phone to display the intake input interface of this food. As Figure 1As shown in (b) thereof, an intake input box b1 and an intake determination control b2 are configured in the intake input interface B1 for each type of food. The user can input the intake of the corresponding food in the intake input box b1 and click the intake determination control b2. When the mobile phone detects that the intake determination control b2 in the intake input interface B1 is clicked, it can calculate the calorie intake corresponding to the food based on the intake of the corresponding food input by the user in the intake input box b1 and the calories corresponding to the unit portion of the food, and return to the breakfast record interface A1. Exemplarily, as Figure 1 shown in (c) thereof, a calorie record control a2 is also configured in the breakfast record interface A1. After the user adds the portions of all the foods consumed for breakfast and obtains the calorie intakes corresponding to all the foods, the user can click the calorie record control a2 in the breakfast record interface A1. After the mobile phone detects that the calorie record control a2 in the breakfast record interface A1 is clicked, it calculates the total calorie intake of the user's breakfast and displays the details interface of the calorie intake information as shown in Figure 1 (d) thereof.

[0075] It can be seen that the above calorie intake calculation method requires the user to weigh or estimate each type of food consumed in advance and manually input the weighing result or estimation result into the calorie management APP. The operation process is relatively cumbersome, resulting in a poor user experience. Moreover, in the absence of weighing equipment, the intakes of various foods input by the user are usually arbitrarily estimated by humans, resulting in inaccurate calculation of calorie intake.

[0076] Exemplarily, another calorie management APP provided by the related art can automatically identify the total calories of all the foods to be consumed included in the image taken by the user and use it as the calorie intake of the user for the current meal. However, in actual applications, it often occurs that some of the foods to be consumed in the image are not consumed by the user, and the shooting angle of the image has a great impact on the volume estimation of the foods to be consumed, resulting in inaccurate calculation of calorie intake.

[0077] In view of this, in order to accurately detect the calorie intake information of each meal of the user and simplify the manual operation process in the calorie intake information detection process, so as to provide the user with a better experience. An embodiment of the present application provides a method for detecting calorie intake information. By using a wearable device worn on the head to periodically collect a first image and perform object detection on the first image. When it is detected that the first image contains food to be put into the mouth, audio data and the movement data of the user's head are collected by the wearable device, so that the sound and head posture when the user intakes the food to be put into the mouth can be automatically monitored; by determining the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time according to the audio data, and determining the head posture of the user when the user intakes the food to be put into the mouth according to the movement data of the user's head, and comprehensively considering the user's single default swallowing amount, the head posture of the user when the user intakes the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time, the actual intake amount of the food to be put into the mouth is determined, thus realizing the accurate detection of the user's actual food intake; and then the calorie intake information of the user can be accurately calculated according to the actual intake amounts of all the foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portions. Since the wearable device can automatically monitor the entire eating process of the user and automatically detect the calorie intake information of the user, the manual operation process in the calorie intake information detection process is simplified, and the user has a better experience.

[0078] The method for detecting calorie intake information provided by the embodiment of the present application can be applied to multiple different application scenarios. The method for detecting calorie intake information provided by the embodiment of the present application will be described in detail below in combination with each different application scenario.

[0079] Application scenario one:

[0080] In application scenario one, the method for detecting calorie intake information can be applied in Figure 2 the calorie intake information detection system shown in Figure 2 As shown in, the calorie intake information detection system can include a terminal device 10 and at least one wearable device 20. The terminal device 10 and at least one wearable device 20 cooperate with each other to jointly realize the detection, statistics, display, etc. of calorie intake information.

[0081] Specifically, when detecting calorie intake information, the terminal device 10 and the wearable device 20 can establish a communication connection. Exemplarily, the above communication connection may include a wireless communication connection. The wireless communication connection may include a Bluetooth connection or a wireless local area network (WLAN) (e.g., a wireless fidelity (WIFI) connection, etc.). The embodiments of the present application do not limit the communication connection method between the terminal device 10 and the wearable device 20.

[0082] Exemplarily, the terminal device 10 may be included in a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not limit the device form of the terminal device 10.

[0083] Figure 3 It is a schematic structural diagram of a terminal device 10 provided by an embodiment of the present application. As Figure 3 shown, the terminal device 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180I, a touch sensor 180J, a bone conduction sensor 180K, an ambient light sensor 180L, etc.

[0084] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0085] The wireless communication function of the terminal device 10 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc. Among them, the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 may provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the terminal device 10. The wireless communication module 160 may provide solutions for wireless communications such as Bluetooth, WLAN, global navigation satellite system (GNSS), near field communication (NFC), infrared technology (IR), etc. applied to the terminal device 10. Exemplarily, the wireless communication module 160 may be one or more devices integrating at least one communication processing module.

[0086] The camera 193 may be used to capture still images or videos. Exemplarily, the terminal device 10 may include 1 or N cameras 193, where N is an integer greater than 1.

[0087] The display screen 194 may be used to display images, videos, etc. Exemplarily, the display screen 194 may include a display panel.

[0088] The terminal device 10 may implement the display function through the GPU, the display screen 194, and the application processor, etc.

[0089] The terminal device 10 may implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0090] The terminal device 10 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc.

[0091] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 10. In some embodiments, the angular velocity of the terminal device 10 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B.

[0092] The acceleration sensor 180E can be used to detect the magnitude of the acceleration of the terminal device 10 in each direction (generally three axes). When the terminal device 10 is stationary, the magnitude and direction of gravity can be detected.

[0093] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect a touch operation acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the terminal device 10, at a different position from that of the display screen 194.

[0094] The keys 190 include a power-on key, a volume key, etc. The keys 190 can be mechanical keys or touch keys, etc. The terminal device 10 can receive key inputs and generate key signal inputs related to the user settings and function controls of the terminal device 10.

[0095] A software system (i.e., an operating system) can run on the terminal device 10. The above software system can be an Android® system, an iOS® system, a Microsoft® system, a Harmony OS system, or other operating systems, etc.

[0096] Exemplarily, the software system of the terminal device 10 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture, etc. In the embodiments of the present application, taking the layered architecture as an example, the software system of the terminal device 10 is exemplarily described.

[0097] Figure 4 This is an architecture diagram of a software system of a terminal device 10 provided in the embodiments of the present application. As Figure 4As shown, the layered architecture divides the software system of the terminal device 10 into several layers. Each layer has a clear role and division of labor, and the layers communicate through software interfaces. In some embodiments, the software system of the terminal device 10 may include an application (APP) layer, an application framework layer, a runtime, system libraries, and a kernel layer.

[0098] Exemplarily, the application layer may include a series of applications. For example, a camera APP, a heat management APP, etc.

[0099] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0100] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0101] Exemplarily, the wearable device 20 may be a general term for devices that can be worn, which are developed by applying wearable technologies to the intelligent design of daily wear. Optionally, in Figure 2 the case where the shown heat intake information detection system only includes one wearable device 20, the wearable device 20 may be a wearable device for wearing on the user's head. For example, smart glasses, smart earrings, or smart headphones, etc. Optionally, in Figure 2 the case where the shown heat intake information detection system includes multiple wearable devices 20, the multiple wearable devices 20 may include, in addition to the wearable devices for wearing on the user's head, wearable devices for wearing on other parts. Other parts may refer to parts other than the head, such as the neck or wrist, etc. The wearable devices for wearing on other parts may include, for example, smart watches or smart necklaces, etc.

[0102] In some embodiments of Application Scenario 1, the wearable device 20 may have a data collection function and a data transmission function, and does not have a data processing function and a display function, etc. That is, in this embodiment, the wearable device 20 may be used only as a sensor. Among them, the data collection function may include functions such as image collection, audio collection, and motion data collection.

[0103] Based on this, optionally, in Figure 2 the case where the shown heat intake information detection system only includes one wearable device 20, the wearable device 20 may have all of the above data collection functions. For example, assuming Figure 2If the wearable device 20 in it is a smart glasses, the smart glasses can have functions such as image acquisition, audio acquisition, and motion data acquisition. Exemplarily, as Figure 5 shown in (a) of, a camera module, an audio module, a motion sensor, a wireless communication module, etc. can be configured in the smart glasses. The audio module can include a microphone. The motion sensor can include a gyroscope and an acceleration sensor. The wireless communication module can include a Bluetooth communication module and / or a WIFI communication module, etc. The smart glasses can implement an image shooting function through the camera module, an audio data acquisition function through the audio module, a motion data acquisition function through the motion sensor, and a data transmission function through the wireless communication module.

[0104] Optionally, in Figure 2 the case where the detection system of the calorie intake information shown includes multiple wearable devices 20, each wearable device 20 can have some of the above data acquisition functions, and all wearable devices 20 together have all the above data acquisition functions, and all wearable devices 20 have a data transmission function. For example, assume Figure 2 the wearable device 20 in includes a smart glasses and a smart necklace, then the smart glasses can have an image acquisition function and a motion data acquisition function, and the smart necklace can have an audio acquisition function. Exemplarily, as Figure 5 shown in (b) of, a camera module, a motion sensor, a wireless communication module, etc. can be configured in the smart glasses. An audio module and a wireless communication module, etc. can be configured in the smart necklace. The smart glasses can implement a shooting function through the camera module, a motion data acquisition function through the motion sensor, and a data transmission function through the wireless communication module. The smart necklace can implement an audio data acquisition function through the audio module and a data transmission function through the wireless communication module.

[0105] Taking the terminal device 10 in the detection system of the calorie intake information as a mobile phone and the wearable device 20 as a smart glasses as an example below, the detection method of the calorie intake information provided by some embodiments of Application Scenario 1 will be described.

[0106] Exemplarily, Figure 6 is a schematic flowchart of the detection method of the calorie intake information provided by some embodiments of Application Scenario 1 of the present application. As Figure 6 shown, the detection method of the calorie intake information can include S601 to S607, which are described in detail as follows:

[0107] S601, the smart glasses periodically collect a first image based on a preset time interval and send the first image to the mobile phone.

[0108] In application scenario 1, when the user needs to automatically detect the calorie intake information for each meal, the user can wear the smart glasses throughout the eating process of each meal and turn on the smart glasses before eating.

[0109] In an alternative implementation, after the smart glasses are turned on, they can automatically collect the first image periodically based on a preset time interval and send the first image to the mobile phone. In this implementation, since the smart glasses can automatically perform image collection after being turned on without the user having to manually perform additional operations, the convenience of detecting calorie intake information can be improved, providing the user with a better experience. In addition, the user can turn off the smart glasses after finishing eating to save the power of the smart glasses and extend the battery life of the smart glasses.

[0110] In another alternative implementation, after the smart glasses are turned on, in response to receiving a diet calorie detection instruction, they can collect the first image periodically based on a preset time interval and send the first image to the mobile phone.

[0111] In some embodiments of this implementation, as Figure 2 shown, a preset control button 201 can be configured on the smart glasses. The preset control button 201 can be used to control the smart glasses to start or stop diet calorie detection. Exemplarily, the preset control button 201 can be a mechanical button or a touch button, etc. Based on this, the diet calorie detection instruction can be sent by the user to the smart glasses by triggering the preset control button 201 on the smart glasses. For example, the user can trigger the preset control button 201 on the smart glasses before eating to send a diet calorie detection instruction to the smart glasses. After receiving the diet calorie detection instruction, the smart glasses can collect the first image periodically based on a preset time interval. In addition, the user can trigger the preset control button 201 on the smart glasses again after finishing eating to send a stop detection instruction to the smart glasses. After receiving the stop detection instruction, the smart glasses can stop collecting the first image.

[0112] In some other embodiments of this implementation, the diet calorie detection instruction can be sent by the mobile phone to the smart glasses. Exemplarily, as Figure 7 shown, a diet calorie detection control 71 can be configured in the calorie management APP on the mobile phone. The diet calorie detection control 71 can be used to control the smart glasses to start or stop diet calorie detection. Specifically, the diet calorie detection control 71 can include Figure 7 the unenabled state shown in (a) of Figure 7The enabled state shown in (b) therein. When the dietary calorie detection control 71 is in the disabled state, it indicates that the dietary calorie detection is not currently being performed; when the dietary calorie detection control 71 is in the enabled state, it indicates that the dietary calorie detection is currently being performed.

[0113] Based on this, the user can trigger Figure 7 the dietary calorie detection control 71 in the disabled state shown in (a) therein. When the mobile phone detects that the dietary calorie detection control 71 in the disabled state is triggered, it can send a dietary calorie detection instruction to the smart glasses and switch the dietary calorie detection control 71 from the disabled state to Figure 7 the enabled state shown in (b) therein. After receiving the dietary calorie detection instruction from the mobile phone, the smart glasses can periodically collect the first images based on a preset time interval. In addition, the user can also trigger Figure 7 the dietary calorie detection control 71 in the enabled state shown in (b) therein after finishing eating. When the mobile phone detects that the dietary calorie detection control 71 in the enabled state is triggered, it can send a stop detection instruction to the smart glasses. After receiving the stop detection instruction from the mobile phone, the smart glasses can stop collecting the first images.

[0114] Among them, the preset time interval can be set according to actual needs. For example, the preset time interval can be 0.1 second.

[0115] Exemplarily, the smart glasses can sequentially send the first images to the mobile phone in the order of the acquisition time of the first images. For example, the smart glasses can send the first image acquired each time to the mobile phone after each acquisition of a first image.

[0116] S602, the mobile phone performs object detection on each of the first images in sequence.

[0117] The purpose of the mobile phone performing object detection on the first image is to detect whether the first image contains the food to be eaten, and in the case where the first image contains the food to be eaten, determine the contour of the food to be eaten, the first candidate name, and the confidence level corresponding to the first candidate name in the first image.

[0118] Among them, the food to be put into the mouth refers to the food that has been picked up by the user with tableware (such as chopsticks, spoons or forks, etc.) or hands and is about to be put into the mouth. It should be noted that the food to be put into the mouth is usually a small part of all the food to be ingested. For example, assuming the food to be ingested is a bowl of rice, the food to be put into the mouth can be a spoonful of rice scooped up by the user from the bowl. Another example, assuming the food to be ingested includes multiple strawberries, the food to be put into the mouth can be one strawberry picked up by the user from the multiple strawberries. That is to say, only when the first image contains food and the food is on the tableware or in the hand can it be considered that the first image contains the food to be put into the mouth. Based on this, the mobile phone can determine that the first image contains the food to be put into the mouth when it detects that the first image contains food on the tableware or in the hand.

[0119] It can be understood that in practical applications, the food (i.e., the food to be put into the mouth) put into the mouth by the user each time may be one kind or multiple kinds. For example, when the food to be ingested is curry rice, the food put into the mouth by the user each time may only contain rice, may only contain curry, or may contain both rice and curry at the same time. Therefore, the food to be put into the mouth contained in the first image can be one kind or multiple kinds.

[0120] Exemplarily, the mobile phone can use a pre-trained object detection model for detecting the food to be put into the mouth to perform object detection on the first image. Specifically, the input of the above object detection model can be an image, and the output can be the contours of various foods to be put into the mouth in the image, the first candidate name, and the confidence level corresponding to the first candidate name.

[0121] Exemplarily, an object detection algorithm can be configured in the object detection model. The object detection algorithm can include, for example, an object detection algorithm based on region-based convolutional neural network (R-CNN), an object detection algorithm based on single shot multibox detector (SDD), and an object detection algorithm based on you only look once (YOLO), etc.

[0122] Exemplarily, the mobile phone can perform object detection on each first image in sequence according to the reception order of the first images.

[0123] S603. When the mobile phone detects that the first image contains the food to be put into the mouth, it sends an eating monitoring instruction to the smart glasses.

[0124] The eating monitoring instruction can be used to instruct the smart glasses to start collecting audio data and the motion data of the user's head.

[0125] It can be understood that when it is detected that the first image contains food to be put into the mouth, it indicates that the user is about to put the food to be put into the mouth into the mouth. Exemplarily, the food to be put into the mouth can be liquid food or non-liquid food. Liquid food refers to food that has fluidity and can adapt to the shape of various containers, and can usually be directly swallowed. For example, milk, soup, and milkshakes, etc. Non-liquid food refers to food that exists in solid or semi-solid form and usually needs to be chewed before it can be swallowed. For example, rice, vegetables, and meat, etc. Based on this, when the food to be put into the mouth is liquid food, the user will swallow it after putting the food to be put into the mouth, so a swallowing sound will be generated. When the food to be put into the mouth is non-liquid food, the user will chew and swallow it after putting the food to be put into the mouth, so a chewing sound and a swallowing sound will be generated. Therefore, when the mobile phone detects that the first image contains food to be put into the mouth, by controlling the smart glasses to start collecting audio data, it can monitor the swallowing sound and / or chewing sound generated when the user intakes the food to be put into the mouth, which is convenient for accurately calculating the actual intake amount of the food to be put into the mouth (that is, the actual food intake amount of the user for the food to be put into the mouth) according to the collected audio data later.

[0126] It can also be understood that during the eating process of the user, different head postures will result in different swallowing amounts of the user when swallowing food each time. Exemplarily, as Figure 8 shown, during the eating process of the user, the head may be in the non-head-up state shown in (a) in Figure 8 , or may be in the head-up state shown in (b) in Figure 8 . The swallowing amount of the user when swallowing food each time in the head-up state is usually greater than that in the non-head-up state, and the greater the head-up angle of the user, the greater the swallowing amount of the user when swallowing food each time. Therefore, when the mobile phone detects that the first image contains food to be put into the mouth, by controlling the smart glasses to start collecting the motion data of the user's head, it can monitor the head posture of the user when intaking the food to be put into the mouth, which is convenient for more accurately calculating the actual intake amount of the food to be put into the mouth in combination with the motion data of the user's head later.

[0127] S604. In response to the eating monitoring instruction from the mobile phone, the smart glasses start to collect audio data and the motion data of the user's head, and send the audio data and the motion data of the user's head to the mobile phone.

[0128] Exemplarily, after receiving the eating monitoring instruction, the smart glasses can collect audio data through the microphone in its internal audio module.

[0129] Optionally, when the food to be put into the mouth in the first image is liquid food, the audio data collected by the smart glasses may include the swallowing sound generated when the user intakes the food to be put into the mouth.

[0130] Optionally, when the food to be put into the mouth in the first image is non-liquid food, the audio data collected by the smart glasses may include the chewing sound and swallowing sound generated when the user ingests the food to be put into the mouth.

[0131] It can be understood that when the smart glasses are worn by the user, the smart glasses and the user's head are integrated, that is, the smart glasses will move with the movement of the user's head, and the posture of the smart glasses will be consistent with the head posture of the user. Therefore, after receiving the eating monitoring instruction, the smart glasses can collect the motion data of the smart glasses through the motion sensors inside it, and use the motion data of the smart glasses as the motion data of the user's head. Exemplarily, the above motion data at least includes the angular velocity data collected by the gyroscope. In addition, it may also include the acceleration data collected by the acceleration sensor.

[0132] Exemplarily, after receiving the eating monitoring instruction, the smart glasses can collect the audio data and the motion data of the user's head every first period, and sequentially send the collected audio data and the motion data of the user's head to the mobile phone according to the chronological order of collection. Among them, the first period can be set according to actual needs, and this embodiment does not limit it.

[0133] S605. The mobile phone determines the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound for each swallowing according to the audio data, and determines the head posture of the user when ingesting the food to be put into the mouth according to the motion data of the user's head.

[0134] It can be understood that since the audio data collected by the smart glasses may only include the swallowing sound, or may include both the chewing sound and the swallowing sound at the same time. Therefore, after receiving the audio data from the smart glasses, the mobile phone can identify and extract the audio data corresponding to the swallowing sound, or identify and extract the audio data corresponding to the chewing sound and the audio data corresponding to the swallowing sound from the audio data based on the audio characteristics of the chewing sound and the audio characteristics of the swallowing sound. Among them, the audio characteristics of the chewing sound are different from the audio characteristics of the swallowing sound. It should be noted that the specific differences between the audio characteristics of the chewing sound and the audio characteristics of the swallowing sound can refer to the descriptions in the related technologies, and will not be elaborated here.

[0135] It can also be understood that since the audio data corresponding to the swallowing sound can accurately reflect the actual swallowing situation of the food to be put into the mouth, for example, the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound for each swallowing. Therefore, after the mobile phone extracts the audio data corresponding to the swallowing sound, it can analyze the audio data corresponding to the swallowing sound to determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound for each swallowing from the audio data corresponding to the swallowing sound.

[0136] In a specific embodiment, the mobile phone can determine the number of target peaks appearing in the audio data corresponding to the swallowing sound as the number of times the food to be put into the mouth is swallowed. Wherein, the target peak can refer to a peak whose corresponding amplitude is greater than a preset decibel threshold. Exemplarily, the preset decibel threshold can be the minimum decibel value of the swallowing sound obtained through big data analysis, and this minimum decibel value can distinguish the swallowing sound from background noise.

[0137] In another specific embodiment, the mobile phone can respectively determine the size of the swallowing sound each time the food to be put into the mouth is swallowed according to the amplitude size corresponding to each target peak.

[0138] In an alternative implementation, the head posture can be described by the pitch angle of the head. The pitch angle of the head can refer to the angle at which the head bows forward or tilts backward relative to the plane where the body is located. Exemplarily, in Figure 8 as shown in (a), since the user is in a state of neither tilting the head back nor bowing the head forward, therefore, the pitch angle of the user's head can be 0 degrees. Exemplarily, in Figure 8 as shown in (b), since the user is in a state of tilting the head back, therefore, the pitch angle θ of the user's head is greater than 0 degrees. It should be noted that when the user is in a state of bowing the head (not shown), the pitch angle of the user's head can be less than 0 degrees.

[0139] Based on this, the mobile phone can determine the pitch angle of the head when the user ingests the food to be put into the mouth according to the angular velocity data and / or acceleration data in the movement data of the user's head. It should be noted that the specific process of determining the pitch angle according to the angular velocity data and / or acceleration data can refer to the description in related technologies, and the embodiments of the present application will not elaborate on it.

[0140] S606. The mobile phone determines the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the head posture when the user ingests the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time it is swallowed.

[0141] It can be understood that users with different body parameters usually have different swallowing amounts of food each time. Body parameters can include, for example, gender, age, height, and weight, etc. Exemplarily, the swallowing amount of food each time for middle-aged people is usually greater than that of the elderly with the same gender, the same height, and the same weight each time they swallow food, and the swallowing amount of food each time for men is usually greater than that of women with the same age, the same height, and the same weight each time they swallow food.

[0142] Exemplarily, the corresponding relationship between each body parameter and the single default swallowing amount can be stored in the mobile phone. Based on this, the mobile phone can determine the user's single default swallowing amount according to the various body parameters set by the user in the calorie management APP.

[0143] It should be noted that the user's single default swallowing amount is a basic single swallowing amount roughly determined based on the user's body parameters. However, in actual applications, the head posture of the user when ingesting the food to be put into the mouth and the magnitude of the swallowing sound of the food to be put into the mouth each time it is swallowed will both affect the actual swallowing amount corresponding to each swallowing of the food to be put into the mouth. For example, the larger the pitch angle of the head when the user ingests the food to be put into the mouth, the larger the actual swallowing amount corresponding to each swallowing of the food to be put into the mouth; the smaller the pitch angle of the head when the user ingests the food to be put into the mouth, the smaller the actual swallowing amount corresponding to each swallowing of the food to be put into the mouth. For another example, the larger the swallowing sound of the food to be put into the mouth each time it is swallowed, the larger the corresponding actual swallowing amount; the smaller the swallowing sound of the food to be put into the mouth each time it is swallowed, the usually smaller the corresponding actual swallowing amount. Based on this, the mobile phone can, on the basis of the user's single default swallowing amount, combine the head posture of the user when ingesting the food to be put into the mouth and the magnitude of the swallowing sound of the food to be put into the mouth each time it is swallowed, determine the actual swallowing amount corresponding to each swallowing of the food to be put into the mouth, and then accurately calculate the actual intake amount of the food to be put into the mouth according to the actual swallowing amount corresponding to each swallowing of the food to be put into the mouth.

[0144] Optionally, the mobile phone can store the corresponding relationship between the magnitude of the swallowing sound and the first swallowing amount coefficient. Among them, the first swallowing amount coefficient can refer to the adjustment factor for adjusting the actual swallowing amount determined based on the magnitude of the swallowing sound. Exemplarily, the value of the first swallowing amount coefficient can be less than 1, can be equal to 1, or can be greater than 1.

[0145] In actual applications, the first swallowing amount coefficient can be obtained by the mobile phone when the user first uses the calorie management APP by prompting the user to eat a certain amount of food, collecting the audio data during the user's eating process, and determining based on the user's eating amount and the audio data during the eating process. For example, when the user first uses the calorie management APP, the mobile phone can prompt the user to drink 100 milliliters of water and collect the audio data during the user's drinking process. The mobile phone can analyze the audio data to determine the number of times the water is swallowed and the magnitude of the swallowing sound each time it is swallowed, and can determine the actual swallowing amount corresponding to each swallowing of the water according to the user's drinking amount, the number of times the water is swallowed, and the magnitude of the swallowing sound each time it is swallowed, and then determine the ratio of the actual swallowing amount corresponding to each swallowing of the water to the user's default swallowing amount as the first swallowing amount coefficient corresponding to the magnitude of the swallowing sound of the water each time it is swallowed.

[0146] Based on this, the mobile phone can determine the first swallowing amount coefficient corresponding to each swallowing of the food to be put into the mouth according to the magnitude of the swallowing sound of the food to be put into the mouth each time it is swallowed through the corresponding relationship between the magnitude of the swallowing sound and the first swallowing amount coefficient.

[0147] Optionally, the mobile phone may also store the correspondence between the pitch angle of the head and the second swallowing volume coefficient. Herein, the second swallowing volume coefficient may refer to an adjustment factor for adjusting the actual swallowing volume determined based on the pitch angle of the head. Exemplarily, the value of the second swallowing volume coefficient may be less than 1, equal to 1, or greater than 1.

[0148] It can be understood that since liquid food is easier to swallow than solid food, therefore, at the same head posture, the actual swallowing volume corresponding to each swallowing of liquid food is generally greater than the actual swallowing volume corresponding to each swallowing of solid food. Based on this, the mobile phone may separately store the correspondence between the pitch angle of the head corresponding to liquid food and the second swallowing volume coefficient, and the correspondence between the pitch angle of the head corresponding to solid food and the second swallowing volume coefficient. It should be noted that at the same pitch angle, the second swallowing volume coefficient corresponding to liquid food is greater than the second swallowing volume coefficient corresponding to solid food.

[0149] Based on this, before determining the second swallowing coefficient corresponding to each swallowing of the food to be put into the mouth, the mobile phone may first determine the texture type of the food to be put into the mouth, and then according to the pitch angle of the head when the user ingests the food to be put into the mouth, determine the second swallowing volume coefficient corresponding to each swallowing of the food to be put into the mouth through the correspondence between the pitch angle of the head corresponding to the texture type of the food to be put into the mouth and the second swallowing volume coefficient. Herein, the texture type may include liquid food and solid food.

[0150] In an optional implementation manner, the mobile phone may determine the real name of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by performing object detection on the first image and the confidence corresponding to the first candidate name, and determine the texture type of the food to be put into the mouth according to the real name of the food to be put into the mouth. Exemplarily, the mobile phone may determine the first candidate name with the corresponding confidence greater than or equal to the preset confidence threshold as the real name of the food to be put into the mouth. Herein, the preset confidence threshold may be set according to actual requirements. For example, the preset confidence threshold may be 98%. Based on this, assuming that the first candidate name of the food to be put into the mouth obtained by performing object detection on the first image is juice, and the confidence corresponding to juice is greater than 98%, then the mobile phone may determine that the food to be put into the mouth in the first image is juice and determine the texture type of the food to be put into the mouth as liquid food.

[0151] In another optional implementation manner, the mobile phone may determine the texture type of the food to be put into the mouth according to the audio data collected by the smart glasses. Exemplarily, when the audio data does not include chewing sounds, the mobile phone may determine that the food to be put into the mouth is liquid food; when the audio data includes chewing sounds, the mobile phone may determine that the food to be put into the mouth is solid food.

[0152] In yet another alternative implementation, the mobile phone can combine the audio data, the first candidate name of the food to be put into the mouth obtained by performing object detection on the first image, and the confidence level corresponding to the first candidate name to determine the texture type of the food to be put into the mouth. Specifically, the mobile phone can first determine the real name of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by performing object detection on the first image and the confidence level corresponding to the first candidate name, and then determine the texture type of the food to be put into the mouth according to the audio data and the real name of the food to be put into the mouth. Exemplarily, when the real name of the food to be put into the mouth belongs to the name of liquid food and the audio data does not include chewing sounds, the mobile phone can determine that the food to be put into the mouth is liquid food; when the real name of the food to be put into the mouth belongs to the name of non-liquid food, or the audio data includes chewing sounds, the mobile phone can determine that the food to be put into the mouth is non-liquid food. This implementation method comprehensively determines the texture type of the food to be put into the mouth based on image detection and audio detection, and can improve the accuracy of determining the texture type of the food to be put into the mouth.

[0153] Optionally, after the mobile phone determines the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing of the food to be put into the mouth, it can determine the actual intake amount of the food to be put into the mouth according to the user's single default swallowing volume, the number of times the food to be put into the mouth is swallowed, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing of the food to be put into the mouth.

[0154] Exemplarily, the mobile phone can use the following formula (1) to determine the actual intake amount of the food to be put into the mouth:

[0155] ; Formula (1)

[0156] Wherein, TY sj is the actual intake amount of the food to be put into the mouth, TY mr is the user's single default swallowing volume, n is the number of times the food to be put into the mouth is swallowed, α i is the first swallowing volume coefficient corresponding to the i th swallowing of the food to be put into the mouth, β i is the second swallowing volume coefficient corresponding to the i th swallowing of the food to be put into the mouth.

[0157] S607, the mobile phone determines the user's calorie intake information according to the actual intake amounts of all the foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portions of each food to be put into the mouth.

[0158] Exemplarily, the calorie intake information may include the name, portion size, calories of each food consumed by the user, and the total calorie intake of the user, etc. The total calorie intake is the sum of the calories of all the foods consumed by the user.

[0159] It can be understood that during the entire eating process, the user usually picks up food and puts it into the mouth multiple times. That is, the entire eating process may include multiple operations of putting the food to be eaten into the mouth. Since after the user puts the food to be eaten into the mouth each time, the food to be eaten needs to be chewed and / or swallowed, etc., there will be a certain time interval between every two adjacent operations of putting the food to be eaten into the mouth. Based on this, during the entire eating process, the mobile phone can detect the food to be eaten from the first images collected in multiple discontinuous time periods. Each of the discontinuous time periods may correspond to an operation of putting the food to be eaten into the mouth. Exemplarily, Figure 9 is a timing diagram of a kind of first image collected by the smart glasses. As Figure 9 shown, assuming that the t1 time period corresponds to an operation of the user putting the food to be eaten into the mouth, the t2 time period corresponds to the operation of the user chewing and / or swallowing the food to be eaten, and the t3 time period corresponds to another operation of the user putting the food to be eaten into the mouth, then the mobile phone can detect the food to be eaten from the first images collected in the t1 time period and the t3 time period.

[0160] Based on this, the mobile phone can calculate the actual intake amount of the food to be eaten detected in multiple discontinuous time periods through S601 - S606 respectively.

[0161] Since the calories corresponding to the unit portion sizes of different foods are different, a food ingredient library can also be configured in the mobile phone. The food ingredient library can be used to store information about various foods. Optionally, the information about the food may include the name, texture type, image, and calories corresponding to the unit portion size of the food, etc. Exemplarily, for liquid foods, the food ingredient library can store the calories corresponding to the unit volume of the food, and the unit volume can be, for example, 100 milliliters (ml); for non - liquid foods, the food ingredient library can store the calories corresponding to the unit weight of the food, and the unit weight can be, for example, 100 grams (g). Optionally, for non - liquid foods, the information about the food may further include the audio characteristics of the chewing sound of the food. Exemplarily, the audio characteristics of the chewing sounds of various foods can be obtained by collecting and analyzing the chewing sounds when the user eats various foods in advance.

[0162] Exemplarily, the information about various foods stored in the food ingredient library can be as shown in Table 1:

[0163] Table 1

[0164]

[0165] Based on this, for the food to be put into the mouth detected by the mobile phone within any time period, in an alternative implementation, when there is only one type of food to be put into the mouth, the mobile phone can query from the food ingredient library the calories corresponding to the unit portion of the food to be put into the mouth according to the real name of the food to be put into the mouth, and then calculate the calorie intake corresponding to the food to be put into the mouth according to the actual intake amount of the food to be put into the mouth and the calories corresponding to the unit portion. For example, the mobile phone can determine the product of the actual intake amount of the food to be put into the mouth and the calories corresponding to the unit portion as the calorie intake corresponding to the food to be put into the mouth.

[0166] In another alternative implementation, when there are multiple types of food to be put into the mouth, the mobile phone can first determine the volume proportion of each type of food to be put into the mouth according to the contours of various foods to be put into the mouth obtained by performing object detection on the first image; then query from the food ingredient library the calories corresponding to the unit portion of each type of food to be put into the mouth according to the real name of each type of food to be put into the mouth; and calculate the calorie intake corresponding to the food to be put into the mouth in the first image according to the actual intake amount of the food to be put into the mouth in the first image, the volume proportion of each type of food to be put into the mouth, and the calories corresponding to the unit portion.

[0167] Exemplarily, the first image can be a depth image. Based on this, the mobile phone can respectively determine the area of the region where each type of food to be put into the mouth is located according to the contours of various foods to be put into the mouth obtained by performing object detection on the first image; determine the volume of each type of food to be put into the mouth according to the area of the region where each type of food to be put into the mouth is located and the depth information; and can respectively determine the ratio of the volume of each type of food to be put into the mouth to the sum of the volumes of all foods to be put into the mouth as the volume proportion of each type of food to be put into the mouth.

[0168] Exemplarily, the mobile phone can calculate the calorie intake corresponding to the food to be put into the mouth through the following formula (2) according to the actual intake amount of the food to be put into the mouth, the volume proportion of each type of food to be put into the mouth, and the calories corresponding to the unit portion:

[0169] ; Formula (2)

[0170] Wherein, RL is the calorie intake corresponding to the food to be put into the mouth, TY sj is the actual intake amount of the food to be put into the mouth, m is the number of types of food to be put into the mouth, μ i is the j th volume proportion of the food to be put into the mouth, Δ RL is the j th calories corresponding to the unit portion of the food to be put into the mouth.

[0171] Exemplarily, the mobile phone can determine the sum of the calorie intakes corresponding to all the foods to be eaten that are detected respectively within multiple discontinuous time periods as the total calorie intake of the user.

[0172] In an alternative implementation, when the audio data collected by the smart glasses does not include the audio data corresponding to the chewing sound, the mobile phone can determine the real name of the food to be eaten according to the first candidate name of the food to be eaten obtained by performing object detection on the first image and the confidence corresponding to the first candidate name. Exemplarily, the mobile phone can determine the first candidate name with the corresponding confidence greater than the preset confidence threshold as the real name of the food to be eaten.

[0173] In another alternative implementation, when the audio data collected by the smart glasses includes the audio data corresponding to the chewing sound, the mobile phone can determine the second candidate name of the food to be eaten and the confidence corresponding to the second candidate name according to the audio features of the chewing sound included in the audio data; and can determine the real name of the food to be eaten according to the first candidate name of the food to be eaten obtained by performing object detection on the first image, the confidence corresponding to the first candidate name, the second candidate name, and the confidence corresponding to the second candidate name.

[0174] Exemplarily, when the first candidate name is the same as the second candidate name, and the confidence corresponding to the first candidate name and the confidence corresponding to the second candidate name are both greater than or equal to the preset confidence threshold, the mobile phone can determine the first candidate name or the second candidate name as the real name of the food to be eaten.

[0175] Exemplarily, when the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is greater than or equal to the preset confidence threshold, and the confidence corresponding to the second candidate name is less than the preset confidence threshold, the mobile phone can determine the first candidate name as the real name of the food to be eaten.

[0176] Exemplarily, when the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is less than the preset confidence threshold, and the confidence corresponding to the second candidate name is greater than or equal to the preset confidence threshold, the mobile phone can determine the second candidate name as the real name of the food to be eaten.

[0177] After the mobile phone determines the calorie intake information of the user, it can display the calorie intake information for the user to view.

[0178] As can be seen from the above, the detection method of calorie intake information provided in this embodiment periodically collects a first image by using a wearable device worn on the head, and uses a mobile phone to perform object detection on the first image. When it is detected that the first image contains food to be put into the mouth, the wearable device is controlled to collect audio data and the movement data of the user's head, so as to realize the automatic monitoring of the sound and head posture when the user intakes the food to be put into the mouth; by using the mobile phone to determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time according to the audio data, and determine the head posture of the user when the user intakes the food to be put into the mouth according to the movement data of the user's head, and comprehensively consider the user's single default swallowing amount, the head posture of the user when the user intakes the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time, to determine the actual intake amount of the food to be put into the mouth, thus realizing the accurate detection of the user's actual food intake; so that the mobile phone can accurately calculate the calorie intake information of the user according to the actual intake amounts of all the foods to be put into the mouth detected respectively in multiple discontinuous time periods and the calories corresponding to the unit portions. Since the wearable device can automatically monitor the entire eating process of the user, the mobile phone can automatically detect the calorie intake information of the user according to the data collected by the wearable device, thus simplifying the manual operation process in the detection process of calorie intake information and enabling the user to have a better experience.

[0179] In addition, since the smart glasses in this embodiment are only used as sensors, there is no need to configure a processor in the smart glasses, thus being able to reduce the production cost of the smart glasses.

[0180] In some other embodiments of Application Scenario 1, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, and thus further improve the accuracy of calorie intake information detection, as Figure 10 shown, in an optional implementation manner, the detection method of calorie intake information may further include S608 - S611, which are described in detail as follows:

[0181] S608, the mobile phone obtains a second image containing all the foods to be ingested, and determines the total calories of all the foods to be ingested in the second image.

[0182] Among them, the foods to be ingested may refer to the foods that the user is going to ingest. Exemplarily, assuming that the user is going to ingest milk, bread, and eggs for breakfast, the second image may include milk, bread, and eggs.

[0183] Optionally, the second image may be taken by the mobile phone by the user before eating. Based on this, as Figure 11As shown in the figure, the heat management APP can also be configured with a first image adding control 72 for adding an image of the food to be ingested. The user can click on the first image adding control 72 to control the mobile phone to obtain a second image containing all the food to be ingested from the photo album.

[0184] Optionally, the second image can be taken by the smart glasses before eating by the user and sent to the mobile phone by the smart glasses.

[0185] Exemplarily, the mobile phone can determine the total calories of all the food to be ingested in the second image through the following steps X1 to X3:

[0186] Step X1, determine the real names of various foods to be ingested in the second image.

[0187] Exemplarily, the mobile phone can perform object detection on the second image to obtain the outlines, third candidate names, and the confidence levels corresponding to the third candidate names of various foods to be ingested in the second image, and determine the real names of various foods to be ingested according to the third candidate names and the confidence levels corresponding to the third candidate names of various foods to be ingested in the second image. For example, the mobile phone can determine the third candidate names with the corresponding confidence levels greater than the preset confidence threshold as the real names of various foods to be ingested.

[0188] Step X2, determine the portions of various foods to be ingested in the second image.

[0189] Exemplarily, the second image can be a depth image. Based on this, the mobile phone can respectively determine the areas of the regions where various foods to be ingested are located according to the outlines of various foods to be ingested obtained by performing object detection on the second image, and respectively determine the portions of various foods to be ingested according to the volumes and depth information of the regions where various foods to be ingested are located.

[0190] Step X3, respectively determine the calories corresponding to the unit portions of various foods to be ingested according to the real names of various foods to be ingested in the second image, and determine the total calories of all the food to be ingested in the second image according to the portions of various foods to be ingested and the calories corresponding to the unit portions.

[0191] Exemplarily, the mobile phone can query the calories corresponding to the unit portions of various foods to be ingested from the food ingredient library according to the real names of various foods to be ingested in the second image, and determine the calories of each food to be ingested as the product of the portion of each food and the calories corresponding to the unit portion, and determine the total calories of all the food to be ingested in the second image as the sum of the calories of all the food to be ingested.

[0192] S609, the mobile phone obtains a third image containing the remaining food and determines the total calories of the remaining food in the third image.

[0193] Optionally, the third image can be taken by the user using a mobile phone after finishing eating. Based on this, as Figure 11 shown, the calorie management APP can also be configured with a second image adding control 73 for adding an image of the remaining food. The user can click on the second image adding control 73 to control the mobile phone to obtain the third image containing the remaining food from the photo album.

[0194] Optionally, the third image can be taken by the user using smart glasses after finishing eating and sent by the smart glasses to the mobile phone.

[0195] Exemplarily, the mobile phone can determine the total calorie of all the remaining food in the third image through the following steps K1 to K3:

[0196] Step K1, determine the real names of various remaining foods in the third image.

[0197] Exemplarily, the mobile phone can perform object detection on the third image to obtain the contours of various remaining foods in the third image, the fourth candidate names, and the confidence levels corresponding to the fourth candidate names, and determine the real names of various remaining foods according to the fourth candidate names of various remaining foods in the third image and the confidence levels corresponding to the fourth candidate names. For example, the mobile phone can determine the fourth candidate names with corresponding confidence levels greater than the preset confidence level threshold as the real names of various remaining foods.

[0198] Step K2, determine the portions of various remaining foods in the third image.

[0199] Exemplarily, the third image can be a depth image. Based on this, the mobile phone can respectively determine the areas of the regions where various remaining foods are located according to the contours of various remaining foods obtained by performing object detection on the third image, and respectively determine the portions of various remaining foods according to the volumes and depth information of the regions where various remaining foods are located.

[0200] Step K3, respectively determine the calories corresponding to the unit portions of various remaining foods according to the real names of various remaining foods in the third image, and determine the total calorie of all the remaining foods in the third image according to the portions of various remaining foods and the calories corresponding to the unit portions.

[0201] Exemplarily, the mobile phone can query the calories corresponding to the unit portions of various remaining foods from the food ingredient library according to the real names of various remaining foods in the third image, determine the calorie of each remaining food as the product of the portion of each food and the calorie corresponding to the unit portion, and determine the total calorie of all the remaining foods in the third image as the sum of the calories of all the remaining foods.

[0202] S610, the mobile phone determines the total second calorie intake of the user by taking the difference between the total calories of all the foods to be ingested in the second image and the total calories of all the remaining foods in the third image.

[0203] S611, based on the number of times various foods to be put into the mouth are swallowed, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the total second calorie intake of the user detected in multiple discontinuous time periods, the mobile phone adjusts the user's single default swallowing volume.

[0204] It can be understood that, when the number of times various foods to be put into the mouth are swallowed, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the total second calorie intake of the user are all known, the user's single actual swallowing volume can be determined accordingly.

[0205] To make the user's single default swallowing volume more matched to the user, the mobile phone can adjust the user's single default swallowing volume according to the determined single actual swallowing volume. For example, the mobile phone can use the single actual swallowing volume as the new single default swallowing volume, so that the user's calorie intake information can be determined more accurately based on the adjusted single default swallowing volume in the future.

[0206] To improve the matching degree between the user's single default swallowing volume and the user's actual swallowing situation, as Figure 12 shown, in another alternative implementation, the method for detecting calorie intake information may further include S612 - S613, which are described in detail as follows:

[0207] S612, the mobile phone determines the total third calorie intake of the user according to the portions of various foods to be ingested input by the user and the calories corresponding to the unit portions of various foods to be ingested.

[0208] In specific applications, the user can weigh each food to be ingested before eating to obtain the portions of various foods to be ingested, and can input the portions of each food to be ingested in the calorie management APP of the mobile phone. The portions of the foods to be ingested can include, for example, weight or volume, etc. It should be noted that the user interface and manual operation steps involved in S612 can refer to Figure 1 the relevant descriptions of the corresponding text part, which will not be elaborated here.

[0209] S613, based on the number of times various foods to be put into the mouth are swallowed, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the total third calorie intake of the user detected in multiple discontinuous time periods, the mobile phone adjusts the user's single default swallowing volume.

[0210] It should be noted that the adjustment process of the single default swallowing volume involved in S613 is similar to the adjustment process of the single default swallowing volume involved in S611. Therefore, for the specific implementation process of S613, reference can be made to the relevant description in S611, which will not be elaborated here.

[0211] In some other embodiments of Application Scenario 1, in addition to the data collection function and the data transmission function, the wearable device 20 may further have a data processing function, that is, in this embodiment, the terminal device 10 can be used only as a display device.

[0212] Based on this, optionally, in Figure 2 the case where the detection system of the calorie intake information shown only includes one wearable device 20, this wearable device 20 can have a data processing function, a data transmission function, and all the above data collection functions. For example, assuming that the wearable device 20 is smart glasses, then as Figure 13 shown in (a) of, a camera module, an audio module, a motion sensor, a processor, and a wireless communication module can be configured in the smart glasses.

[0213] Optionally, in Figure 2 the case where the detection system of the calorie intake information shown includes multiple wearable devices 20, at least one of the multiple wearable devices 20 has a data processing function, each wearable device 20 can have some of the above data collection functions, all the wearable devices 20 together have all the above data collection functions, and all the wearable devices 20 have a data transmission function. For example, assuming Figure 2 the wearable devices 20 in include smart glasses and a smart necklace, then as Figure 13 shown in (b) of, a camera module, a motion sensor, a processor, and a wireless communication module can be configured in the smart glasses. An audio module and a wireless communication module can be configured in the smart necklace.

[0214] Taking the terminal device 10 in the detection system of the calorie intake information as a mobile phone and the wearable device 20 as smart glasses as an example, the detection method of the calorie intake information provided in some other embodiments of Application Scenario 1 will be described below.

[0215] Exemplarily, Figure 14 is a schematic flowchart of the detection method of the calorie intake information provided in some other embodiments of this application's Application Scenario 1. As Figure 14 shown, the detection method of the calorie intake information may include S1401 to S1408, which are described in detail as follows:

[0216] S1401, the smart glasses periodically collect first images based on a preset time interval.

[0217] It should be noted that the acquisition step of the first image in S1401 is the same as that in the above-mentioned S601. Therefore, for the specific implementation process of S1401, reference can be made to the relevant description in the above-mentioned S601, and details will not be elaborated here.

[0218] S1402, the smart glasses perform object detection on each of the first images in sequence.

[0219] It should be noted that the specific process of the smart glasses performing object detection on the first image is similar to the specific process of the mobile phone performing object detection on the first image in the above-mentioned S602. The only difference is the execution entity. Therefore, for the specific implementation process of S1402, reference can be made to the relevant description in the above-mentioned S602, and details will not be elaborated here.

[0220] S1403, when the smart glasses detect that the first image contains the food to be put into the mouth, they start to collect audio data and the motion data of the user's head.

[0221] Exemplarily, the smart glasses can collect audio data through the microphone in its internal audio module.

[0222] Optionally, the above audio data may include the swallowing sound generated when the user ingests the food to be put into the mouth. Optionally, the above audio data may include the chewing sound and swallowing sound generated when the user ingests the food to be put into the mouth.

[0223] Exemplarily, the smart glasses can collect the motion data of the smart glasses through the motion sensors inside it and use the motion data of the smart glasses as the motion data of the user's head. Optionally, the above motion data at least includes the angular velocity data collected by the gyroscope. In addition, it may also include the acceleration data collected by the acceleration sensor.

[0224] S1404, the smart glasses determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time according to the audio data, and determine the head posture of the user when ingesting the food to be put into the mouth according to the motion data of the user's head.

[0225] It should be noted that S1404 is similar to the above-mentioned S605. The only difference is the execution entity. Therefore, for the specific implementation process of S1404, reference can be made to the relevant description in the above-mentioned S605, and details will not be elaborated here.

[0226] S1405, the smart glasses determine the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the head posture of the user when ingesting the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time.

[0227] It should be noted that S1405 is similar to the above-mentioned S606, and the only difference between them is the execution entity. Therefore, the specific implementation process of S1405 can refer to the relevant description in the above-mentioned S606, and will not be elaborated here.

[0228] S1406. The smart glasses determine the calorie intake information of the user based on the actual intake amounts of all the foods to be put into the mouth detected respectively in multiple discontinuous time periods and the calories corresponding to the unit portions of each food to be put into the mouth.

[0229] It should be noted that S1406 is similar to the above-mentioned S607, and the only difference between them is the execution entity. Therefore, the specific implementation process of S1406 can refer to the relevant description in the above-mentioned S607, and will not be elaborated here.

[0230] S1407. The smart glasses send the calorie intake information of the user to the mobile phone.

[0231] S1408. The mobile phone displays the calorie intake information of the user.

[0232] In some other embodiments of Application Scenario 2, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, and thus further improve the accuracy of calorie intake information detection, in an optional implementation manner, the method for detecting calorie intake information may further include steps P1 to P4, which are described in detail as follows:

[0233] Step P1. The smart glasses obtain a second image containing all the foods to be ingested and determine the total calories of all the foods to be ingested in the second image.

[0234] Optionally, the second image may be taken by the smart glasses by the user before eating.

[0235] Optionally, the second image may be taken by the mobile phone by the user before eating and sent to the smart glasses by the mobile phone.

[0236] It should be noted that the specific process of the smart glasses determining the total calories of all the foods to be ingested in step P1 is similar to the specific process of the mobile phone determining the total calories of all the foods to be ingested in the above-mentioned S608, and the only difference between them is the execution entity. Therefore, the specific implementation process of step P1 can refer to the relevant description in the above-mentioned S608, and will not be elaborated here.

[0237] Step P2. The smart glasses obtain a third image containing the remaining foods and determine the total calories of the remaining foods in the third image.

[0238] Optionally, the third image may be taken by the smart glasses by the user after eating.

[0239] Optionally, the third image can be taken by the user with a mobile phone after finishing eating and sent by the mobile phone to the smart glasses.

[0240] It should be noted that the specific process of the smart glasses determining the total calories of all remaining foods in step P2 is similar to the specific process of the mobile phone determining the total calories of all remaining foods in S609 above. The only difference is the execution entity. Therefore, the specific implementation process of step P2 can refer to the relevant description in S609 above and will not be elaborated here.

[0241] Step P3, the smart glasses determine the difference between the total calories of all foods to be ingested in the second image and the total calories of all remaining foods in the third image as the user's second total calorie intake.

[0242] Step P4, the smart glasses adjust the user's single default swallowing volume based on the number of times various foods to be put into the mouth are swallowed, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the user's second total calorie intake, respectively, detected in multiple discontinuous time periods.

[0243] It should be noted that the specific process of the smart glasses adjusting the single default swallowing volume in step P4 is similar to the specific process of the mobile phone adjusting the single default swallowing volume in S611 above. The only difference is the execution entity. Therefore, the specific implementation process of step P4 can refer to the relevant description in S611 above and will not be elaborated here.

[0244] Application scenario two:

[0245] In application scenario two, the method for detecting calorie intake information can be applied to a single wearable device. The wearable device can be a wearable device for wearing on the user's head, such as smart glasses, smart earrings, or smart headphones, etc. Exemplarily, the wearable device can have functions such as data collection, data processing, data transmission, and display. Exemplarily, assuming the wearable device is smart glasses, then as Figure 15 shown, the smart glasses can be configured with a camera module, an audio module, a motion sensor, a processor, a display module, and a wireless communication module, etc.

[0246] Taking the wearable device as smart glasses as an example, the method for detecting calorie intake information provided in some embodiments of application scenario two will be described in detail. Exemplarily, Figure 16 is a schematic flowchart of the method for detecting calorie intake information provided in some embodiments of application scenario two of the present application. As Figure 16 shown, the method for detecting calorie intake information can include S1601 - S1607, which are described in detail as follows:

[0247] S1601, The smart glasses periodically collect the first image based on a preset time interval.

[0248] It should be noted that the process of collecting the first image in S1601 is the same as that in the above-mentioned S601. Therefore, for the specific implementation process of S1601, reference can be made to the relevant description in the above-mentioned S601, and details will not be elaborated here.

[0249] S1602, The smart glasses perform object detection on each of the first images in sequence.

[0250] It should be noted that the specific process of the smart glasses performing object detection on the first image is similar to the specific process of the mobile phone performing object detection on the first image in the above-mentioned S602. The only difference is the execution entity. Therefore, for the specific implementation process of S1602, reference can be made to the relevant description in the above-mentioned S602, and details will not be elaborated here.

[0251] S1603, When the smart glasses detect that the first image contains the food to be put into the mouth, they start to collect audio data and the motion data of the user's head.

[0252] Exemplarily, the smart glasses can collect audio data through the microphone in its internal audio module.

[0253] Optionally, the above audio data may include the swallowing sound generated when the user ingests the food to be put into the mouth. Optionally, the above audio data may include the chewing sound and swallowing sound generated when the user ingests the food to be put into the mouth.

[0254] Exemplarily, the smart glasses can collect the motion data of the smart glasses through the motion sensors inside it and use the motion data of the smart glasses as the motion data of the user's head. Optionally, the above motion data at least includes the angular velocity data collected by the gyroscope. In addition, it may also include the acceleration data collected by the acceleration sensor.

[0255] S1604, The smart glasses determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time based on the audio data, and determine the head posture of the user when ingesting the food to be put into the mouth based on the motion data of the user's head.

[0256] It should be noted that S1604 is similar to the above-mentioned S605. The only difference is the execution entity. Therefore, for the specific implementation process of S1604, reference can be made to the relevant description in the above-mentioned S605, and details will not be elaborated here.

[0257] S1605, The smart glasses determine the actual intake amount of the food to be put into the mouth based on the user's single default swallowing amount, the head posture of the user when ingesting the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time.

[0258] It should be noted that S1605 is similar to the above-mentioned S606, and the only difference between them is the execution entity. Therefore, for the specific implementation process of S1605, reference can be made to the relevant description in the above-mentioned S606, and it will not be elaborated here.

[0259] S1606, the smart glasses determine the calorie intake information of the user according to the actual intake of all foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portion of each food to be put into the mouth.

[0260] It should be noted that S1606 is similar to the above-mentioned S607, and the only difference between them is the execution entity. Therefore, for the specific implementation process of S1606, reference can be made to the relevant description in the above-mentioned S607, and it will not be elaborated here.

[0261] S1607, the smart glasses display the calorie intake information of the user.

[0262] In some other embodiments of Application Scenario 2, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, the calorie intake information detection method may further include the above-mentioned steps P1 to P4. For the specific implementation process of steps P1 to P4, reference can be made to the relevant description in the above-mentioned embodiments, and it will not be elaborated here.

[0263] As can be seen above, in this embodiment, the automatic detection of calorie intake information can be achieved only through a single wearable device, thereby improving the convenience of calorie information detection.

[0264] Application Scenario 3:

[0265] In Application Scenario 3, the calorie intake monitoring method can be applied to Figure 17 the diet calorie information detection system shown in Figure 17 As shown, the calorie intake information detection system may include multiple wearable devices, and the multiple wearable devices at least include wearable devices for wearing on the user's head, such as smart glasses, smart earrings or smart headphones, etc.; in addition, it may also include wearable devices for wearing on other parts, such as smart watches and smart necklaces, etc. The multiple wearable devices combined have functions such as data collection, data processing, data transmission, and display.

[0266] Taking the wearable devices including smart glasses and smart necklaces as an example, where the smart glasses have functions of data and image collection, motion data collection, data processing, data transmission, and display, and the smart necklace has an audio collection function, some embodiments of the calorie intake information detection method provided in Application Scenario 3 of the present application will be described in detail.

[0267] Exemplarily, Figure 18 is a schematic flowchart of a method for detecting calorie intake information provided for some embodiments of Application Scenario 3 of the present application. As Figure 18 shown, the method for detecting calorie intake information may include S1801 to S1808, which are described in detail as follows:

[0268] S1801, the smart glasses periodically collect first images based on a preset time interval.

[0269] It should be noted that the step of collecting the first image in S1801 is the same as that in S601 above. Therefore, for the specific implementation process of S1801, reference may be made to the relevant description in S601 above, and details will not be repeated here.

[0270] S1802, the smart glasses perform object detection on each first image in sequence.

[0271] It should be noted that the specific process of the smart glasses performing object detection on the first image is similar to the specific process of the mobile phone performing object detection on the first image in S602 above. The only difference is the execution entity. Therefore, for the specific implementation process of S1802, reference may be made to the relevant description in S602 above, and details will not be repeated here.

[0272] S1803, when the smart glasses detect that the first image contains food to be put into the mouth, they start collecting the motion data of the user's head and send an audio collection instruction to the smart necklace.

[0273] S1804, the smart necklace starts collecting audio data in response to the audio collection instruction from the smart glasses and sends the collected audio data to the smart glasses.

[0274] S1805, the smart glasses determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time according to the audio data, and determine the head posture of the user when the user intakes the food to be put into the mouth according to the motion data of the user's head.

[0275] It should be noted that S1805 is similar to S605 above. The only difference is the execution entity. Therefore, for the specific implementation process of S1805, reference may be made to the relevant description in S605 above, and details will not be repeated here.

[0276] S1806, the smart glasses determine the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the head posture of the user when the user intakes the food to be put into the mouth, the number of times the food to be put into the mouth is swallowed, and the size of the swallowing sound each time.

[0277] It should be noted that S1806 is similar to the above-mentioned S606, and the only difference between them is the execution entity. Therefore, for the specific implementation process of S1806, reference can be made to the relevant description in the above-mentioned S606, and details will not be elaborated here.

[0278] S1807, the smart glasses determine the calorie intake information of the user according to the actual intake of all foods to be put into the mouth detected respectively in multiple discontinuous time periods and the calories corresponding to the unit portion of each food to be put into the mouth.

[0279] It should be noted that S1807 is similar to the above-mentioned S607, and the only difference between them is the execution entity. Therefore, for the specific implementation process of S1807, reference can be made to the relevant description in the above-mentioned S607, and details will not be elaborated here.

[0280] S1808, the smart glasses display the calorie intake information of the user.

[0281] In some other embodiments of Application Scenario 2, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, the calorie intake information detection method may further include the above-mentioned steps P1 to P4. For the specific implementation process of steps P1 to P4, reference can be made to the relevant description in the above-mentioned embodiments, and details will not be elaborated here.

[0282] Based on the same technical concept, the embodiments of the present application further provide an electronic device, which may be the wearable device in the above-mentioned Application Scenario 2.

[0283] Based on the same technical concept, the embodiments of the present application further provide a computer-readable storage medium, which stores a computer-executable program. When the computer-executable program is called by the electronic device, the electronic device is enabled to execute one or more steps in any of the above method embodiments.

[0284] Based on the same technical concept, the embodiments of the present application further provide a chip system, including a processor, the processor is coupled to a memory, and the processor executes the computer-executable program stored in the memory to implement one or more steps in any of the above method embodiments. The chip system may be a single chip or a chip module composed of multiple chips.

[0285] Based on the same technical concept, the embodiments of the present application further provide a computer-executable program product. When the computer-executable program product runs on the electronic device, the electronic device is enabled to execute one or more steps in any of the above method embodiments.

[0286] Based on the same inventive concept, an embodiment of the present application further provides a detection system for calorie intake information, and the detection system for calorie intake information is the wearable device in the above Application Scenario 1 or Application Scenario 3.

[0287] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or subsequent. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0288] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0289] Those of ordinary skill in the art can understand all or part of the processes of implementing the methods in the above embodiments. These processes can be completed by relevant hardware instructed by a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The aforementioned storage medium includes: various media such as ROM or random access memory RAM, magnetic disks, or optical discs that can store program codes.

[0290] As described above, it is only the specific implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for detecting calorie intake information, characterized in that, Including: Periodically collect a first image through a wearable device worn on the head, and perform object detection on the first image; When it is detected that the first image contains food to be put into the mouth, collect audio data and movement data of the user's head through the wearable device; The audio data at least includes the swallowing sound of the user swallowing the food to be put into the mouth; Determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound each time it is swallowed according to the audio data, and determine the head posture of the user when ingesting the food to be put into the mouth according to the movement data; Determine a first swallowing volume coefficient corresponding to each swallowing according to the size of the swallowing sound each time it is swallowed; Determine the texture type of the food to be put into the mouth; the texture type includes liquid food and non-liquid food; Determine a second swallowing volume coefficient corresponding to each swallowing according to the texture type and the head posture; Determine the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the number of times swallowed, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing; Determine the user's calorie intake information according to the actual intake amounts of all foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portion; The calorie intake information includes a total first calorie intake.

2. The detection method of calorie intake information according to claim 1, characterized in that Determine the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the number of times swallowed, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, including: Determine the actual intake amount of the food to be put into the mouth according to the following formula: ； Wherein, TY sj is the actual intake amount of the food to be put into the mouth, TY mr is the single default swallowing amount of the user, n is the number of times of swallowing, α i is the first swallowing amount coefficient corresponding to the i th swallowing of the food to be put into the mouth, β i is the second swallowing amount coefficient corresponding to the i th swallowing of the food to be put into the mouth.

3. The detection method of calorie intake information according to claim 1, characterized in that Determine the texture type of the food to be put into the mouth, including: Determine the texture type of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by the object detection and the confidence corresponding to the first candidate name, and / or according to the audio data.

4. The detection method of calorie intake information according to any one of claims 1 to 3, characterized in that Determine the user's calorie intake information according to the actual intake amounts of all foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portion, including: For the food to be put into the mouth detected in each time period, when there are multiple types of the food to be put into the mouth, determine the volume ratio of each type of the food to be put into the mouth according to the outlines of various foods to be put into the mouth obtained by the object detection; Calculate the calorie intake corresponding to the food to be put into the mouth in the first image according to the actual intake amount, the volume ratio of each type of the food to be put into the mouth, and the calories corresponding to the unit portion; Determine the sum of the calorie intakes corresponding to all foods to be put into the mouth detected in multiple discontinuous time periods as the total first calorie intake of the user.

5. The method for detecting calorie intake information according to any one of claims 1 to 3, characterized in that, Before determining the user's calorie intake information according to the actual intake amounts of all foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portion, it further includes: Determine the true name of the food to be put into the mouth according to the first candidate name of the food to be put into the mouth obtained by the object detection and the confidence corresponding to the first candidate name; Determine the calories corresponding to the unit portion of the food to be ingested according to the true name of the food to be ingested.

6. The detection method of calorie intake information according to any one of claims 1 to 3, characterized in that The audio data further includes the chewing sound of the user chewing the food to be ingested; correspondingly, before determining the user's calorie intake information according to the actual intake of all the foods to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit portion, it further includes: Determine the second candidate name of the food to be ingested and the confidence level corresponding to the second candidate name according to the audio characteristics of the chewing sound included in the audio data; Determine the true name of the food to be ingested according to the first candidate name of the food to be ingested obtained by the target detection, the confidence level corresponding to the first candidate name, the second candidate name, and the confidence level corresponding to the second candidate name; Determine the calories corresponding to the unit portion of the food to be ingested according to the true name of the food to be ingested.

7. The detection method of calorie intake information according to claim 6, characterized in that Determine the true name of the food to be ingested according to the first candidate name of the food to be ingested obtained by the target detection, the confidence level corresponding to the first candidate name, the second candidate name, and the confidence level corresponding to the second candidate name, including: In the case where the first candidate name is the same as the second candidate name, and the confidence levels corresponding to both the first candidate name and the second candidate name are greater than or equal to the preset confidence threshold, determine either the first candidate name or the second candidate name as the true name of the food to be ingested; In the case where the first candidate name is different from the second candidate name, the confidence level corresponding to the first candidate name is greater than or equal to the preset confidence threshold, and the confidence level corresponding to the second candidate name is less than the preset confidence threshold, determine the first candidate name as the true name of the food to be ingested; In the case where the first candidate name is different from the second candidate name, the confidence level corresponding to the first candidate name is less than the preset confidence threshold, and the confidence level corresponding to the second candidate name is greater than or equal to the preset confidence threshold, determine the second candidate name as the true name of the food to be ingested.

8. The detection method of calorie intake information according to any one of claims 1 to 3, characterized in that, It further includes: Obtain a second image including all the foods to be ingested, and determine the total calories of all the foods to be ingested in the second image; Obtain a third image including the remaining foods, and determine the total calories of the remaining foods in the third image; Determine the difference between the total calories of all the foods to be ingested and the total calories of the remaining foods as the user's second total calorie intake; Adjust the user's single default swallowing volume based on the number of swallowing times of various foods to be ingested detected in multiple discontinuous time periods, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the second total calorie intake.

9. The detection method of calorie intake information according to any one of claims 1 to 3, characterized in that Determine the number of swallowing times of the food to be ingested and the size of the swallowing sound for each swallowing according to the audio data, including: Extract the audio data corresponding to the swallowing sound from the audio data; Determine the number of target peaks appearing in the audio data corresponding to the swallowing sound as the number of times of being swallowed; According to the amplitude size corresponding to each of the target peaks, respectively determine the size of the swallowing sound for each time of being swallowed.

10. An electronic device, characterized in that, The electronic device is a wearable device for wearing on the head; the electronic device is used to execute the detection method of calorie intake information as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when the instructions run on an electronic device, cause the electronic device to execute the detection method of calorie intake information as described in any one of claims 1 to 9.

12. A chip system, characterized in that, The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions to cause the electronic device to execute the detection method of calorie intake information as described in any one of claims 1 to 9.

13. A detection system for calorie intake information, characterized in that, Including a terminal device and at least one wearable device; The wearable device is used to periodically collect a first image and send the first image to the terminal device; The terminal device is used to perform target detection on the first image, and in the case where it is detected that the first image contains food to be put into the mouth, send an eating monitoring instruction to the wearable device; The wearable device is further used to respond to the eating monitoring instruction, start collecting audio data and movement data of the user's head, and send the audio data and the movement data to the terminal device; The terminal device is further used to determine the number of times the food to be put into the mouth is swallowed and the size of the swallowing sound for each time of being swallowed according to the audio data, and determine the head posture of the user when ingesting the food to be put into the mouth according to the movement data; The terminal device is further used to: Determine a first swallowing volume coefficient corresponding to each time of being swallowed according to the size of the swallowing sound for each time of being swallowed; Determine the texture type of the food to be put into the mouth; the texture type includes liquid food and non-liquid food; Determine a second swallowing volume coefficient corresponding to each time of being swallowed according to the texture type and the head posture; Determine the actual intake amount of the food to be put into the mouth according to the user's single default swallowing amount, the number of times of being swallowed, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each time of being swallowed; The terminal device is further used to determine the calorie intake information of the user according to the actual intake amounts of all the foods to be put into the mouth detected in multiple discontinuous time periods and the calories corresponding to the unit portions.

Images