Calorific intake information detection method, electronic equipment and calorie intake information detection system

Abstract

The invention is suitable for the technical field of terminals, and provides a calorie intake information detection method, electronic equipment and a calorie intake information detection system. The method comprises the following steps: periodically acquiring a first image through wearable equipment, and performing target detection on the first image; when it is detected that the first image contains the food to be eaten, audio data and motion data of the head of the user are collected through the wearable device; according to the audio data, determining the number of times of swallowing of the food to be eaten and the swallowing sound volume of each time of swallowing, and according to the motion data, determining a head posture; determining the actual intake of the food to be eaten based on the single default swallowing amount of the user, the head posture, the swallowing times of the food to be eaten and the swallowing sound of each time of swallowing; and the calorie intake information is determined according to the detected actual intake of all the food to be eaten and the calorie corresponding to the unit weight, so that the accuracy of calorie intake information detection can be improved, and the manual operation flow in the calorie intake information detection process is simplified.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method, electronic device and system for detecting calorie intake information. Background Art

[0002] At present, as people's health awareness continues to increase, personal weight management and health management are receiving more and more attention. Reasonable control of the calorie intake of each meal is the key to managing weight and maintaining health. For this reason, more and more people are beginning to calculate and record the calorie intake of each meal through calorie management apps installed on electronic devices such as mobile phones.

[0003] A calorie management APP provided by the related art requires the user to manually input the amount of various foods consumed in each meal (for example, weight, volume or quantity, etc.) when calculating the calorie intake of each meal of the user, and then calculates the calorie intake of each meal of the user based on the calories corresponding to the unit portions of various foods stored in the food library. The operation process is relatively cumbersome, resulting in a poor user experience. Another calorie management APP provided by the related art can automatically identify the total calories of the food to be consumed in the image taken by the user, as the calorie intake of the user for that meal. However, in actual applications, some of the food in the image may not be consumed by the user, resulting in inaccurate calculation of the calorie intake. Summary of the invention

[0004] The embodiments of the present application provide a method, electronic device and system for detecting calorie intake information, which can not only accurately detect the calorie intake information of each meal of the user, but also simplify the manual operation process in the calorie intake information detection process, so that the user has a better experience.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting calorie intake information, comprising: periodically collecting a first image through a wearable device worn on the head, and performing target detection on the first image; when it is detected that the first image contains food to be ingested, collecting audio data and movement data of the user's head through the wearable device; the audio data at least includes the swallowing sound of the user swallowing the food to be ingested; determining the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed based on the audio data, and determining the head posture of the user when ingesting the food to be ingested based on the movement data; determining the actual intake of the food to be ingested based on the user's single default swallowing amount, head posture, number of swallowings, and the volume of the swallowing sound each time it is swallowed; determining the user's calorie intake information based on the actual intake of all foods to be ingested and the calories corresponding to the unit serving size detected in multiple discontinuous time periods; the calorie intake information includes a first total calorie intake.

[0006] Wearable devices worn on the head may include smart glasses, smart earrings, smart necklaces, and smart headphones.

[0007] The food to be eaten refers to the food that has been picked up by the user through tableware (e.g., chopsticks, spoon or fork, etc.) or hands and is about to be put into the mouth. The food to be eaten is usually a small part of all the food to be ingested. Based on this, if the first image contains food in the tableware or in the hands of the person, it can be determined that the first image contains the food to be eaten.

[0008] When a user is eating, different head postures will result in different amounts of food being swallowed each time. Specifically, the amount of food swallowed each time when the user is tilting his head back is usually greater than the amount of food swallowed each time when the user is not tilting his head back, and the greater the angle at which the user tilts his head back, the greater the amount of food swallowed each time.

[0009] Exemplarily, the user's head posture may be described by the pitch angle of the user's head.

[0010] For example, the number of target peaks appearing in the audio data corresponding to the swallowing sound can be determined as the number of times the food to be ingested is swallowed. The target peak can refer to a peak whose corresponding amplitude is greater than a preset decibel threshold.

[0011] For example, the swallowing sound volume of each swallow of the food to be ingested can be determined according to the amplitude corresponding to each target peak value.

[0012] The user's single default swallowing volume can be based on the various body parameters set by the user in the calorie management app, and usually queries the correspondence between the pre-stored various body parameters and the single default swallowing volume. Among them, the body parameters may include gender, age, height, and weight. The single default swallowing volume of users with different body parameters is different.

[0013] The calorie intake information detection method provided in the embodiment of the present application uses a wearable device worn on the head to periodically capture a first image and perform target detection on the first image. When it is detected that the first image contains food to be ingested, audio data and movement data of the user's head are captured by the wearable device, so as to realize automatic monitoring of the sound and head posture made when the user ingests the food to be ingested; the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time is swallowed are determined according to the audio data, the head posture of the user when ingesting the food to be ingested is determined according to the movement data of the user's head, and the actual intake of the food to be ingested is determined by combining the user's single default swallowing amount, the head posture of the user when ingesting the food to be ingested, the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time is swallowed, thereby realizing accurate detection of the user's actual food intake; thereby, the user's calorie intake information can be accurately calculated based on the actual intake of all the food to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit serving size. Since wearable devices can automatically monitor the user's entire eating process and automatically detect the user's calorie intake information, the manual operation process in the calorie intake information detection process is simplified, giving users a better experience.

[0014] In an optional implementation of the first aspect, the actual intake of food to be ingested is determined based on the user's single default swallowing volume, head posture, number of swallowings, and the swallowing sound volume of each swallowing, including: determining a first swallowing volume coefficient corresponding to each swallowing according to the swallowing sound volume of each swallowing; determining the texture type of the food to be ingested; the texture type includes liquid food and non-liquid food; determining a second swallowing volume coefficient corresponding to each swallowing according to the texture type and head posture; determining the actual intake of food to be ingested according to the user's single default swallowing volume, number of swallowings, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing.

[0015] The first swallowing volume coefficient refers to an adjustment factor for adjusting the actual swallowing volume determined based on the swallowing sound. The first swallowing volume coefficient corresponding to each swallowing of the food to be ingested can be determined by querying the corresponding relationship between the swallowing sound volume and the first swallowing volume coefficient based on the swallowing sound volume of the food to be ingested each time.

[0016] The second swallowing volume coefficient refers to an adjustment factor determined based on the head posture for adjusting the actual swallowing volume. The second swallowing volume coefficient corresponding to each swallowing of the food to be ingested can be determined based on the pitch angle of the head when the user ingests the food to be ingested, by querying the pre-stored correspondence between the pitch angle of the head corresponding to the texture type of the food to be ingested and the second swallowing volume coefficient.

[0017] The calorie intake information detection method provided by the present implementation manner determines a first swallowing volume coefficient corresponding to each swallowing of the food to be ingested according to the swallowing sound volume of the food to be ingested each time; and determines a second swallowing volume coefficient corresponding to each swallowing of the food to be ingested according to the head posture of the user when ingesting the food to be ingested and the texture type of the food to be ingested. Thus, the actual swallowing volume of the user each time swallowing the food can be accurately determined based on the first swallowing volume coefficient, the second swallowing volume coefficient and the user's single default swallowing volume, which is conducive to improving the accuracy of calorie intake information detection.

[0018] In an optional implementation of the first aspect, determining the actual intake of the food to be ingested according to the user's single default swallowing amount, the number of swallowings, and the first swallowing amount coefficient and the second swallowing amount coefficient corresponding to each swallowing includes: determining the actual intake of the food to be ingested according to the following formula: ; in, TY sj is the actual amount of food to be ingested, TY mr The default swallowing volume for a single user. n is the number of times swallowed, α i For food to be eaten i The first swallowing volume coefficient corresponding to the swallowing, β i For food to be eaten i The second swallowing volume coefficient corresponding to the second swallowing.

[0019] In an optional implementation of the first aspect, determining the texture type of food to be eaten includes: determining the texture type of food to be eaten based on a first candidate name of the food to be eaten obtained based on target detection and a confidence level corresponding to the first candidate name, and / or determining the texture type of food to be eaten based on audio data.

[0020] Exemplarily, the real name of the food to be eaten can be determined based on the first candidate name of the food to be eaten obtained by target detection of the first image and the confidence level corresponding to the first candidate name; and the texture type of the food to be eaten can be determined based on the real name of the food to be eaten.

[0021] For example, the texture type of the food to be eaten can be determined based on the audio data collected by the smart glasses. For example, if the audio data does not include chewing sounds, the mobile phone can determine that the food to be eaten is liquid food; if the audio data includes chewing sounds, the mobile phone can determine that the food to be eaten is non-liquid food.

[0022] Exemplarily, the texture type of the food to be eaten can be determined in combination with the audio data and the first candidate name of the food to be eaten obtained by target detection of the first image and the confidence corresponding to the first candidate name. For example, the real name of the food to be eaten can be determined first based on the first candidate name of the food to be eaten obtained by target detection of the first image and the confidence corresponding to the first candidate name, and then the texture type of the food to be eaten can be determined based on the audio data and the real name of the food to be eaten. Exemplarily, when the real name of the food to be eaten belongs to the name corresponding to liquid food, and the audio data does not include chewing sounds, the mobile phone can determine that the food to be eaten is liquid food; when the real name of the food to be eaten does not belong to liquid food, or the audio data includes chewing sounds, the mobile phone can determine that the food to be eaten is non-liquid food.

[0023] The calorie intake information detection method provided by the present implementation manner can improve the accuracy of determining the texture type of the food to be ingested by comprehensively determining the texture type of the food to be ingested based on image detection and audio detection.

[0024] In an optional implementation of the first aspect, before determining the user's calorie intake information based on the actual intake of all foods to be eaten and the calories corresponding to a unit serving size detected in multiple discontinuous time periods, it also includes: determining the real name of the food to be eaten based on the first candidate name of the food to be eaten obtained by target detection and the confidence level corresponding to the first candidate name; and determining the calories corresponding to a unit serving size of the food to be eaten based on the real name of the food to be eaten.

[0025] In an optional implementation of the first aspect, the user's calorie intake information is determined based on the actual intake of all foods to be eaten detected in multiple discontinuous time periods and the calories corresponding to the unit serving size, including: for the foods to be eaten detected in each time period, when there are multiple types of foods to be eaten, the volume proportions of various foods to be eaten are determined based on the contours of various foods to be eaten obtained by target detection; the calorie intake corresponding to the foods to be eaten in the first image is calculated based on the actual intake, the volume proportion of each type of food to be eaten and the calories corresponding to the unit serving size; and the sum of the calorie intakes corresponding to all foods to be eaten detected in multiple discontinuous time periods is determined as the user's first total calorie intake.

[0026] Exemplarily, the first image may be a depth image. Based on this, the areas of the regions where the various foods to be eaten are located can be determined according to the contours of the various foods to be eaten obtained by performing target detection on the first image; the volumes of the various foods to be eaten can be determined according to the areas and depth information of the regions where the various foods to be eaten are located; and the ratio of the volume of each food to be eaten to the sum of the volumes of all foods to be eaten can be determined as the volume proportion of each food to be eaten.

[0027] For example, the caloric intake of the food to be ingested can be calculated according to the actual amount of food to be ingested, the volume proportion of each food to be ingested, and the caloric intake per unit serving by the following formula: ; in, RL is the calorie intake corresponding to the food to be eaten, TY sj is the actual amount of food to be ingested, m is the number of types of food to be eaten, μ i For the j The volume ratio of the food to be eaten, Δ RL For the j The calories corresponding to a unit portion of food to be eaten.

[0028] The calorie intake information detection method provided by the present implementation can improve the accuracy of determining the calorie intake corresponding to the food to be eaten, thereby further improving the accuracy of calorie intake information detection, by calculating the calorie intake corresponding to the food to be eaten in the first image based on the actual intake of all the food to be eaten in the first image, the volume proportion of various foods to be eaten, and the calories corresponding to the unit portion, when the first image contains multiple types of food to be eaten.

[0029] In an optional implementation of the first aspect, the audio data also includes the chewing sound of the user chewing the food to be eaten; correspondingly, before determining the user's calorie intake information based on the actual intake of all foods to be eaten and the calories corresponding to the unit serving size detected in multiple discontinuous time periods, it also includes: determining the second candidate name of the food to be eaten and the confidence corresponding to the second candidate name based on the audio features of the chewing sound included in the audio data; determining the real name of the food to be eaten based on the first candidate name of the food to be eaten obtained by target detection, the confidence corresponding to the first candidate name, the second candidate name and the confidence corresponding to the second candidate name; and determining the calories corresponding to the unit serving size of the food to be eaten based on the real name of the food to be eaten.

[0030] The calorie intake information detection method provided in the embodiment of the present application can improve the accuracy of food name recognition by comprehensively determining the name of the food to be ingested based on image detection and audio detection, thereby further improving the accuracy of calorie intake information detection.

[0031] In an optional implementation of the first aspect, the real name of the food to be eaten is determined based on the first candidate name of the food to be eaten obtained by target detection, the confidence corresponding to the first candidate name, the second candidate name, and the confidence corresponding to the second candidate name, including: when the first candidate name is the same as the second candidate name, and the confidence corresponding to the first candidate name and the confidence corresponding to the second candidate name are both greater than or equal to a preset confidence threshold, the first candidate name or the second candidate name is determined as the real name of the food to be eaten; when the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is greater than or equal to the preset confidence threshold, and the confidence corresponding to the second candidate name is less than the preset confidence threshold, the first candidate name is determined as the real name of the food to be eaten; when the first candidate name is different from the second candidate name, the confidence corresponding to the first candidate name is less than the preset confidence threshold, and the confidence corresponding to the second candidate name is greater than or equal to the preset confidence threshold, the second candidate name is determined as the real name of the food to be eaten.

[0032] In an optional implementation of the first aspect, the method for detecting calorie intake information also includes: obtaining a second image containing all foods to be ingested, and determining the total calories of all foods to be ingested in the second image; obtaining a third image containing remaining food, and determining the total calories of the remaining food in the third image; determining the difference between the total calories of all foods to be ingested and the total calories of the remaining food as the user's second total calorie intake; and adjusting the user's single default swallowing amount based on the number of times various foods to be ingested are swallowed detected in multiple discontinuous time periods, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit portion, and the second total calorie intake.

[0033] The calorie intake information detection method provided by the present implementation adjusts the user's single default swallowing amount based on the user's second total calorie intake obtained by image detection, which can improve the match between the user's single default swallowing amount and the user's actual swallowing situation, and is conducive to improving the accuracy of calorie intake information detection.

[0034] In an optional implementation of the first aspect, the number of times the food to be ingested is swallowed and the size of the swallowing sound each time it is swallowed are determined based on audio data, including: extracting audio data corresponding to the swallowing sound from the audio data; determining the number of target peaks appearing in the audio data corresponding to the swallowing sound as the number of swallowing times; and determining the size of the swallowing sound each time it is swallowed based on the amplitude corresponding to each target peak.

[0035] In a second aspect, an embodiment of the present application provides an electronic device, which is a wearable device for being worn on the head; the electronic device is used to execute a method for detecting calorie intake information as in any implementation of the first aspect described above.

[0036] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes the method for detecting calorie intake information as described in any implementation of the first aspect above.

[0037] In a fourth aspect, an embodiment of the present application provides a computer executable program product. When the computer executable program product is run on an electronic device, the electronic device executes the method for detecting calorie intake information of any implementation of the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes a method for detecting calorie intake information such as any implementation method of the first aspect mentioned above.

[0039] In a sixth aspect, an embodiment of the present application provides a system for detecting calorie intake information, including a terminal device and at least one wearable device; The wearable device is used to periodically collect a first image and send the first image to the terminal device; The terminal device is used to perform target detection on the first image, and send a food intake monitoring instruction to the wearable device when detecting that the first image contains food to be ingested; The wearable device is further used to respond to the eating monitoring instruction, start collecting audio data and motion data of the user's head, and send the audio data and the motion data to the terminal device; The terminal device is further used to determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed according to the audio data, and to determine the head posture of the user when taking in the food to be ingested according to the motion data; The terminal device is further used to determine the actual intake amount of the food to be ingested based on the user's single default swallowing amount, the head posture, the number of swallowing times, and the swallowing sound volume of each swallowing; The terminal device is also used to determine the user's calorie intake information based on the actual intake amount of all food to be ingested and the calories corresponding to the unit serving size detected in multiple discontinuous time periods.

[0040] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a user interface involved in calculating calorie intake in a calorie management APP provided in the related art; Figure 2 A schematic diagram of a system for detecting calorie intake information provided in some embodiments of the first application scenario of the present application; Figure 3 A schematic diagram of the structure of a terminal device provided for some embodiments of application scenario 1 of the present application; Figure 4 An architecture diagram of a software system of a terminal device provided for some embodiments of application scenario 1 of the present application; Figure 5 A schematic diagram of the structure of a wearable device provided for some embodiments of application scenario 1 of the present application; Figure 6 A flowchart of a method for detecting calorie intake information provided in some embodiments of the first application scenario of the present application; Figure 7 A schematic diagram of a user interface of a heat management APP provided in an embodiment of the present application; Figure 8 A schematic diagram of possible head postures of a user while eating; Fig. 9 A timing diagram of a first image collected by the smart glasses; Fig.10 A flowchart of a method for detecting calorie intake information provided in some other embodiments of the first application scenario of the present application; Fig.11 A schematic diagram of a user interface of a heat management APP provided in another embodiment of the present application; Fig.12 A flowchart of a method for detecting calorie intake information provided in some other embodiments of the first application scenario of the present application; Fig.13 A schematic diagram of the structure of a wearable device provided for other embodiments of application scenario 1 of the present application; Fig.14 A flowchart of a method for detecting calorie intake information provided in some other embodiments of the first application scenario of the present application; Fig.15 A schematic diagram of the structure of a wearable device provided for some embodiments of the second application scenario of the present application; Fig.16 A flowchart of a method for detecting calorie intake information provided in some embodiments of the second application scenario of the present application; Fig.17 A schematic diagram of a system for detecting calorie intake information provided in some embodiments of the third application scenario of the present application; Fig.18 A flowchart of a method for detecting calorie intake information provided in some embodiments of application scenario three of the present application. DETAILED DESCRIPTION

[0042] It should be noted that the terms used in the implementation methods of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, "at least one" and "one or more" mean one, two or more than two.

[0043] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, it is defined that the "first" and "second" features may explicitly or implicitly include one or more of the features.

[0044] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing at different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0045] Calorie intake refers to the amount of calories that the human body takes in from food through eating, usually measured in calories (cal) or thousand times calories (abbreviated as kilocalories or kcal, kcal). At present, with the continuous enhancement of people's health awareness, personal weight management and health management are receiving more and more attention. Reasonable control of the calorie intake of each meal is the key to managing weight and maintaining health. For this reason, more and more people have begun to calculate and record the calorie intake of each meal through calorie management applications (applications, APPs) installed on electronic devices such as mobile phones or smart watches.

[0046] For example, when calculating the calorie intake of a user for each meal, a calorie management APP provided by related technology requires the user to manually input the amount of various foods consumed in each meal (for example, weight, volume or quantity, etc.), and then calculates the total calorie intake of the user for each meal based on the calories corresponding to the unit portions of various foods stored in the food library.

[0047] Figure 1 A schematic diagram of a user interface involved in calculating calorie intake in a calorie management APP provided in the related art. For example, Figure 1 As shown in FIG. 1 , when the user wants to record the calorie intake of breakfast, the user can control the mobile phone to enter the breakfast recording interface of the calorie management APP. Figure 1 As shown in (a) in the figure, the breakfast record interface A1 of the calorie management APP can be configured with various food addition controls a1. For each food consumed by the user for breakfast, the user can control the mobile phone to display the food intake input interface by clicking the food addition control in the breakfast record interface A1. Figure 1 As shown in (b) in the figure, each food intake input interface B1 is configured with an intake input box b1 and an intake determination control b2. The user can enter the intake of the corresponding food in the intake input box b1 and click the intake determination control b2. When the mobile phone detects that the intake determination control b2 in the intake input interface B1 is clicked, it can calculate the calorie intake corresponding to the food based on the intake of the corresponding food entered by the user in the intake input box b1 and the calories corresponding to the unit serving of the food, and return to the breakfast record interface A1. For example, Figure 1 As shown in (c) in the figure, the breakfast recording interface A1 is also equipped with a calorie recording control a2. After the user has added the portions of all the food consumed for breakfast and obtained the calorie intake corresponding to all the food, he can click the calorie recording control a2 in the breakfast recording interface A1. After the mobile phone detects that the calorie recording control a2 in the breakfast recording interface A1 is clicked, it calculates the total calorie intake of the user's breakfast and displays it as follows: Figure 1 (d) shows the details interface of calorie intake information.

[0048] It can be seen that the above-mentioned calorie intake calculation method requires users to weigh or estimate each food they consume in advance, and manually input the weighing or estimation results into the calorie management APP. The operation process is relatively cumbersome, resulting in a poor user experience; moreover, in the absence of a weighing device, the intake of various foods entered by the user is usually arbitrarily estimated by humans, resulting in inaccurate calorie intake calculation.

[0049] For example, another calorie management app provided by the related art can automatically identify the total calories of all the food to be ingested in the image taken by the user, and use it as the calorie intake of the user for the meal. However, in actual applications, some of the food to be ingested in the image is often not ingested by the user, and the shooting angle of the image will have a great impact on the volume estimation of the food to be ingested, resulting in inaccurate calculation of calorie intake.

[0050] In view of this, in order to accurately detect the calorie intake information of each meal of the user and simplify the manual operation process in the process of calorie intake information detection, so that the user has a better experience. The embodiment of the present application provides a method for detecting calorie intake information, by using a wearable device worn on the head to periodically collect a first image and perform target detection on the first image, when it is detected that the first image contains food to be ingested, the wearable device collects audio data and the motion data of the user's head, so as to realize the automatic monitoring of the sound and head posture of the user when ingesting the food to be ingested; by determining the number of times the food to be ingested is swallowed and the swallowing sound volume of each swallowing according to the audio data, determining the head posture of the user when ingesting the food to be ingested according to the motion data of the user's head, and comprehensively considering the user's single default swallowing amount, the head posture of the user when ingesting the food to be ingested, the number of times the food to be ingested is swallowed and the swallowing sound volume of each swallowing, the actual intake of the food to be ingested is determined, thereby realizing the accurate detection of the actual food intake of the user; thus, the user's calorie intake information can be accurately calculated according to the actual intake of all the food to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit serving. Since wearable devices can automatically monitor the user's entire eating process and automatically detect the user's calorie intake information, the manual operation process in the calorie intake information detection process is simplified, giving users a better experience.

[0051] The calorie intake information detection method provided in the embodiment of the present application can be applied to a plurality of different application scenarios. The calorie intake information detection method provided in the embodiment of the present application is described in detail below in combination with each different application scenario.

[0052] Application scenario 1: In application scenario 1, the method for detecting calorie intake information can be applied to Figure 2 In the detection system of calorie intake information shown in FIG. Figure 2 As shown, the calorie intake information detection system may include a terminal device 10 and at least one wearable device 20. The terminal device 10 and at least one wearable device 20 cooperate with each other to jointly realize the detection, statistics and display of calorie intake information.

[0053] Specifically, when detecting calorie intake information, the terminal device 10 and the wearable device 20 may establish a communication connection. Exemplarily, the communication connection may include a wireless communication connection. The wireless communication connection may include a Bluetooth connection or a wireless local area network (WLAN) (e.g., wireless fidelity (WIFI)) connection, etc. The embodiment of the present application does not limit the communication connection method between the terminal device 10 and the wearable device 20.

[0054] For example, the terminal device 10 may include a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not limit the device form of the terminal device 10 .

[0055] Figure 3 This is a schematic diagram of the structure of a terminal device 10 provided in an embodiment of the present application. Figure 3As shown, the terminal device 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180I, a touch sensor 180J, a bone conduction sensor 180K, an ambient light sensor 180L, etc.

[0056] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0057] The wireless communication function of the terminal device 10 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 10. The wireless communication module 160 can provide solutions for wireless communications including Bluetooth, WLAN, global navigation satellite system (GNSS), nearfield communication technology (NFC), infrared technology (IR), etc. applied to the terminal device 10. Exemplarily, the wireless communication module 160 can be one or more devices integrating at least one communication processing module.

[0058] The camera 193 may be used to capture a still image or a video. Exemplarily, the terminal device 10 may include 1 or N cameras 193, where N is an integer greater than 1.

[0059] The display screen 194 may be used to display images or videos, etc. By way of example, the display screen 194 may include a display panel.

[0060] The terminal device 10 can realize the display function through a GPU, a display screen 194, and an application processor.

[0061] The terminal device 10 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0062] The terminal device 10 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

[0063] The gyro sensor 180B may be used to determine the motion posture of the terminal device 10. In some embodiments, the angular velocity of the terminal device 10 around three axes (ie, the x-axis, the y-axis, and the z-axis) may be determined by the gyro sensor 180B.

[0064] The acceleration sensor 180E can be used to detect the magnitude of acceleration in various directions (generally three axes) of the terminal device 10. When the terminal device 10 is stationary, the magnitude and direction of gravity can be detected.

[0065] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 10, which is different from the position of the display screen 194.

[0066] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key, etc. The terminal device 10 may receive key inputs and generate key signal inputs related to user settings and function controls of the terminal device 10.

[0067] A software system (ie, operating system) may be run on the terminal device 10. The software system may be an Android® system, an iOS® system, a Microsoft® system, a Harmony OS system, or other operating systems.

[0068] Exemplarily, the software system of the terminal device 10 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the layered architecture as an example to exemplify the software system of the terminal device 10 .

[0069] Figure 4 This is an architecture diagram of a software system of a terminal device 10 provided in an embodiment of the present application. Figure 4 As shown, the layered architecture divides the software system of the terminal device 10 into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software system of the terminal device 10 may include an application (APP) layer, an application framework (framework) layer, a runtime (runtime) and system library, and a kernel (kernel) layer.

[0070] Exemplarily, the application layer may include a series of applications, such as a camera APP, a heat management APP, etc.

[0071] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0072] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0073] For example, the wearable device 20 may be a general term for wearable devices developed by applying wearable technology to intelligently design daily wear. Figure 2 In the case where the calorie intake information detection system shown includes only one wearable device 20, the wearable device 20 may be a wearable device for wearing on the user's head, such as smart glasses, smart earrings, or smart headphones. Figure 2 In the case where the calorie intake information detection system shown includes multiple wearable devices 20, the multiple wearable devices 20 may include wearable devices for wearing on other parts in addition to the wearable device for wearing on the user's head. The other parts may refer to parts other than the head, such as the neck or wrist. The wearable devices worn on other parts may include, for example, smart watches or smart necklaces.

[0074] In some embodiments of application scenario 1, the wearable device 20 may have data acquisition and data transmission functions, but not data processing and display functions, that is, in this embodiment, the wearable device 20 may be used only as a sensor. The data acquisition function may include image acquisition, audio acquisition, and motion data acquisition functions.

[0075] Based on this, optionally, Figure 2 In the case where the calorie intake information detection system shown includes only one wearable device 20, the wearable device 20 may have all the above data collection functions. Figure 2 The wearable device 20 in the embodiment is a smart glasses, and the smart glasses may have functions such as image acquisition, audio acquisition, and motion data acquisition. Figure 5 As shown in (a) in FIG. 1 , the smart glasses may be configured with a camera module, an audio module, a motion sensor, and a wireless communication module. The audio module may include a microphone. The motion sensor may include a gyroscope and an acceleration sensor. The wireless communication module may include a Bluetooth communication module and / or a WIFI communication module. The smart glasses may realize an image capture function through a camera module, an audio data collection function through an audio module, a motion data collection function through a motion sensor, and a data transmission function through a wireless communication module.

[0076] Optional, in Figure 2In the case where the calorie intake information detection system shown includes multiple wearable devices 20, each wearable device 20 may have some of the above data collection functions, all wearable devices 20 combined have all of the above data collection functions, and all wearable devices 20 have data transmission functions. Figure 2 The wearable device 20 in the embodiment includes smart glasses and a smart necklace, the smart glasses may have image acquisition function and motion data acquisition function, and the smart necklace may have audio acquisition function. Figure 5 As shown in (b) in FIG, the smart glasses can be equipped with a camera module, a motion sensor, and a wireless communication module. The smart necklace can be equipped with an audio module and a wireless communication module. The smart glasses can realize the shooting function through the camera module, realize the motion data collection function through the motion sensor, and realize the data transmission function through the wireless communication module. The smart necklace can realize the audio data collection function through the audio module, and realize the data transmission function through the wireless communication module.

[0077] In the following, the terminal device 10 in the calorie intake information detection system is taken as a mobile phone, and the wearable device 20 is a pair of smart glasses as an example to illustrate the calorie intake information detection method provided in some embodiments of application scenario one.

[0078] For example, Figure 6 A flowchart of a method for detecting calorie intake information provided in some embodiments of the first application scenario of the present application. Figure 6 As shown, the method for detecting calorie intake information may include S601 to S607, which are described in detail as follows: S601, the smart glasses periodically capture a first image based on a preset time interval and send the first image to a mobile phone.

[0079] In application scenario one, when the user needs to automatically detect the calorie intake information of each meal, the user can wear the smart glasses throughout the meal and control the smart glasses to turn on before eating.

[0080] In an optional implementation, the smart glasses can automatically and periodically capture the first image based on a preset time interval after being turned on, and send the first image to the mobile phone. In this implementation, since the smart glasses can automatically capture images after being turned on, the user does not need to manually perform additional operations, so the convenience of calorie intake information detection can be improved, so that the user has a better experience. In addition, the user can also control the smart glasses to shut down after eating to save the power of the smart glasses and extend the battery life of the smart glasses.

[0081] In another optional implementation, after being turned on, the smart glasses can periodically capture the first image based on a preset time interval and send the first image to the mobile phone when receiving a dietary calorie detection instruction.

[0082] In some embodiments of this implementation, Figure 2 As shown, the smart glasses may be configured with a preset control button 201. The preset control button 201 may be used to control the smart glasses to start or stop dietary calorie detection. Exemplarily, the preset control button 201 may be a mechanical button or a touch button, etc. Based on this, the dietary calorie detection instruction may be issued to the smart glasses by the user by triggering the preset control button 201 on the smart glasses. For example, the user may issue a dietary calorie detection instruction to the smart glasses by triggering the preset control button 201 on the smart glasses before eating. After receiving the dietary calorie detection instruction, the smart glasses may periodically capture the first image based on a preset time interval. In addition, the user may issue a stop detection instruction to the smart glasses by triggering the preset control button 201 on the smart glasses again after finishing eating. After receiving the stop detection instruction, the smart glasses may stop capturing the first image.

[0083] In other embodiments of this implementation, the dietary calorie detection instruction may be sent from a mobile phone to the smart glasses. Figure 7 As shown, the calorie management APP on the mobile phone can be configured with a diet calorie detection control 71, which can be used to control the smart glasses to start or stop diet calorie detection. Specifically, the diet calorie detection control 71 may include Figure 7 The disabled state shown in (a) and Figure 7 When the dietary calorie detection control 71 is in the inactive state, it indicates that dietary calorie detection is not currently being performed; when the dietary calorie detection control 71 is in the active state, it indicates that dietary calorie detection is currently being performed.

[0084] Based on this, users can trigger Figure 7 When the mobile phone detects that the diet calorie detection control 71 in the inactive state is triggered, it can send a diet calorie detection instruction to the smart glasses and switch the diet calorie detection control 71 from the inactive state to the active state. Figure 7 After receiving the dietary calorie detection instruction from the mobile phone, the smart glasses can periodically capture the first image based on a preset time interval. In addition, the user can also trigger the first image after finishing eating. Figure 7The enabled diet calorie detection control 71 shown in (b) in FIG. 1 is shown in FIG. 2 . When the mobile phone detects that the enabled diet calorie detection control 71 is triggered, it can send a stop detection instruction to the smart glasses. After receiving the stop detection instruction from the mobile phone, the smart glasses can stop collecting the first image.

[0085] The preset time interval may be set according to actual needs, for example, the preset time interval may be 0.1 seconds.

[0086] Exemplarily, the smart glasses may send the first images to the mobile phone in the order of the acquisition time of the first images. For example, the smart glasses may send the first image acquired at that time to the mobile phone each time after acquiring a first image.

[0087] S602: The mobile phone performs target detection on each first image in turn.

[0088] The purpose of the mobile phone performing target detection on the first image is to detect whether the first image contains food to be eaten, and if the first image contains food to be eaten, determine the outline of the food to be eaten in the first image, the first candidate name, and the confidence level corresponding to the first candidate name.

[0089] Among them, the food to be eaten refers to the food that has been picked up by the user through tableware (for example, chopsticks, spoon or fork, etc.) or hands and is about to be put into the mouth. It should be noted that the food to be eaten is usually a small part of all the food to be eaten. For example, assuming that the food to be eaten is a bowl of rice, the food to be eaten can be a spoonful of rice scooped from the bowl by the user. For another example, assuming that the food to be eaten includes multiple strawberries, the food to be eaten can be a strawberry picked up by the user from multiple strawberries. That is to say, the first image contains food and the food is in the tableware or the hands of the person, then it is considered that the first image contains the food to be eaten. Based on this, the mobile phone can determine that the first image contains the food to be eaten when it is detected that the first image contains food in the tableware or the hands of the person.

[0090] It is understandable that, in actual applications, the food (i.e., the food to be ingested) that the user puts into the mouth each time may be one type or multiple types. For example, when the food to be ingested is curry rice, the food that the user puts into the mouth each time may contain only rice, only curry, or both rice and curry. Therefore, the food to be ingested contained in the first image may be one type or multiple types.

[0091] Exemplarily, the mobile phone may use a pre-trained object detection model for detecting food to be eaten to perform object detection on the first image. Specifically, the input of the object detection model may be an image, and the output may be the outline of various food to be eaten in the image, the first candidate name, and the confidence level corresponding to the first candidate name.

[0092] Exemplarily, a target detection algorithm may be configured in the target detection model. The target detection algorithm may include, for example, a target detection algorithm based on a region-based convolutional neural network (R-CNN), a target detection algorithm based on a single shot multibox detector (SDD), and a target detection algorithm based on a single observation (you only look once, YOLO).

[0093] Exemplarily, the mobile phone may perform target detection on each first image in sequence according to the order in which the first images are received.

[0094] S603: When the mobile phone detects that the first image contains food to be eaten, the mobile phone sends an eating monitoring instruction to the smart glasses.

[0095] The eating monitoring instruction can be used to instruct the smart glasses to start collecting audio data and the user's head movement data.

[0096] It is understandable that when it is detected that the first image contains food to be eaten, it means that the user is about to put the food to be eaten into the mouth. Exemplarily, the food to be eaten can be liquid food or non-liquid food. Liquid food refers to food that is fluid and can adapt to various container shapes, and can usually be swallowed directly, such as milk, soup, and milkshake. Non-liquid food refers to food that exists in a solid or semi-solid form and usually needs to be chewed before it can be swallowed, such as rice, vegetables, and meat. Based on this, when the food to be eaten is liquid food, the user will swallow the food to be eaten after putting it into the mouth, so a swallowing sound will be generated. When the food to be eaten is non-liquid food, the user will chew and swallow the food to be eaten after putting it into the mouth, so a chewing sound and a swallowing sound will be generated. Therefore, when the mobile phone detects that the first image contains food to be eaten, it controls the smart glasses to start collecting audio data, thereby being able to monitor the swallowing sound and / or chewing sound generated when the user ingests the food to be eaten, so as to facilitate the subsequent accurate calculation of the actual intake of the food to be eaten (i.e. the actual amount of food to be eaten by the user) based on the collected audio data.

[0097] It is also understandable that different head postures of the user during eating may result in different amounts of food being swallowed each time. Figure 8 As shown, when the user is eating, the head may be in Figure 8 The non-head-up state shown in (a) may also be in Figure 8 The head tilted state shown in (b) in FIG. The amount of food swallowed each time when the user tilts his head back is usually greater than the amount of food swallowed each time when the user does not tilt his head back, and the greater the angle at which the user tilts his head back, the greater the amount of food swallowed each time. Therefore, when the mobile phone detects that the first image contains food to be ingested, it controls the smart glasses to start collecting the movement data of the user's head, thereby enabling the monitoring of the user's head posture when ingesting the food to be ingested, so as to facilitate the subsequent more accurate calculation of the actual amount of food ingested in combination with the movement data of the user's head.

[0098] S604, the smart glasses respond to the eating monitoring instruction from the mobile phone, start collecting audio data and the movement data of the user's head, and send the audio data and the movement data of the user's head to the mobile phone.

[0099] Exemplarily, after receiving the eating monitoring instruction, the smart glasses can collect audio data through the microphone in the internal audio module.

[0100] Optionally, when the food to be ingested in the first image is liquid food, the audio data collected by the smart glasses may include swallowing sounds generated when the user ingests the food to be ingested.

[0101] Optionally, when the food to be ingested in the first image is non-liquid food, the audio data collected by the smart glasses may include chewing sounds and swallowing sounds generated when the user ingests the food to be ingested.

[0102] It is understandable that when the smart glasses are worn by the user, the smart glasses are integrated with the user's head, that is, the smart glasses will move with the movement of the user's head, and the posture of the smart glasses will be consistent with the posture of the user's head. Therefore, after the smart glasses receive the eating monitoring instruction, the motion data of the smart glasses can be collected through the motion sensor inside the smart glasses, and the motion data of the smart glasses can be used as the motion data of the user's head. Exemplarily, the above-mentioned motion data at least includes angular velocity data collected by the gyroscope, and in addition, it can also include acceleration data collected by the acceleration sensor.

[0103] For example, after receiving the eating monitoring instruction, the smart glasses can collect audio data and the user's head movement data every first time period, and send the collected audio data and the user's head movement data to the mobile phone in the order of the collection time. The first time period can be set according to actual needs, and this embodiment does not limit it.

[0104] S605, the mobile phone determines the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed according to the audio data, and determines the head posture of the user when taking in the food to be ingested according to the movement data of the user's head.

[0105] It is understandable that the audio data collected by the smart glasses may only include swallowing sounds, or may include chewing sounds and swallowing sounds at the same time. Therefore, after the mobile phone receives the audio data from the smart glasses, it can identify and extract the audio data corresponding to the swallowing sounds from the audio data based on the audio features of the chewing sounds and the audio features of the swallowing sounds, or identify and extract the audio data corresponding to the chewing sounds and the audio data corresponding to the swallowing sounds. Among them, the audio features of the chewing sounds are different from the audio features of the swallowing sounds. It should be noted that the specific differences between the audio features of the chewing sounds and the audio features of the swallowing sounds can be referred to the description in the relevant technology, which will not be described in detail here.

[0106] It can also be understood that since the audio data corresponding to the swallowing sound can accurately reflect the actual swallowing situation of the food to be ingested, for example, the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed, after the mobile phone extracts the audio data corresponding to the swallowing sound, the audio data corresponding to the swallowing sound can be analyzed to determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed from the audio data corresponding to the swallowing sound.

[0107] In a specific embodiment, the mobile phone can determine the number of target peaks appearing in the audio data corresponding to the swallowing sound as the number of times the food to be ingested is swallowed. The target peak may refer to a peak whose corresponding amplitude is greater than a preset decibel threshold. Exemplarily, the preset decibel threshold may be the minimum decibel value of the swallowing sound obtained through big data analysis, and the minimum decibel value can distinguish the swallowing sound from background noise.

[0108] In another specific embodiment, the mobile phone can determine the swallowing sound volume of each swallowing of the food to be ingested according to the amplitude size corresponding to each target peak value.

[0109] In an optional implementation, the head posture may be described by the pitch angle of the head. The pitch angle of the head may refer to the angle at which the head is lowered forward or tilted backward relative to the plane where the body is located. Figure 8In (a), since the user is neither looking up nor looking down, the pitch angle of the user's head may be 0 degrees. Figure 8 In (b), since the user is in a state of looking up, the pitch angle θ of the user's head is greater than 0 degrees. It should be noted that when the user is in a state of looking down (not shown), the pitch angle of the user's head may be less than 0 degrees.

[0110] Based on this, the mobile phone can determine the pitch angle of the user's head when the user ingests the food to be ingested according to the angular velocity data and / or acceleration data in the motion data of the user's head. It should be noted that the specific process of determining the pitch angle according to the angular velocity data and / or acceleration data can be referred to the description in the relevant technology, and the embodiments of the present application will not be described in detail.

[0111] S606, the mobile phone determines the actual intake amount of the food to be ingested according to the user's single default swallowing amount, the user's head posture when taking in the food to be ingested, the number of times the food to be ingested is swallowed, and the volume of the swallowing sound of each swallowing.

[0112] It is understandable that users with different physical parameters usually swallow different amounts of food each time. Physical parameters may include, for example, gender, age, height, and weight. For example, the amount of food swallowed by a middle-aged person each time is usually greater than the amount of food swallowed by an elderly person of the same gender, height, and weight each time, and the amount of food swallowed by a male each time is usually greater than the amount of food swallowed by a female of the same age, height, and weight each time.

[0113] For example, the mobile phone may store the corresponding relationship between various body parameters and the single default swallowing volume. Based on this, the mobile phone may determine the user's single default swallowing volume according to the various body parameters set by the user in the calorie management APP.

[0114] It should be noted that the user's single default swallowing volume is a basic single swallowing volume roughly determined based on the user's body parameters. However, in actual applications, the user's head posture when ingesting the food to be ingested and the size of the swallowing sound of each time the food to be ingested is swallowed will affect the actual swallowing volume corresponding to each swallowing of the food to be ingested. For example, the larger the pitch angle of the user's head when ingesting the food to be ingested, the larger the actual swallowing volume corresponding to each swallowing of the food to be ingested; the smaller the pitch angle of the user's head when ingesting the food to be ingested, the smaller the actual swallowing volume corresponding to each swallowing of the food to be ingested. For another example, the louder the swallowing sound of each swallowing of the food to be ingested, the larger the corresponding actual swallowing volume; the smaller the swallowing sound of each swallowing of the food to be ingested, the smaller the corresponding actual swallowing volume is usually. Based on this, the mobile phone can determine the actual swallowing amount corresponding to each swallowing of the food to be ingested based on the user's single default swallowing amount, combined with the user's head posture when ingesting the food to be ingested and the swallowing sound volume of each swallowing of the food to be ingested, and then accurately calculate the actual intake of the food to be ingested based on the actual swallowing amount corresponding to each swallowing of the food to be ingested.

[0115] Optionally, the mobile phone may store a correspondence between the swallowing sound volume and the first swallowing volume coefficient. The first swallowing volume coefficient may refer to an adjustment factor for adjusting the actual swallowing volume determined based on the swallowing sound volume. Exemplarily, the value of the first swallowing volume coefficient may be less than 1, equal to 1, or greater than 1.

[0116] In actual applications, the first swallowing volume coefficient can be determined by the mobile phone according to the user's food intake and the audio data during the eating process, by prompting the user to eat a certain amount of food and collecting audio data of the user during the eating process when the user first uses the calorie management app. For example, when the user first uses the calorie management app, the mobile phone can prompt the user to drink 100 ml of water and collect audio data of the user during the drinking process. The mobile phone can determine the number of times the water is swallowed and the volume of the swallowing sound each time by analyzing the audio data, and can determine the actual swallowing volume corresponding to each swallowing of water according to the user's water intake, the number of times the water is swallowed, and the volume of the swallowing sound each time it is swallowed, and then determine the ratio of the actual swallowing volume corresponding to each swallowing of water to the user's default swallowing volume as the first swallowing volume coefficient corresponding to the volume of the swallowing sound each time the water is swallowed.

[0117] Based on this, the mobile phone can determine the first swallowing volume coefficient corresponding to each swallowing of the food to be ingested according to the swallowing sound volume of the food to be ingested each time, through the correspondence between the swallowing sound volume and the first swallowing volume coefficient.

[0118] Optionally, the mobile phone may also store a correspondence between the pitch angle of the head and the second swallowing volume coefficient. The second swallowing volume coefficient may refer to an adjustment factor for adjusting the actual swallowing volume determined based on the pitch angle of the head. Exemplarily, the value of the second swallowing volume coefficient may be less than 1, may be equal to 1, or may be greater than 1.

[0119] It is understandable that, since liquid food is easier to swallow than non-liquid food, under the same head posture, the actual swallowing amount corresponding to each liquid food being swallowed is usually greater than the actual swallowing amount corresponding to each non-liquid food being swallowed. Based on this, the mobile phone can store the corresponding relationship between the pitch angle of the head corresponding to liquid food and the second swallowing amount coefficient, and the corresponding relationship between the pitch angle of the head corresponding to non-liquid food and the second swallowing amount coefficient. It should be noted that under the same pitch angle, the second swallowing amount coefficient corresponding to liquid food is greater than the second swallowing amount coefficient corresponding to non-liquid food.

[0120] Based on this, before determining the second swallowing coefficient corresponding to each swallowing of the food to be ingested, the mobile phone can first determine the texture type of the food to be ingested, and then determine the second swallowing coefficient corresponding to each swallowing of the food to be ingested according to the corresponding relationship between the head pitch angle corresponding to the texture type of the food to be ingested and the second swallowing coefficient according to the pitch angle of the head when the user ingests the food to be ingested. The texture type may include liquid food and non-liquid food.

[0121] In an optional implementation, the mobile phone can determine the real name of the food to be eaten based on the first candidate name of the food to be eaten obtained by target detection of the first image and the confidence corresponding to the first candidate name, and determine the texture type of the food to be eaten based on the real name of the food to be eaten. Exemplarily, the mobile phone can determine the first candidate name whose corresponding confidence is greater than or equal to a preset confidence threshold as the real name of the food to be eaten. The preset confidence threshold can be set according to actual needs. For example, the preset confidence threshold can be 98%. Based on this, assuming that the first candidate name of the food to be eaten obtained by target detection of the first image is juice, and the confidence corresponding to the juice is greater than 98%, the mobile phone can determine that the food to be eaten in the first image is juice, and determine that the texture type of the food to be eaten is liquid food.

[0122] In another optional implementation, the mobile phone can determine the texture type of the food to be ingested based on the audio data collected by the smart glasses. For example, if the audio data does not include chewing sounds, the mobile phone can determine that the food to be ingested is liquid food; if the audio data includes chewing sounds, the mobile phone can determine that the food to be ingested is non-liquid food.

[0123] In another optional implementation, the mobile phone can determine the texture type of the food to be eaten in combination with the audio data and the first candidate name of the food to be eaten obtained by target detection of the first image and the confidence corresponding to the first candidate name. Specifically, the mobile phone can first determine the real name of the food to be eaten based on the first candidate name of the food to be eaten obtained by target detection of the first image and the confidence corresponding to the first candidate name, and then determine the texture type of the food to be eaten based on the audio data and the real name of the food to be eaten. Exemplarily, when the real name of the food to be eaten belongs to the name of liquid food and the audio data does not include chewing sounds, the mobile phone can determine that the food to be eaten is liquid food; when the real name of the food to be eaten belongs to the name of non-liquid food, or the audio data includes chewing sounds, the mobile phone can determine that the food to be eaten is non-liquid food. This implementation method comprehensively determines the texture type of the food to be eaten based on image detection and audio detection, which can improve the accuracy of determining the texture type of the food to be eaten.

[0124] Optionally, after determining the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing of the food to be ingested, the mobile phone can determine the actual intake of the food to be ingested based on the user's single default swallowing volume, the number of times the food to be ingested is swallowed, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing of the food to be ingested.

[0125] For example, the mobile phone may use the following formula (1) to determine the actual amount of food to be ingested: ;Formula (1) in, TY sj is the actual amount of food to be ingested, TY mr The default swallowing volume for a single user. n is the number of times the food is swallowed. α i For food to be eaten i The first swallowing volume coefficient corresponding to the swallowing, β i For food to be eaten i The second swallowing volume coefficient corresponding to the second swallowing.

[0126] S607: The mobile phone determines the user's calorie intake information based on the actual intake of all foods to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit servings of each food to be ingested.

[0127] For example, the calorie intake information may include the name, portion, calories of each food consumed by the user, and the user's first total calorie intake, etc. The first total calorie intake is the sum of the calories of all foods consumed by the user.

[0128] It is understandable that the user usually picks up food and puts it into the mouth multiple times during the entire eating process, that is, the entire eating process may include multiple operations of putting the food to be eaten into the mouth. Since the user needs to chew and / or swallow the food to be eaten each time after putting the food to be eaten into the mouth, there will be a certain time interval between each two adjacent operations of putting the food to be eaten into the mouth. Based on this, during the entire eating process, the mobile phone can detect the food to be eaten from the first images captured in multiple discontinuous time periods. Among them, each discontinuous time period can correspond to an operation of putting the food to be eaten into the mouth. Exemplarily, Fig. 9 This is a timing diagram of a first image collected by smart glasses. Fig. 9 As shown, assuming that time period t1 corresponds to an operation in which the user puts the food to be eaten into the mouth, time period t2 corresponds to an operation in which the user chews and / or swallows the food to be eaten, and time period t3 corresponds to an operation in which the user puts the food to be eaten into the mouth again, then the mobile phone can detect the food to be eaten from the first images captured in time periods t1 and t3.

[0129] Based on this, the mobile phone can respectively calculate the actual intake amount of the food to be ingested detected in multiple discontinuous time periods through the above S601 to S606.

[0130] Since the calories corresponding to the unit servings of different foods are different, a food material library can also be configured in the mobile phone. The food material library can be used to store information about various foods. Optionally, the food information may include the name of the food, the texture type, the image, and the calories corresponding to the unit serving size. Exemplarily, for liquid food, the food material library may store the calories corresponding to the unit volume of the food, and the unit volume may be, for example, 100 milliliters (ml); for non-liquid food, the food material library may store the calories corresponding to the unit weight of the food, and the unit weight may be, for example, 100 grams (g). Optionally, for non-liquid food, the food information may also include the audio features of the chewing sound of the food. Exemplarily, the audio features of the chewing sound of various foods may be obtained by pre-collecting and analyzing the chewing sounds of the user when eating various foods.

[0131] For example, the information of various foods stored in the food library may be as shown in Table 1: Table 1

[0132] Based on this, for the food to be eaten detected by the mobile phone in any time period, in an optional implementation, when the type of the food to be eaten is one, the mobile phone can query the calories corresponding to the unit serving of the food to be eaten from the food library according to the real name of the food to be eaten, and then calculate the calorie intake corresponding to the food to be eaten according to the actual intake of the food to be eaten and the calories corresponding to the unit serving. For example, the mobile phone can determine the calorie intake corresponding to the food to be eaten as the product of the actual intake of the food to be eaten and the calories corresponding to the unit serving.

[0133] In another optional implementation, when there are multiple types of food to be eaten, the mobile phone can first determine the volume proportion of various foods to be eaten based on the contours of various foods to be eaten obtained by target detection of the first image; then, according to the real name of each food to be eaten, query the calories corresponding to the unit serving of each food to be eaten from the food library; and calculate the calorie intake corresponding to the food to be eaten in the first image based on the actual intake of the food to be eaten in the first image, the volume proportion of each food to be eaten and the calories corresponding to the unit serving.

[0134] Exemplarily, the first image may be a depth image. Based on this, the mobile phone may determine the area of ​​the region where the various foods to be eaten are located according to the contours of the various foods to be eaten obtained by performing target detection on the first image; determine the volume of the various foods to be eaten according to the area and depth information of the region where the various foods to be eaten are located; and determine the ratio of the volume of each food to be eaten to the sum of the volumes of all foods to be eaten as the volume proportion of each food to be eaten.

[0135] For example, the mobile phone can calculate the calorie intake corresponding to the food to be ingested according to the actual amount of food to be ingested, the volume ratio of each food to be ingested, and the calorie intake corresponding to the unit serving size by the following formula (2): ;Formula (2) in, RL is the calorie intake corresponding to the food to be eaten, TY sj is the actual amount of food to be ingested, m is the number of types of food to be eaten, μ i For the j The volume ratio of the food to be eaten, Δ RL For the j The calories corresponding to a unit portion of food to be eaten.

[0136] Exemplarily, the mobile phone may determine the sum of the calorie intake corresponding to all foods to be ingested detected in multiple discontinuous time periods as the first total calorie intake of the user.

[0137] In an optional implementation, when the audio data collected by the smart glasses does not include audio data corresponding to the chewing sound, the mobile phone can determine the real name of the food to be ingested based on the first candidate name of the food to be ingested obtained by performing target detection on the first image and the confidence level corresponding to the first candidate name. Exemplarily, the mobile phone can determine the first candidate name whose corresponding confidence level is greater than a preset confidence threshold as the real name of the food to be ingested.

[0138] In another optional implementation, when the audio data collected by the smart glasses includes audio data corresponding to chewing sounds, the mobile phone can determine the second candidate name of the food to be eaten and the confidence level corresponding to the second candidate name based on the audio features of the chewing sounds included in the audio data; and can determine the real name of the food to be eaten based on the first candidate name of the food to be eaten, the confidence level corresponding to the first candidate name, the second candidate name, and the confidence level corresponding to the second candidate name obtained by performing target detection on the first image.

[0139] Exemplarily, when the first candidate name is the same as the second candidate name, and the confidence corresponding to the first candidate name and the confidence corresponding to the second candidate name are both greater than or equal to a preset confidence threshold, the mobile phone can determine the first candidate name or the second candidate name as the real name of the food to be eaten.

[0140] Exemplarily, when the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is greater than or equal to a preset confidence threshold, and the confidence corresponding to the second candidate name is less than the preset confidence threshold, the mobile phone can determine the first candidate name as the real name of the food to be eaten.

[0141] Exemplarily, when the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is less than a preset confidence threshold, and the confidence corresponding to the second candidate name is greater than or equal to the preset confidence threshold, the mobile phone can determine the second candidate name as the real name of the food to be eaten.

[0142] After the mobile phone determines the user's calorie intake information, the calorie intake information can be displayed for the user to view.

[0143] As can be seen from the above, the method for detecting calorie intake information provided in this embodiment uses a wearable device worn on the head to periodically collect a first image, and uses a mobile phone to perform target detection on the first image. When it is detected that the first image contains food to be ingested, the wearable device is controlled to collect audio data and movement data of the user's head, so that automatic monitoring of the sound and head posture emitted when the user ingests the food to be ingested can be achieved; the mobile phone is used to determine the number of times the food to be ingested is swallowed and the size of the swallowing sound each time it is swallowed according to the audio data, and the head posture of the user when ingesting the food to be ingested is determined according to the movement data of the user's head, and the actual intake of the food to be ingested is determined by combining the user's single default swallowing amount, the head posture of the user when ingesting the food to be ingested, the number of times the food to be ingested is swallowed, and the size of the swallowing sound each time it is swallowed, thereby achieving accurate detection of the user's actual food intake; thereby, the mobile phone can accurately calculate the user's calorie intake information based on the actual intake of all the food to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit serving. Since wearable devices can automatically monitor the user's entire eating process, mobile phones can automatically detect the user's calorie intake information based on the data collected by the wearable devices, thereby simplifying the manual operation process in the calorie intake information detection process and giving users a better experience.

[0144] In addition, since the smart glasses in this embodiment are only used as sensors, there is no need to configure a processor in the smart glasses, thereby reducing the production cost of the smart glasses.

[0145] In other embodiments of application scenario 1, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, thereby further improving the accuracy of calorie intake information detection, such as Fig.10 As shown, in an optional implementation, the method for detecting calorie intake information may further include S608 to S611, which are described in detail as follows: S608: The mobile phone obtains a second image containing all the food to be ingested, and determines the total calories of all the food to be ingested in the second image.

[0146] The food to be ingested may refer to the food that the user is going to ingest. For example, assuming that the user is going to ingest milk, bread, and eggs for breakfast, the second image may include milk, bread, and eggs.

[0147] Optionally, the second image can be taken by the user using a mobile phone before eating. Fig.11As shown, the calorie management APP may also be configured with a first image adding control 72 for adding images of food to be ingested. The user may control the mobile phone to obtain a second image containing all the food to be ingested from the album by clicking the first image adding control 72.

[0148] Optionally, the second image may be taken by the user through the smart glasses before eating and sent to the mobile phone by the smart glasses.

[0149] Exemplarily, the mobile phone may determine the total calories of all the food to be ingested in the second image through the following steps X1 to X3: Step X1, determining the real names of various foods to be ingested in the second image.

[0150] Exemplarily, the mobile phone can perform target detection on the second image, obtain the outlines of various foods to be ingested in the second image, the third candidate names, and the confidences corresponding to the third candidate names, and determine the real names of the various foods to be ingested according to the third candidate names of the various foods to be ingested in the second image and the confidences corresponding to the third candidate names. For example, the mobile phone can determine the third candidate names whose corresponding confidences are greater than a preset confidence threshold as the real names of the various foods to be ingested.

[0151] Step X2, determining the amount of each food to be ingested in the second image.

[0152] For example, the second image may be a depth image. Based on this, the mobile phone can determine the area of ​​the region where the various foods to be ingested are located according to the contours of the various foods to be ingested obtained by performing target detection on the second image, and determine the amount of the various foods to be ingested according to the volume and depth information of the region where the various foods to be ingested are located.

[0153] Step X3, according to the real names of various foods to be ingested in the second image, respectively determine the calories corresponding to the unit servings of various foods to be ingested, and according to the servings of various foods to be ingested and the calories corresponding to the unit servings, determine the total calories of all foods to be ingested in the second image.

[0154] Exemplarily, the mobile phone can query the calories corresponding to unit servings of various foods to be consumed from the food library based on the real names of the various foods to be consumed in the second image, and determine the calories of each food to be consumed by multiplying the serving size of each food by the calories corresponding to the unit serving, and determine the sum of the calories of all foods to be consumed as the total calories of all foods to be consumed in the second image.

[0155] S609: The mobile phone obtains a third image containing the remaining food, and determines the total calories of the remaining food in the third image.

[0156] Optionally, the third image may be taken by the user through a mobile phone after the user finishes eating. Fig.11 As shown, the calorie management APP may also be configured with a second image adding control 73 for adding an image of remaining food. The user may control the mobile phone to obtain a third image containing remaining food from the album by clicking the second image adding control 73.

[0157] Optionally, the third image can be taken by the user through the smart glasses after the user finishes eating, and sent to the mobile phone by the smart glasses.

[0158] Exemplarily, the mobile phone can determine the total calories of all remaining food in the third image through the following steps K1 to K3: Step K1, determining the real names of various leftover foods in the third image.

[0159] Exemplarily, the mobile phone can perform target detection on the third image, obtain the outlines of various leftover foods in the third image, the fourth candidate names, and the confidences corresponding to the fourth candidate names, and determine the real names of the various leftover foods according to the fourth candidate names of the various leftover foods in the third image and the confidences corresponding to the fourth candidate names. For example, the mobile phone can determine the fourth candidate names whose corresponding confidences are greater than a preset confidence threshold as the real names of the various leftover foods.

[0160] Step K2, determining the amounts of various remaining foods in the third image.

[0161] For example, the third image may be a depth image. Based on this, the mobile phone may determine the areas of the regions where the various leftover foods are located according to the contours of the various leftover foods obtained by performing target detection on the third image, and determine the amounts of the various leftover foods according to the volumes and depth information of the regions where the various leftover foods are located.

[0162] Step K3, according to the real names of various leftover foods in the third image, respectively determine the calories corresponding to the unit servings of various leftover foods, and according to the servings of various leftover foods and the calories corresponding to the unit servings, determine the total calories of all the leftover foods in the third image.

[0163] Exemplarily, the mobile phone can query the calories corresponding to unit servings of various leftover foods from the food library based on the real names of the various leftover foods in the third image, and determine the calories of each leftover food by multiplying the serving size of each food by the calories corresponding to the unit serving, and determine the sum of the calories of all the leftover foods as the total calories of all the leftover foods in the third image.

[0164] S610: The mobile phone determines the difference between the total calories of all the food to be ingested in the second image and the total calories of all the remaining food in the third image as the second total calorie intake of the user.

[0165] S611, the mobile phone adjusts the user's single default swallowing amount based on the number of times various foods to be ingested are swallowed detected in multiple discontinuous time periods, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit serving size, and the user's second total calorie intake.

[0166] It can be understood that when the number of times various foods are swallowed, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit serving and the user's second total calorie intake are all known, the user's actual single swallowing volume can be determined based on this.

[0167] In order to make the user's single default swallowing amount more suitable for the user, the mobile phone can adjust the user's single default swallowing amount according to the determined single actual swallowing amount. For example, the mobile phone can use the single actual swallowing amount as the new single default swallowing amount, so that the user's calorie intake information can be determined more accurately based on the adjusted single default swallowing amount.

[0168] In order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, such as Fig.12 As shown, in another optional implementation, the method for detecting calorie intake information may further include S612 to S613, which are described in detail as follows: S612: The mobile phone determines a third total calorie intake of the user according to the amounts of various foods to be ingested and the calories corresponding to the unit amounts of various foods to be ingested input by the user.

[0169] In a specific application, the user can weigh each food to be ingested before eating to obtain the amount of each food to be ingested, and can enter the amount of each food to be ingested in the calorie management APP on the mobile phone. The amount of food to be ingested may include, for example, weight or volume. It should be noted that the user interface and manual operation steps involved in S612 can be referred to in Figure 1 The description of the corresponding text part will not be repeated here.

[0170] S613, the mobile phone adjusts the user's single default swallowing amount based on the number of times various foods to be ingested are swallowed detected in multiple discontinuous time periods, the first swallowing coefficient and the second swallowing coefficient corresponding to each swallowing, the calories corresponding to the unit serving size, and the user's third total calorie intake.

[0171] It should be noted that the adjustment process of the single default swallowing volume involved in S613 is similar to the adjustment process of the single default swallowing volume involved in S611. Therefore, for the specific implementation process of S613, reference can be made to the relevant description in S611, which will not be repeated here.

[0172] In some other embodiments of application scenario one, the wearable device 20 may have a data processing function in addition to the data collection function and the data transmission function. That is, in this embodiment, the terminal device 10 may be used only as a display device.

[0173] Based on this, optionally, Figure 2 In the case where the calorie intake information detection system shown includes only one wearable device 20, the wearable device 20 may have a data processing function, a data transmission function, and all the above-mentioned data collection functions. For example, assuming that the wearable device 20 is a smart glasses, Fig.13 As shown in (a) in the figure, the smart glasses can be configured with a camera module, an audio module, a motion sensor, a processor and a wireless communication module.

[0174] Optional, in Figure 2 In the case where the calorie intake information detection system shown includes multiple wearable devices 20, at least one of the multiple wearable devices 20 has a data processing function, each wearable device 20 may have the above-mentioned partial data collection function, all wearable devices 20 combined have all the above-mentioned data collection functions, and all wearable devices 20 have a data transmission function. For example, assuming Figure 2 The wearable device 20 includes smart glasses and a smart necklace. Fig.13 As shown in (b) in FIG. 1 , the smart glasses may be equipped with a camera module, a motion sensor, a processor, and a wireless communication module. The smart necklace may be equipped with an audio module and a wireless communication module.

[0175] In the following, the terminal device 10 in the calorie intake information detection system is taken as a mobile phone, and the wearable device 20 is a pair of smart glasses as an example to illustrate the calorie intake information detection method provided by other embodiments of application scenario one.

[0176] For example, Fig.14 A flowchart of a method for detecting calorie intake information provided in some other embodiments of the first application scenario of the present application. Fig.14 As shown, the method for detecting calorie intake information may include S1401 to S1408, which are described in detail as follows: S1401: The smart glasses periodically capture a first image based on a preset time interval.

[0177] It should be noted that S1401 is the same as the first image acquisition step in the above S601. Therefore, the specific implementation process of S1401 can refer to the relevant description in the above S601, which will not be repeated here.

[0178] S1402: The smart glasses perform target detection on each first image in turn.

[0179] It should be noted that the specific process of the smart glasses performing target detection on the first image is similar to the specific process of the mobile phone performing target detection on the first image in the above S602. The two only have different execution entities. Therefore, for the specific implementation process of S1402, please refer to the relevant description in the above S602, which will not be repeated here.

[0180] S1403: When the smart glasses detect that the first image contains food to be eaten, they start collecting audio data and movement data of the user's head.

[0181] Exemplarily, the smart glasses can collect audio data through a microphone in an internal audio module.

[0182] Optionally, the audio data may include swallowing sounds produced when the user ingests the food to be ingested. Optionally, the audio data may include chewing sounds and swallowing sounds produced when the user ingests the food to be ingested.

[0183] Exemplarily, the smart glasses can collect motion data of the smart glasses through the motion sensor inside the smart glasses, and use the motion data of the smart glasses as the motion data of the user's head. Optionally, the above motion data at least includes angular velocity data collected by the gyroscope, and in addition, may also include acceleration data collected by the acceleration sensor.

[0184] S1404, the smart glasses determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed based on the audio data, and determine the head posture of the user when taking in the food to be ingested based on the movement data of the user's head.

[0185] It should be noted that S1404 is similar to the above S605, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1404, reference can be made to the relevant description in the above S605, which will not be repeated here.

[0186] S1405, the smart glasses determine the actual intake of the food to be ingested based on the user's single default swallowing amount, the user's head posture when ingesting the food to be ingested, the number of times the food to be ingested is swallowed, and the volume of the swallowing sound each time it is swallowed.

[0187] It should be noted that S1405 is similar to the above S606, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1405, reference can be made to the relevant description in the above S606, which will not be repeated here.

[0188] S1406, the smart glasses determine the user's calorie intake information based on the actual intake of all foods to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit servings of each food to be ingested.

[0189] It should be noted that S1406 is similar to the above S607, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1406, reference can be made to the relevant description in the above S607, which will not be repeated here.

[0190] S1407, the smart glasses send the user's calorie intake information to the mobile phone.

[0191] S1408, the mobile phone displays the user's calorie intake information.

[0192] In some other embodiments of the second application scenario, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, thereby further improving the accuracy of calorie intake information detection, in an optional implementation, the calorie intake information detection method may further include steps P1 to P4, which are described in detail as follows: Step P1, the smart glasses obtain a second image containing all the food to be ingested, and determine the total calories of all the food to be ingested in the second image.

[0193] Optionally, the second image may be taken by the user through the smart glasses before eating.

[0194] Optionally, the second image can be taken by the user using a mobile phone before eating and sent by the mobile phone to the smart glasses.

[0195] It should be noted that the specific process of the smart glasses determining the total calories of all foods to be ingested in step P1 is similar to the specific process of the mobile phone determining the total calories of all foods to be ingested in the above S608. The two are just different in execution subjects. Therefore, for the specific implementation process of step P1, please refer to the relevant description in the above S608, which will not be repeated here.

[0196] Step P2: the smart glasses obtain a third image containing the remaining food, and determine the total calories of the remaining food in the third image.

[0197] Optionally, the third image can be taken by the user through the smart glasses after the user finishes eating.

[0198] Optionally, the third image can be taken by the user using a mobile phone after finishing eating and sent to the smart glasses by the mobile phone.

[0199] It should be noted that the specific process of the smart glasses determining the total calories of all remaining food in step P2 is similar to the specific process of the mobile phone determining the total calories of all remaining food in the above S609. The two are just different in the execution subjects. Therefore, for the specific implementation process of step P2, please refer to the relevant description in the above S609 and will not be repeated here.

[0200] Step P3: The smart glasses determine the difference between the total calories of all the food to be ingested in the second image and the total calories of all the remaining food in the third image as the second total calorie intake of the user.

[0201] Step P4, the smart glasses adjust the user's single default swallowing amount based on the number of times various foods to be ingested are swallowed detected in multiple discontinuous time periods, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit serving size, and the user's second total calorie intake.

[0202] It should be noted that the specific process of the smart glasses adjusting the single default swallowing volume in step P4 is similar to the specific process of the mobile phone adjusting the single default swallowing volume in the above S611. The two are just different in the execution subjects. Therefore, for the specific implementation process of step P4, please refer to the relevant description in the above S611, which will not be repeated here.

[0203] Application scenario 2: In application scenario 2, the method for detecting calorie intake information can be applied to a single wearable device. The wearable device can be a wearable device for wearing on the user's head, such as smart glasses, smart earrings, or smart headphones. For example, the wearable device can have data collection functions, data processing functions, data transmission functions, and display functions. For example, assuming that the wearable device is smart glasses, then Fig.15 As shown, smart glasses can be configured with a camera module, an audio module, a motion sensor, a processor, a display module, and a wireless communication module.

[0204] The following takes the wearable device as smart glasses as an example to describe in detail the method for detecting calorie intake information provided by some embodiments of the second application scenario. Fig.16 A flowchart of a method for detecting calorie intake information provided in some embodiments of the second application scenario of the present application. Fig.16 As shown, the method for detecting calorie intake information may include S1601 to S1607, which are described in detail as follows: S1601: The smart glasses periodically capture a first image based on a preset time interval.

[0205] It should be noted that S1601 is the same as the first image acquisition step in the above S601. Therefore, the specific implementation process of S1601 can refer to the relevant description in the above S601, which will not be repeated here.

[0206] S1602: The smart glasses perform target detection on each first image in turn.

[0207] It should be noted that the specific process of the smart glasses performing target detection on the first image is similar to the specific process of the mobile phone performing target detection on the first image in the above S602. The two only have different execution entities. Therefore, for the specific implementation process of S1602, please refer to the relevant description in the above S602, which will not be repeated here.

[0208] S1603: When the smart glasses detect that the first image contains food to be eaten, they start collecting audio data and movement data of the user's head.

[0209] Exemplarily, the smart glasses can collect audio data through a microphone in an internal audio module.

[0210] Optionally, the audio data may include swallowing sounds produced when the user ingests the food to be ingested. Optionally, the audio data may include chewing sounds and swallowing sounds produced when the user ingests the food to be ingested.

[0211] Exemplarily, the smart glasses can collect motion data of the smart glasses through the motion sensor inside the smart glasses, and use the motion data of the smart glasses as the motion data of the user's head. Optionally, the above motion data at least includes angular velocity data collected by the gyroscope, and in addition, may also include acceleration data collected by the acceleration sensor.

[0212] S1604, the smart glasses determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed based on the audio data, and determine the head posture of the user when taking in the food to be ingested based on the movement data of the user's head.

[0213] It should be noted that S1604 is similar to the above S605, and the two are just different in execution entities. Therefore, for the specific implementation process of S1604, reference can be made to the relevant description in the above S605, which will not be repeated here.

[0214] S1605, the smart glasses determine the actual intake of the food to be ingested based on the user's single default swallowing amount, the user's head posture when ingesting the food to be ingested, the number of times the food to be ingested is swallowed, and the volume of the swallowing sound each time it is swallowed.

[0215] It should be noted that S1605 is similar to the above S606, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1605, reference can be made to the relevant description in the above S606, which will not be repeated here.

[0216] S1606, the smart glasses determine the user's calorie intake information based on the actual intake of all foods to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit servings of each food to be ingested.

[0217] It should be noted that S1606 is similar to the above S607, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1606, please refer to the relevant description in the above S607, which will not be repeated here.

[0218] S1607, the smart glasses display the user's calorie intake information.

[0219] In other embodiments of application scenario two, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, the calorie intake information detection method may also include the above-mentioned steps P1 to P4. For the specific implementation process of steps P1 to P4, please refer to the relevant description in the above-mentioned embodiments, which will not be repeated here.

[0220] It can be seen from the above that the present embodiment can realize automatic detection of calorie intake information only through one wearable device, thereby improving the convenience of calorie information detection.

[0221] Application scenario three: In application scenario three, the calorie intake monitoring method can be applied to Fig.17 In the detection system of dietary calorie information shown in FIG. Fig.17 As shown, the calorie intake information detection system may include multiple wearable devices, which at least include wearable devices for wearing on the user's head, such as smart glasses, smart earrings or smart headphones, etc. In addition, it may also include wearable devices for wearing on other parts of the body, such as smart watches and smart necklaces, etc. The multiple wearable devices are combined to have data collection functions, data processing functions, data transmission functions, and display functions.

[0222] The following takes the wearable devices including smart glasses and smart necklaces, where the smart glasses have data image acquisition function, motion data acquisition function, data processing function, data transmission function and display function, and the smart necklace has an audio acquisition function as an example, and describes in detail the method for detecting calorie intake information provided in some embodiments of application scenario three of the present application.

[0223] For example, Fig.18A flowchart of a method for detecting calorie intake information provided in some embodiments of the third application scenario of the present application. Fig.18 As shown, the method for detecting calorie intake information may include S1801 to S1808, which are described in detail as follows: S1801: The smart glasses periodically capture a first image based on a preset time interval.

[0224] It should be noted that S1801 is the same as the first image acquisition step in the above S601. Therefore, the specific implementation process of S1801 can refer to the relevant description in the above S601, which will not be repeated here.

[0225] S1802: The smart glasses perform target detection on each first image in turn.

[0226] It should be noted that the specific process of the smart glasses performing target detection on the first image is similar to the specific process of the mobile phone performing target detection on the first image in the above S602. The two only have different execution entities. Therefore, for the specific implementation process of S1802, please refer to the relevant description in the above S602 and will not be repeated here.

[0227] S1803, when the smart glasses detect that the first image contains food to be eaten, they start collecting movement data of the user's head and send audio collection instructions to the smart necklace.

[0228] S1804, the smart necklace starts collecting audio data in response to the audio collection instruction from the smart glasses, and sends the collected audio data to the smart glasses.

[0229] S1805, the smart glasses determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed based on the audio data, and determine the head posture of the user when ingesting the food to be ingested based on the movement data of the user's head.

[0230] It should be noted that S1805 is similar to the above S605, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1805, reference can be made to the relevant description in the above S605, which will not be repeated here.

[0231] S1806, the smart glasses determine the actual intake of the food to be ingested based on the user's single default swallowing amount, the user's head posture when ingesting the food to be ingested, the number of times the food to be ingested is swallowed, and the volume of the swallowing sound each time it is swallowed.

[0232] It should be noted that S1806 is similar to the above S606, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1806, please refer to the relevant description in the above S606, which will not be repeated here.

[0233] S1807, the smart glasses determine the user's calorie intake information based on the actual intake of all foods to be ingested detected in multiple discontinuous time periods and the calories corresponding to the unit servings of each food to be ingested.

[0234] It should be noted that S1807 is similar to the above S607, and the two are just different in execution subjects. Therefore, for the specific implementation process of S1807, please refer to the relevant description in the above S607, which will not be repeated here.

[0235] S1808, the smart glasses display the user's calorie intake information.

[0236] In other embodiments of application scenario two, in order to improve the matching degree between the user's single default swallowing amount and the user's actual swallowing situation, the calorie intake information detection method may also include the above-mentioned steps P1 to P4. For the specific implementation process of steps P1 to P4, please refer to the relevant description in the above-mentioned embodiments, which will not be repeated here.

[0237] Based on the same technical concept, an embodiment of the present application also provides an electronic device, which may be a wearable device in the above-mentioned application scenario two.

[0238] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer executable program. When the computer executable program is called by an electronic device, the electronic device executes one or more steps in any of the above method embodiments.

[0239] Based on the same technical concept, the embodiment of the present application also provides a chip system, including a processor, the processor is coupled to a memory, and the processor executes a computer executable program stored in the memory to implement one or more steps in any of the above method embodiments. The chip system can be a single chip, or a chip module composed of multiple chips.

[0240] Based on the same technical concept, an embodiment of the present application also provides a computer executable program product. When the computer executable program product runs on an electronic device, the electronic device executes one or more steps in any of the above method embodiments.

[0241] Based on the same technical concept, an embodiment of the present application also provides a calorie intake information detection system, which is a wearable device in the above-mentioned application scenario one or application scenario three.

[0242] In the above embodiments, the description of each embodiment has its own emphasis. For the parts not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0243] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0244] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0245] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting calorie intake information, characterized in that: include: The first image is periodically collected by a wearable device worn on the head, and target detection is performed on the first image; When it is detected that the first image contains food to be ingested, collecting audio data and movement data of the user's head through the wearable device; The audio data at least includes the swallowing sound of the user swallowing the food to be ingested; Determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed according to the audio data, and determine the head posture of the user when taking in the food to be ingested according to the motion data; Determining the actual intake of the food to be ingested based on the user's single default swallowing volume, the head posture, the number of swallowing times, and the volume of the swallowing sound of each swallowing; Determine the user's caloric intake information based on the actual intake of all foods to be ingested and the calories corresponding to the unit servings detected in multiple discontinuous time periods; The calorie intake information includes a first calorie intake total amount.

2. The method for detecting calorie intake information according to claim 1, characterized in that: Determining the actual intake amount of the food to be ingested based on the user's single default swallowing amount, the head posture, the number of swallowing times, and the swallowing sound volume of each swallowing includes: Determining a first swallowing volume coefficient corresponding to each swallowing according to the swallowing sound volume of each swallowing; Determine the texture type of the food to be ingested; the texture type includes liquid food and non-liquid food; determining a second swallowing volume coefficient corresponding to each swallowing according to the texture type and the head posture; The actual intake amount of the food to be ingested is determined according to the user's single default swallowing amount, the number of swallowing times, and the first swallowing amount coefficient and the second swallowing amount coefficient corresponding to each swallowing.

3. The method for detecting calorie intake information according to claim 2, characterized in that: Determining the actual intake of the food to be ingested according to the user's single default swallowing volume, the number of swallowings, and the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, including: The actual intake of the food to be ingested is determined according to the following formula: ； in, TY sj is the actual intake of the food to be taken, TY mr The default swallowing volume for a single user. n is the number of swallowings, α i The food to be taken is i The first swallowing volume coefficient corresponding to the swallowing, β i The food to be taken is i The second swallowing volume coefficient corresponding to the swallowing.

4. The method for detecting calorie intake information according to claim 2, characterized in that: Determine the texture type of the food to be ingested, including: The texture type of the food to be ingested is determined based on a first candidate name of the food to be ingested obtained through the target detection and a confidence level corresponding to the first candidate name, and / or based on the audio data.

5. The method for detecting calorie intake information according to any one of claims 1 to 4, characterized in that: The user's caloric intake information is determined based on the actual intake of all foods to be ingested and the calories corresponding to the unit servings detected in multiple discontinuous time periods, including: For the food to be ingested detected in each time period, when there are multiple types of the food to be ingested, the volume proportions of the various types of food to be ingested are determined respectively according to the contours of the various types of food to be ingested obtained by the target detection; Calculating the caloric intake corresponding to the food to be ingested in the first image according to the actual intake, the volume proportion of each type of food to be ingested, and the calorie corresponding to the unit serving; The sum of the calorie intake corresponding to all foods to be ingested detected in multiple discontinuous time periods is determined as the first total calorie intake of the user.

6. The method for detecting calorie intake information according to any one of claims 1 to 4, characterized in that: Before determining the user's calorie intake information based on the actual intake amount of all food to be ingested and the calories corresponding to the unit servings detected in multiple discontinuous time periods, the method further includes: Determining the real name of the food to be ingested according to the first candidate name of the food to be ingested obtained by the target detection and the confidence level corresponding to the first candidate name; The calories corresponding to a unit serving of the food to be eaten are determined according to the real name of the food to be eaten.

7. The method for detecting calorie intake information according to any one of claims 1 to 4, characterized in that: The audio data also includes the chewing sound of the user chewing the food to be ingested; correspondingly, before determining the user's calorie intake information based on the actual intake amount of all the food to be ingested and the calories corresponding to the unit serving size detected in multiple discontinuous time periods, it also includes: Determining a second candidate name of the food to be ingested and a confidence level corresponding to the second candidate name according to audio features of the chewing sound included in the audio data; Determining the real name of the food to be ingested according to the first candidate name of the food to be ingested, the confidence level corresponding to the first candidate name, the second candidate name and the confidence level corresponding to the second candidate name obtained by the target detection; The calories corresponding to a unit serving of the food to be eaten are determined according to the real name of the food to be eaten.

8. The method for detecting calorie intake information according to claim 7, characterized in that: Determining the real name of the food to be ingested according to the first candidate name of the food to be ingested obtained by the target detection, the confidence level corresponding to the first candidate name, the second candidate name, and the confidence level corresponding to the second candidate name, includes: When the first candidate name is the same as the second candidate name, and the confidence level corresponding to the first candidate name and the confidence level corresponding to the second candidate name are both greater than or equal to a preset confidence threshold, determining the first candidate name or the second candidate name as the real name of the food to be ingested; When the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is greater than or equal to the preset confidence threshold, and the confidence corresponding to the second candidate name is less than the preset confidence threshold, determining the first candidate name as the real name of the food to be ingested; When the first candidate name is different from the second candidate name, and the confidence corresponding to the first candidate name is less than the preset confidence threshold, and the confidence corresponding to the second candidate name is greater than or equal to the preset confidence threshold, the second candidate name is determined to be the real name of the food to be ingested.

9. The method for detecting calorie intake information according to any one of claims 1 to 4, characterized in that: Also includes: Acquire a second image containing all the food to be ingested, and determine the total calories of all the food to be ingested in the second image; Acquire a third image containing remaining food, and determine the total calories of the remaining food in the third image; Determine the difference between the total calories of all the food to be ingested and the total calories of the remaining food as the second total calorie intake of the user; The user's single default swallowing amount is adjusted based on the number of times various foods to be ingested are swallowed detected in multiple discontinuous time periods, the first swallowing volume coefficient and the second swallowing volume coefficient corresponding to each swallowing, the calories corresponding to the unit serving size and the second total calorie intake.

10. The method for detecting calorie intake information according to any one of claims 1 to 4, characterized in that: Determining the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time the food is swallowed according to the audio data includes: extracting audio data corresponding to the swallowing sound from the audio data; Determine the number of target peaks appearing in the audio data corresponding to the swallowing sound as the number of swallowing times; The magnitude of the swallowing sound of each swallowing is determined according to the amplitude corresponding to each target peak value.

11. An electronic device, characterized in that: The electronic device is a wearable device for being worn on the head; the electronic device is used to execute the method for detecting calorie intake information as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes the method for detecting caloric intake information according to any one of claims 1 to 10.

13. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the calorie intake information detection method as described in any one of claims 1 to 10.

14. A calorie intake information detection system, characterized in that: comprising a terminal device and at least one wearable device; The wearable device is used to periodically collect a first image and send the first image to the terminal device; The terminal device is used to perform target detection on the first image, and send a food intake monitoring instruction to the wearable device when detecting that the first image contains food to be ingested; The wearable device is further used to respond to the eating monitoring instruction, start collecting audio data and motion data of the user's head, and send the audio data and the motion data to the terminal device; The terminal device is further used to determine the number of times the food to be ingested is swallowed and the volume of the swallowing sound each time it is swallowed according to the audio data, and to determine the head posture of the user when taking in the food to be ingested according to the motion data; The terminal device is further used to determine the actual intake amount of the food to be ingested based on the user's single default swallowing amount, the head posture, the number of swallowing times, and the swallowing sound volume of each swallowing; The terminal device is also used to determine the user's calorie intake information based on the actual intake amount of all food to be ingested and the calories corresponding to the unit serving size detected in multiple discontinuous time periods.

Images