Diabetic diet data acquisition method and system based on Internet of Things

Through the Internet of Things technology combined with deep learning models, using food weight data and image features, the problem of cumbersome and inaccurate dietary data records in diabetic patients is solved, automatic collection and accurate identification of dietary data is achieved, and the effectiveness of health management is improved.

CN119007938AInactive Publication Date: 2024-11-22HEFEI NO 3 PEOPLES HOSPITAL
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411198344.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Diabetic patients have cumbersome processes when recording dietary data and are prone to missing, resulting in inaccurate data and affecting health management.

Method used

The dietary data acquisition method of diabetic patients based on the Internet of Things is adopted. By obtaining the weight data and images of food, the deep learning model is used to extract features, and the dietary data is collected in combination with weight data and image features to improve the accuracy and reliability of the data.

Benefits of technology

It realizes automatic collection and recording of diet data, improves the accuracy and reliability of diet data identification, ensures that diabetic patients can accurately monitor their diet and help with healthy management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007938A_ABST
    Figure CN119007938A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a diabetic diet data acquisition method and system based on the Internet of Things, and relates to the technical field of diet management and data acquisition. The method comprises the steps of obtaining weight data corresponding to target food and a to-be-recognized food image; performing feature extraction on the to-be-identified food image by using the first feature extraction network to obtain surface image features; performing first depth feature extraction processing on the surface image features through a second feature extraction network to obtain first depth image features, and performing second depth feature extraction processing on the surface image features through a third feature extraction network to obtain second depth image features; determining feature weights corresponding to the first depth image feature and the second depth image feature by using an adaptive network; determining diet data features according to the first depth image features, the second depth image features and corresponding feature weights; and executing a diet data acquisition task based on the weight data and the diet data characteristics.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of diet management and data collection, and in particular to a method and system for collecting diet data of diabetic patients based on the Internet of Things. Background Art

[0002] Diabetic patients must pay special attention to controlling the amount of sugar they consume in their daily diet, and also pay attention to the total amount of food they eat and the balance of nutrition. In order to stay healthy, diabetic patients should try to choose light foods and avoid consuming too much high-fat foods. Therefore, it is particularly important for diabetic patients to strictly monitor their diet data.

[0003] In real life, diabetic patients usually use manual recording to track their dietary data. For example, they may record the type and quantity of each food in detail by taking notes. However, this manual recording method is not only cumbersome, but also prone to various problems. For example, patients may forget to record the intake of certain foods for various reasons, or make mistakes during the recording process, or even have deviations when estimating the amount of food. These problems will lead to inaccurate dietary data, which in turn affects the health management of diabetic patients.

[0004] Based on this, it is necessary to study a more scientific and simple dietary data collection method and system to monitor the dietary habits of diabetic patients. Summary of the invention

[0005] To solve the above problems, one aspect of an embodiment of the present application provides a method for collecting dietary data of diabetic patients based on the Internet of Things, the method comprising: Obtaining weight data of a target food and a food image to be identified corresponding to the target food; Using the first feature extraction network in the feature recognition model, performing surface feature extraction processing on the to-be-recognized food image corresponding to the target food to obtain surface image features; Performing a first deep feature extraction process on the surface image features through the second feature extraction network in the feature recognition model to obtain a first deep image feature, and performing a second deep feature extraction process on the surface image features through the third feature extraction network in the feature recognition model to obtain a second deep image feature; wherein the first deep feature extraction process focuses on extracting features from partitioned food images, and the second deep feature extraction process focuses on extracting features from mixed food images; Determine, by using an adaptive network in the feature recognition model, feature weights corresponding to the first depth image feature and the second depth image feature according to the surface image feature; Determining a dietary data feature reflected by the to-be-recognized food image according to the first depth image feature, the second depth image feature, and feature weights corresponding to the first depth image feature and the second depth image feature respectively; A diet data collection task is performed based on the weight data and the diet data characteristics.

[0006] In some embodiments, the weight data of the target food includes first weight data before consumption and second weight data after consumption, and the to-be-recognized food image corresponding to the target food includes at least one first state image collected before consumption and at least one second state image collected after consumption; The dietary data features reflected by the food image to be identified include the first dietary data features reflected by the at least one first state image and the second dietary data features reflected by the at least one second state image; The performing of the diet data collection task based on the weight data and the diet data characteristics includes: performing the diet data collection task based on the difference between the first weight data and the second weight data, and the difference between the first diet data characteristics and the second diet data characteristics.

[0007] In some embodiments, when the second weight data and / or the second state image are not collected, the diet data collection task is performed based on preset second weight data and / or preset second diet data features.

[0008] In some embodiments, performing the diet data collection task based on the difference between the first weight data and the second weight data, and the difference between the first diet data feature and the second diet data feature, includes: A first feature vector is obtained based on a difference between the first weight data and the second weight data, and a second feature vector is obtained based on a difference between a feature of the first diet data and a feature of the second diet data; fusing the first feature vector and the second feature vector to obtain a fused feature vector; The fused feature vector is input into a trained diet data recognition model for processing to obtain target diet data, wherein the target diet data includes at least one type of ingested food and its corresponding quantity.

[0009] In some embodiments, the dietary data recognition model is trained based on the following method: Acquire a sample fusion feature vector and label information corresponding to the sample fusion feature vector, wherein the label information includes the food type and food quantity corresponding to the sample fusion feature vector; Inputting the sample fusion feature vector into an initial diet data recognition model to obtain an initial prediction result; Calculating the difference between the initial prediction result and the label information to obtain an error signal, and adjusting the parameters of the initial diet data recognition model according to the error signal to reduce the error signal; When the error signal is less than a preset threshold, a trained diet data recognition model is obtained.

[0010] In some embodiments, the first feature extraction network is trained based on a first sample set, the first sample set includes partitioned food images and mixed food images, and the first feature extraction network is used to perform surface feature extraction processing on the partitioned food images and / or mixed food images to obtain surface training image features; The second feature extraction network is trained based on a second sample set, the second sample set includes partitioned food images, and the second feature extraction network is used to perform deep feature extraction processing on the surface training image features output by the first feature extraction network to obtain first deep training image features; The third feature extraction network is trained based on a third sample set, the third sample set at least includes mixed food images, and the third feature extraction network is used to perform deep feature extraction processing on the surface training image features output by the first feature extraction network to obtain second deep training image features; The adaptive network is used to determine the feature weights corresponding to the first deep training image features output by the second feature extraction network and the second deep training image features output by the third feature extraction network, respectively, according to the surface training image features output by the first feature extraction network; The feature recognition model is constructed based on the first feature extraction network, the second feature extraction network, the third feature extraction network and the adaptive network.

[0011] In some embodiments, the feature recognition model and the diet data recognition model are obtained through collaborative training, and the collaborative training process includes: adjusting the parameters of the feature recognition model and the initial diet data recognition model according to the error signal to reduce the error signal.

[0012] In some embodiments, the feature recognition model includes m third feature extraction networks, where m is an integer greater than or equal to 1; when m is greater than 1, the m third feature extraction networks respectively focus on feature extraction of mixed food images with different mixing methods, and the m third feature extraction networks are trained based on a large number of mixed food images with different mixing methods; The using the adaptive network in the feature recognition model to determine the feature weights corresponding to the first depth image feature and the second depth image feature respectively according to the surface image feature includes: Through the adaptive network, multidimensional feature weights are determined according to the surface image features, wherein the multidimensional feature weights include feature weights corresponding to the first depth image features and feature weights corresponding to m second depth image features, and the m second depth image features are obtained by feature extraction through m third feature extraction networks.

[0013] Another aspect of the embodiment of the present application further provides a diabetic patient diet data collection system based on the Internet of Things, the system comprising: An acquisition module, used to acquire weight data of a target food and a food image to be identified corresponding to the target food; A surface feature extraction module, used to perform surface feature extraction processing on the to-be-recognized food image corresponding to the target food using the first feature extraction network in the feature recognition model, to obtain surface image features; a deep feature extraction module, configured to perform a first deep feature extraction process on the surface image features through a second feature extraction network in the feature recognition model to obtain a first deep image feature, and perform a second deep feature extraction process on the surface image features through a third feature extraction network in the feature recognition model to obtain a second deep image feature; wherein the first deep feature extraction process focuses on feature extraction of partitioned food images, and the second deep feature extraction process focuses on feature extraction of mixed food images; a feature weight determination module, configured to determine, by using an adaptive network in the feature recognition model, feature weights corresponding to the first depth image feature and the second depth image feature according to the surface image feature; a dietary data feature determination module, configured to determine the dietary data feature reflected by the to-be-recognized food image according to the first depth image feature, the second depth image feature, and feature weights corresponding to the first depth image feature and the second depth image feature respectively; A diet data collection module is used to perform a diet data collection task based on the weight data and the diet data characteristics.

[0014] In some embodiments, the weight data of the target food includes first weight data before consumption and second weight data after consumption, and the to-be-recognized food image corresponding to the target food includes at least one first state image collected before consumption and at least one second state image collected after consumption; The dietary data features reflected by the food image to be identified include the first dietary data features reflected by the at least one first state image and the second dietary data features reflected by the at least one second state image; The dietary data collection module is specifically used for: A first feature vector is obtained based on a difference between the first weight data and the second weight data, and a second feature vector is obtained based on a difference between a feature of the first diet data and a feature of the second diet data; fusing the first feature vector and the second feature vector to obtain a fused feature vector; The fused feature vector is input into a trained diet data recognition model for processing to obtain target diet data, wherein the target diet data includes at least one type of ingested food and its corresponding quantity.

[0015] The beneficial effects that the method and system for collecting dietary data of diabetic patients based on the Internet of Things provided in the embodiments of the present application may bring include at least: (1) By combining the weight data corresponding to the target food and the dietary data features determined based on the food image to be identified corresponding to the target food to perform the dietary data collection task, the weight data of the target food can be integrated on the basis of image recognition and used as a constraint condition in the model recognition process, thereby improving the accuracy and reliability of the recognition results of the dietary data recognition model; (2) The user's dietary data is collected by first weight data and first state image before consumption and second weight data and second state image after consumption, thereby further improving the accuracy and reliability of dietary data collection; (3) performing a first deep feature extraction process on the surface image features of the partitioned food images through a second feature extraction network, and performing a second deep feature extraction process on the surface image features of the mixed food images through a third feature extraction network, so that the feature recognition model can accurately extract both the deep features of the partitioned food images and the deep features of the mixed food images; (4) By training the third feature extraction network together with the surface training image features corresponding to the partitioned food images and the mixed food images, the third feature extraction network can learn the deep image features of the mixed food images while enriching the third feature extraction network's deep feature extraction capability for the mixed food images by learning the deep features of the partitioned food images, thereby improving the deep feature extraction effect of the third feature extraction network for the mixed food images.

[0016] Additional features will be described in part in the following description. For those skilled in the art, it will become apparent by reviewing the following and accompanying drawings, or it may be understood by the production or operation of the examples. The features of this specification may be implemented and obtained by practicing or using various aspects of the methods, tools, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein: Figure 1 is a schematic diagram of an exemplary application scenario of a diabetes patient diet data collection system based on the Internet of Things according to some embodiments of this specification; Figure 2 is an exemplary module diagram of a diabetes patient diet data collection system based on the Internet of Things according to some embodiments of this specification; Figure 3 is an exemplary flow chart of a method for collecting dietary data of diabetic patients based on the Internet of Things according to some embodiments of this specification; Figure 4 It is an exemplary structural diagram of a feature recognition model shown in some embodiments of this specification. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, this specification can also be applied to other similar scenarios based on these drawings. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0019] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0020] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0021] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0022] In order to solve the problem that the process of recording dietary data for diabetic patients is cumbersome and easy to miss, the embodiment of the present application provides a method and system for collecting dietary data for diabetic patients based on the Internet of Things to realize automatic collection and recording of dietary data. Specifically, through the method and system provided in the embodiment of the present application, the user can collect relevant data on food before and after eating (for example, collect weight data and image data), and then process the weight data and image data through a deep learning model, so as to identify the type of food intake and the corresponding quantity of the diabetic patient, and realize the collection of dietary data.

[0023] The following is a detailed description of the method and system for collecting dietary data of diabetic patients based on the Internet of Things provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0024] Figure 1 This is a schematic diagram of an exemplary application scenario of a diabetes patient diet data collection system based on the Internet of Things according to some embodiments of this specification.

[0025] Reference Figure 1 In some embodiments, the application scenario 100 of the diabetes patient diet data collection system based on the Internet of Things may include a data collection device 110, a storage device 120, a processing device 130, a terminal device 140, and a network 150. The various components in the application scenario 100 can be connected in a variety of ways to form an Internet of Things system. For example, the data collection device 110 can be connected to the storage device 120 and / or the processing device 130 through the network 150, or directly connected to the storage device 120 and / or the processing device 130. For another example, the storage device 120 can be directly connected to the processing device 130 or connected through the network 150. For another example, the terminal device 140 can be connected to the storage device 120 and / or the processing device 130 through the network 150, or directly connected to the storage device 120 and / or the processing device 130.

[0026] The data collection device 110 can be used to collect data related to target food, which can refer to food that the diabetic patient is about to eat, such as breakfast, lunch, dinner, cakes, fruits, etc. Figure 1In an embodiment of the present application, the data acquisition device 110 may include a weight acquisition device 111 (e.g., a weight sensor) and an image acquisition device 112 (e.g., a camera), wherein the weight acquisition device 111 may be used to acquire weight data of the target food, and the image acquisition device 112 may be used to acquire food images corresponding to the target food. In some embodiments, the weight acquisition device 111 and the image acquisition device 112 may be integrated or independent of each other, and may have a common power supply or an independent power supply, and send the acquired relevant data to other components (e.g., storage device 120, processing device 130, terminal device 140) in the application scenario 100 via wired or wireless (e.g., Bluetooth, WiFi, etc.) means. In some embodiments, the application scenario 100 may include multiple (e.g., two or more) image acquisition devices 112, and the multiple image acquisition devices 112 may acquire food images from different perspectives for the target food.

[0027] In some embodiments, the data acquisition device 110 may send the collected relevant data (e.g., the weight data of the target food collected by the weight acquisition device 111, and the food image collected by the image acquisition device 112) to the storage device 120, the processing device 130, the terminal device 140, etc. through the network 150. In some embodiments, the relevant data collected by the data acquisition device 110 may be processed by the processing device 130. For example, the processing device 130 may determine the dietary data features reflected by the food image based on the dietary data features, and then perform the dietary data collection task according to the dietary data features and the weight data collected by the weight acquisition device 111.

[0028] The network 150 may facilitate the exchange of information and / or data. The network 150 may include any suitable network capable of facilitating the exchange of information and / or data for the application scenario 100. In some embodiments, at least one component of the application scenario 100 (e.g., the data acquisition device 110, the storage device 120, the processing device 130, the terminal device 140) may exchange information and / or data with at least one other component in the application scenario 100 via the network 150. For example, the processing device 130 may obtain relevant data collected for the target food from the data acquisition device 110 and / or the storage device 120 via the network 150. For another example, the processing device 130 may obtain user operation instructions from the terminal device 140 via the network 150, and exemplary operation instructions may include but are not limited to accessing relevant data of the target food, reading the food type determined based on the relevant data and its corresponding quantity, etc.

[0029] In some embodiments, the network 150 may be any form of wired or wireless network, or any combination thereof. As an example only, the network 150 may include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, etc., or any combination thereof. In some embodiments, the network 150 may include at least one network access point, and at least one component of the application scenario 100 may be connected to the network 150 through the access point to exchange data and / or information.

[0030] The storage device 120 may store data, instructions, and / or any other information. In some embodiments, the storage device 120 may store data obtained from the data acquisition device 110, the processing device 130, and / or the terminal device 140. For example, the storage device 120 may store relevant data collected by the data acquisition device 110; for another example, the storage device 120 may store data calculated by the processing device 130. In some embodiments, the storage device 120 may store data and / or instructions used by the processing device 130 to execute or use to complete the exemplary methods described in this specification. In some embodiments, the storage device 120 may include a large-capacity memory, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. Exemplary large-capacity memories may include magnetic disks, optical disks, solid-state disks, etc. In some embodiments, the storage device 120 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.

[0031] In some embodiments, the storage device 120 may be connected to the network 150 to communicate with at least one other component in the application scenario 100 (e.g., the data acquisition device 110, the processing device 130, the terminal device 140). At least one component in the application scenario 100 may access data, instructions, or other information stored in the storage device 120 through the network 150. In some embodiments, the storage device 120 may be directly connected or communicate with one or more components in the application scenario 100 (e.g., the data acquisition device 110, the terminal device 140). In some embodiments, the storage device 120 may be part of the data acquisition device 110 and / or the processing device 130.

[0032] The processing device 130 can process data and / or information obtained from the data acquisition device 110, the storage device 120, the terminal device 140 and / or other components of the application scenario 100. In some embodiments, the processing device 130 can obtain relevant data collected for the target food from any one or more of the data acquisition device 110, the storage device 120 or the terminal device 140, and determine the type of food consumed by the user and its corresponding quantity by processing the relevant data. In some embodiments, the processing device 130 can obtain pre-stored computer instructions from the storage device 120, and execute the computer instructions to implement the diabetic patient diet data collection method based on the Internet of Things described in this specification.

[0033] In some embodiments, the processing device 130 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 130 may be local or remote. For example, the processing device 130 may access information and / or data from the data acquisition device 110, the storage device 120, and / or the terminal device 140 via the network 150. For another example, the processing device 130 may be directly connected to the data acquisition device 110, the storage device 120, and / or the terminal device 140 to access information and / or data. In some embodiments, the processing device 130 may be implemented on a cloud platform. For example, a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cloud-to-cloud cloud, a multi-cloud, etc., or any combination thereof.

[0034] The terminal device 140 can receive, send and / or display data. The received data may include data collected by the data acquisition device 110, data stored in the storage device 120, data processed by the processing device 130, etc. For example, the data received and / or displayed by the terminal device 140 may include relevant data collected by the data acquisition device 110, the food type and its corresponding quantity determined by the processing device 130 based on the relevant data, etc. The sent data may include user input data and instructions, etc. For example, the terminal device 140 may send the operation instructions input by the user to the data acquisition device 110 through the network 150 to control the data acquisition device 110 to perform corresponding data processing (such as data screening, data supplementation, data modification, etc.).

[0035] In some embodiments, the terminal device 140 may include a mobile device 141, a tablet computer 142, a laptop computer 143, etc., or any combination thereof. For example, the mobile device 141 may include a mobile phone, a personal digital assistant (PDA), a dedicated mobile terminal, etc., or any combination thereof. In some embodiments, the terminal device 140 may include an input device (such as a keyboard, a touch screen), an output device (such as a display, a speaker), etc. In some embodiments, the processing device 130 may be a part of the terminal device 140.

[0036] It should be noted that the above description of the application scenario 100 is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the application scenario 100 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. For example, the data acquisition device 110 may include more or fewer functional components.

[0037] Figure 2 Schematic diagram of a module of a diabetes patient diet data collection system based on the Internet of Things according to some embodiments of this specification. Figure 2 The Internet of Things-based diabetes patient diet data collection system 200 shown can be applied to Figure 1 The application scenario 100 shown, for example, can be configured in the form of software and / or hardware to the processing device 130 and / or the terminal device 140 to process the relevant data collected by the data acquisition device 110 for the target food, and determine the food type and its corresponding quantity based on the relevant data.

[0038] Reference Figure 2 In some embodiments, the Internet of Things-based diabetes patient diet data collection system 200 may include an acquisition module 210, a surface feature extraction module 220, a deep feature extraction module 230, a feature weight determination module 240, a diet data feature determination module 250 and a diet data collection module 260.

[0039] The acquisition module 210 may be used to acquire weight data of a target food and a food image to be identified corresponding to the target food.

[0040] The surface feature extraction module 220 can be used to perform surface feature extraction processing on the to-be-recognized food image corresponding to the target food using the first feature extraction network in the feature recognition model to obtain surface image features.

[0041] The depth feature extraction module 230 can be used to perform a first depth feature extraction process on the surface image features through the second feature extraction network in the feature recognition model to obtain a first depth image feature, and perform a second depth feature extraction process on the surface image features through the third feature extraction network in the feature recognition model to obtain a second depth image feature; wherein the first depth feature extraction process focuses on feature extraction of partitioned food images, and the second depth feature extraction process focuses on feature extraction of mixed food images.

[0042] The feature weight determination module 240 may be configured to utilize an adaptive network in the feature recognition model to determine feature weights corresponding to the first depth image feature and the second depth image feature, respectively, according to the surface image feature.

[0043] The dietary data feature determination module 250 can be used to determine the dietary data features reflected by the food image to be identified based on the first depth image features, the second depth image features, and the feature weights corresponding to the first depth image features and the second depth image features, respectively.

[0044] The dietary data collection module 260 may be configured to perform a dietary data collection task based on the weight data and the dietary data characteristics.

[0045] For more details about the above modules, please refer to other places in this manual (for example Figure 3~Figure 4 part and its related description) will not be repeated here.

[0046] It should be understood that Figure 2The illustrated diabetes patient diet data collection system 200 based on the Internet of Things and its modules can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art can understand that the above-mentioned method and system can be implemented using computer executable instructions and / or included in a processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software (e.g., firmware).

[0047] It should be noted that the above description of the diabetes patient diet data collection system 200 based on the Internet of Things is provided for illustrative purposes only and is not intended to limit the scope of this specification. It is understandable that for those skilled in the art, according to the description of this specification, without departing from this principle, the various modules can be arbitrarily combined, or a subsystem can be formed to connect with other modules. For example, Figure 2 The acquisition module 210, the surface feature extraction module 220, the deep feature extraction module 230, the feature weight determination module 240, the diet data feature determination module 250 and the diet data acquisition module 260 described in the above can be different modules in a system, or a module can realize the functions of two or more of the above modules. Such variations are within the scope of protection of this specification. In some embodiments, the aforementioned modules can be part of the processing device 130 and / or the terminal device 140.

[0048] Figure 3 300 is an exemplary flow chart of a method for collecting dietary data of a diabetic patient based on the Internet of Things according to some embodiments of this specification. In some embodiments, the method 300 for collecting dietary data of a diabetic patient based on the Internet of Things can be executed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on a processing device to perform hardware simulation), etc. or any combination thereof. In some embodiments, Figure 3 One or more operations in the flowchart of the method 300 for collecting dietary data of a diabetic patient based on the Internet of Things can be performed by Figure 1 The processing device 130 and / or the terminal device 140 shown are implemented. For example, the method 300 for collecting dietary data of diabetic patients based on the Internet of Things can be stored in the storage device 120 in the form of instructions or computer programs, and called and / or executed by the processing device 130 and / or the terminal device 140. The following describes the execution process of the method 300 for collecting dietary data of diabetic patients based on the Internet of Things by taking the processing device 130 as an example.

[0049] Reference Figure 3 In some embodiments, the method 300 for collecting dietary data of a diabetic patient based on the Internet of Things may include: Step S310 , obtaining weight data of a target food and a to-be-recognized food image corresponding to the target food. In some embodiments, step S310 may be performed by the obtaining module 210 .

[0050] In an embodiment of the present application, relevant data of the target food that the diabetic patient is about to eat can be collected by the data collection device 110. For example, the weight data of the target food can be collected by the weight collection device 111, and the image data of the target food can be collected by the image collection device 112.

[0051] In some embodiments, the relevant data collected by the data acquisition device 110 for the target food can be stored in the storage device 120, and the acquisition module 210 can acquire the weight data of the target food and the food image to be identified corresponding to the target food from the storage device 120. In some embodiments, the acquisition module 210 can be in communication connection with the weight acquisition device 111 and / or the image acquisition device 112, and the acquisition module 210 can directly acquire the weight data of the target food and / or the food image to be identified corresponding to the target food from the weight acquisition device 111 and / or the image acquisition device 112.

[0052] In some embodiments, the user may manually input weight data corresponding to the target food through the terminal device 140 and transmit it to the storage device 120 for storage.

[0053] Step S320 , using the first feature extraction network in the feature recognition model to perform surface feature extraction processing on the to-be-recognized food image corresponding to the target food, to obtain surface image features. In some embodiments, step S320 may be performed by the surface feature extraction module 220 .

[0054] In an embodiment of the present application, the feature recognition model may be a deep learning model, such as a convolutional neural network (CNN), which includes operations such as convolution calculation, nonlinear activation function (Relu) calculation, and pooling calculation. The first feature extraction network in the model can be used to identify and extract surface features in food images. In this specification, the surface features may include visual attributes such as color, shape, texture, and spatial structure of food, and these surface features may be expressed as a multidimensional vector.

[0055] In some embodiments, the food image to be identified corresponding to the target food may include one or more images. When the food image to be identified corresponding to the target food is multiple images (the multiple images can be acquired from different perspectives), the surface image features can be obtained by feature splicing and / or fusion. For example, the first feature extraction network can be used to extract features from each image, and then feature fusion can be performed by averaging or the like. For another example, when the multiple images have different feature dimensions, the surface image features can be obtained by feature splicing.

[0056] Step S330, performing a first depth feature extraction process on the surface image feature through the second feature extraction network in the feature recognition model to obtain a first depth image feature, and performing a second depth feature extraction process on the surface image feature through the third feature extraction network in the feature recognition model to obtain a second depth image feature. In some embodiments, step S330 can be performed by the depth feature extraction module 230.

[0057] In an embodiment of the present application, in order to improve the recognition accuracy of food features in food images to be identified, two independent and complementary feature extraction networks can be deployed in the feature recognition model, namely the second feature extraction network and the third feature extraction network.

[0058] After obtaining the surface image features in the food image to be identified through the first feature extraction network, the surface image features extracted by the first feature extraction network can be further subjected to a first deep feature extraction process through the second feature extraction network in the feature recognition model to obtain a first deep image feature, and the surface image features extracted by the first feature extraction network can be subjected to a second deep feature extraction process through the third feature extraction network in the feature recognition model to obtain a second deep image feature. Among them, the first deep feature extraction process mainly focuses on performing deep feature extraction on partitioned food images, and the second deep feature extraction process mainly focuses on performing deep feature extraction on mixed food images. It should be noted that, in the embodiment of the present application, the aforementioned partitioned food image may refer to a food image collected when multiple foods are stored in partitions (i.e., independent of each other and not mixed with each other), or a food image collected when only one type of food exists, and a mixed food image refers to a food image collected when multiple foods are mixed with each other.

[0059] It can be understood that when there are multiple foods and the multiple foods are mixed with each other, there may be image recognition errors caused by food occlusion or coverage, but partitioned food images do not have this problem. Therefore, the two types of food images may have different feature extraction methods. In an embodiment of the present application, a first deep feature extraction process is performed on the surface image features of the partitioned food image through a second feature extraction network, and a second deep feature extraction process is performed on the surface image features of the mixed food image through a third feature extraction network. This allows the feature recognition model to accurately extract both the depth features of the partitioned food image and the depth features of the mixed food image.

[0060] It should also be noted that, in the embodiments of the present application, the surface feature extraction processing refers to extracting high-dimensional surface image features from the food image to be identified of the input feature recognition model. The surface image features include rich multi-dimensional spatial information, which is a description of the detailed content of each part of the food image to be identified, and can characterize the characteristics of the food image to be identified to a certain extent, and the surface image features often have a larger size; relatively, the deep feature extraction processing refers to further feature extraction based on the surface image features obtained by the surface feature extraction. The deep image features obtained by the deep feature extraction can contain rich semantic information, which is an abstract description of the overall characteristics of the food image to be identified, and can more accurately reflect the characteristics reflected by the food image to be identified, and the size of the deep image features is usually relatively small.

[0061] Reference Figure 4In an embodiment of the present application, the second feature extraction network and the third feature extraction network can share a first feature extraction network, that is, the second feature extraction network and the third feature extraction network are both used to perform deep feature extraction processing on the surface image features output by the first feature extraction network.

[0062] Step S340 , using the adaptive network in the feature recognition model, determine the feature weights corresponding to the first depth image feature and the second depth image feature according to the surface image feature. In some embodiments, step S340 may be performed by the feature weight determination module 240 .

[0063] In an embodiment of the present application, the adaptive network can be used to determine the feature weights corresponding to the first deep training image features output by the second feature extraction network and the second deep training image features output by the third feature extraction network based on the surface training image features output by the first feature extraction network.

[0064] Specifically, in the embodiment of the present application, the adaptive network can be regarded as a high-level network in the feature recognition model, which can determine the probability that the food image to be recognized is a partitioned food image and a mixed food image according to the surface image features, and use the probabilities corresponding to the partitioned food image and the mixed food image as the feature weights corresponding to the first depth image feature and the second depth image feature. It should be noted that the sum of the feature weight value corresponding to the first depth image feature and the feature weight value corresponding to the second depth image feature is 1.

[0065] Step S350, determining the dietary data features reflected by the food image to be identified based on the first depth image features, the second depth image features, and the feature weights corresponding to the first depth image features and the second depth image features, respectively. In some embodiments, step S350 can be performed by the dietary data feature determination module 250.

[0066] Reference Figure 4 After obtaining the feature weights corresponding to the first depth image feature and the second depth image feature respectively through the aforementioned adaptive network, the first depth image feature and the second depth image feature can be feature fused based on the feature weights to obtain the dietary data features reflected by the food image to be identified.

[0067] It should be noted that, in the process of feature fusion, the first depth image feature and the second depth image feature can be fused according to their corresponding feature weights. For example, when the weight value corresponding to the first depth image feature is 0.2 and the weight value corresponding to the second depth image feature is 0.8, it is considered that the adaptive network determines that the probability of the food image to be identified as a partitioned food image is 20% and the probability of being a mixed food image is 80% based on the surface image features. Then, when performing feature fusion, since the probability of the food image to be identified as a mixed food image is high, more second depth image features are included in the dietary data feature, that is, the feature fusion is performed at a ratio of 80% of the second depth image feature and 20% of the first depth image feature, and finally the dietary data feature corresponding to the food image to be identified is obtained.

[0068] In some embodiments, the dietary data features output by the aforementioned feature recognition model can be expressed as: T = α1×T1+α2×T2, wherein T is the dietary data feature corresponding to the food image to be identified, T1 is the first depth image feature, α1 is the feature weight corresponding to the first depth image feature, T2 is the second depth image feature, and α2 is the feature weight corresponding to the second depth image feature.

[0069] Through the above method, the first feature extraction network, the second feature extraction network, the third feature extraction network and the adaptive network can be combined to obtain a feature recognition model, and through the feature recognition model, reliable image features can be extracted for partitioned food images and mixed food images, thereby ensuring the reliability and accuracy of subsequent dietary data collection tasks.

[0070] Step S360 , executing a diet data collection task based on the weight data and the diet data characteristics. In some embodiments, step S360 may be executed by the diet data collection module 260 .

[0071] After obtaining the dietary data features reflected by the food image to be identified through the above steps, it can be combined with the weight data collected by the weight collection device 111, so as to perform the dietary data collection task based on the weight data and the dietary data features.

[0072] Specifically, in some embodiments, the weight data can be fused with the dietary data features, and the fused feature vector obtained after the fusion is input into the trained dietary data recognition model for processing, thereby obtaining the target dietary data. In the embodiment of the present application, the target dietary data can be used to reflect and record the dietary conditions of the diabetic patient, which may include at least one type of food intake and its corresponding quantity.

[0073] In an embodiment of the present application, the diet data recognition model may be a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), for processing and identifying a fusion feature vector composed of diet data features and the weight data. The diet data recognition model is trained based on the following method: First, a large number of sample fusion feature vectors and their corresponding label information are obtained, where the label information includes the food type (which may include one or more foods) and the food quantity corresponding to each sample fusion feature vector; in this embodiment of the present application, the sample fusion feature vector can be used to identify the sample food image through the above-mentioned feature recognition model, and then be fused with the weight data of the sample food in the sample food image.

[0074] Furthermore, the sample fusion feature vector can be input into the initial diet data recognition model to obtain an initial prediction result; then, the difference between the initial prediction result and the label information is calculated to obtain an error signal (i.e., loss LOSS), and the parameters of the initial diet data recognition model are adjusted according to the error signal to reduce the error signal. Exemplarily, in some embodiments, the parameters of the initial diet data recognition model can be optimized based on back propagation and gradient descent algorithms. The optimization process of the model can be regarded as prior art and will not be discussed in detail in this specification.

[0075] When the error signal is less than the preset threshold, it indicates that the model has good data recognition and generalization capabilities. At this time, the training can be terminated to obtain a trained diet data recognition model. In the embodiment of the present application, the preset threshold can be set according to actual needs, and this specification does not limit it.

[0076] It can be understood that in the embodiment of the present application, by incorporating the weight data of the food, it can be used as a constraint condition in the model recognition process, thereby improving the accuracy and reliability of the recognition results of the dietary data recognition model.

[0077] After being trained with a large amount of dietary image data and weight data, it can accurately identify the type and quantity of food, thereby providing accurate dietary records for diabetic patients.

[0078] In practical applications, the diet data collection module 260 can be combined with a mobile application so that diabetic patients can conveniently record their diet. For example, in some embodiments, the image acquisition device 112 can be a mobile phone used by diabetic patients, and the weight acquisition device 111 can be an Internet of Things electronic scale. When the patient collects the weight data of food through the Internet of Things electronic scale and takes photos of the food and uploads them to the application, the diet data collection module 260 can automatically process the food image data, extract the diet data features, and combine them with the weight data, and finally output the target diet data through the diet data recognition model.

[0079] It should be noted that in the embodiments of the present application, the aforementioned weight data refers to the net weight of the food. In some embodiments, the weight data of the target food can be collected with a designated container, and the designated container has a preset weight. After the weight collection device 111 collects the weight data of the container, the preset weight corresponding to the designated container can be subtracted from it to obtain the weight data (i.e., the net weight) corresponding to the target food. In some embodiments, the container type can be identified by the following identification method, and then the corresponding container weight can be subtracted to obtain the weight data corresponding to the target food.

[0080] In some embodiments, considering that the user may not have eaten all the food, it may not be possible to accurately monitor the user's diet by only collecting weight data and food images once. Therefore, in order to achieve more accurate diet data collection, data collection can be performed once before and after eating, and then the user's diet can be determined based on the data collected twice.

[0081] Specifically, in some embodiments, the weight data of the aforementioned target food may include first weight data before consumption and second weight data after consumption, and the food image to be identified corresponding to the aforementioned target food includes at least one first state image collected before consumption and at least one second state image collected after consumption.

[0082] In some embodiments, the first dietary data feature reflected by the at least one first state image can be determined, and the second dietary data feature reflected by the at least one second state image can be determined. The specific determination process of the first dietary data feature and the second dietary data feature can be referred to above and will not be repeated here.

[0083] In some embodiments, a dietary data collection task can be performed once based on the first dietary data feature and the first weight data collected before consumption, and then the dietary data collection task can be performed again based on the second dietary data feature and the second weight data collected after use, and finally the type of food consumed by the user and its corresponding quantity are determined based on the difference between the two collected data.

[0084] In some embodiments, the user can select a corresponding data type between collected weight data and food image data (for example, the first type can be selected for data collection before eating, and the second type can be selected for data collection after eating), and use the data type as identification information to facilitate matching of the first weight data, second weight data, first state image, and second state image corresponding to the target food in subsequent processes.

[0085] In some embodiments, for the same target food, a first feature vector can be obtained based on the difference between the corresponding first weight data and the second weight data, and a second feature vector can be obtained based on the difference between the corresponding first diet data feature and the second diet data feature. Furthermore, the first feature vector and the second feature vector can be fused to obtain a fused feature vector. Finally, the target diet data can be obtained by inputting the fused feature vector into a trained diet data recognition model for processing. In an embodiment of the present application, the target diet data may include at least one type of ingested food and its corresponding quantity (e.g., weight).

[0086] Considering that the user may forget to collect the aforementioned second weight data and second state image after eating, in order to avoid the inability to complete the diet data collection task in this case, in some embodiments of the present application, the aforementioned diet data collection task can be performed based on the preset second weight data and / or preset second diet data feature when the second weight data and / or second state image corresponding to the target food is not collected. In the embodiment of the present application, the preset second weight data can be set to 0, and the preset second diet data feature can be set to a zero vector.

[0087] It should be understood that in the embodiment of the present application, by collecting the user's dietary data through the first weight data and the first state image before consumption and the second weight data and the second state image after consumption, the accuracy and reliability of dietary data collection can be improved.

[0088] In some embodiments, the nutrients and their amounts ingested by the user can be further determined based on the target diet data determined in the aforementioned process, and the user's diet can be evaluated based on this. For example, if the type of food ingested by the user is rice and the amount is 100g, the amount of nutrients such as carbohydrates (broken down into monosaccharides in the body), protein, and fat ingested by the user can be calculated based on the average nutritional composition table of rice. Furthermore, the intake of these nutrients can be compared with the user's daily dietary recommendations to evaluate whether they meet the healthy diet standards.

[0089] In some embodiments, in order to better help users manage their diet, the diet data recognition model can be combined with a personal health management system. By analyzing the user's diet data, the system can provide personalized diet suggestions, such as adjusting food types, controlling intake, etc., to help users achieve the goal of healthy eating.

[0090] The following is a brief description of the training method and working principle of the feature recognition model and dietary data recognition model involved in the embodiments of the present application.

[0091] Reference Figure 4 In the embodiment of the present application, the aforementioned feature recognition model can be constructed based on the first feature extraction network, the second feature extraction network, the third feature extraction network and the adaptive network. Each network can be regarded as a sub-model of the feature recognition model.

[0092] In an embodiment of the present application, the first feature extraction network can be trained based on a first sample set, where the first sample set includes a plurality of partitioned food images and mixed food images. During the training process using the first sample set, the first feature extraction network can be used to perform surface feature extraction processing on the partitioned food images and / or mixed food images to obtain surface training image features.

[0093] The second feature extraction network can be trained based on the second sample set, and the second sample set can be a part of the aforementioned first sample set, which can include all the partitioned food images in the first sample set. In the process of training the second feature extraction network through the second sample set, the second feature extraction network can be used to perform deep feature extraction processing on the surface training image features output by the first feature extraction network to obtain the first deep training image features. In other words, in an embodiment of the present application, the first feature extraction network can be used to perform surface feature extraction on the partitioned food images in the second sample set to obtain surface training image features, and the surface training image features are used as the input of the second feature extraction network, thereby training the second feature extraction network. It should be noted that when training the second feature extraction network, it is necessary to fix the parameters of the first feature extraction network unchanged, and only adjust the parameters of the second feature extraction network.

[0094] Similarly, the third feature extraction network can be trained based on the third sample set. In an embodiment of the present application, the third sample set includes at least mixed food images. Specifically, in some embodiments, the first feature extraction network can be used to extract surface features of the mixed food images in the third sample set to obtain surface training image features, and the surface training image features are used as inputs of the third feature extraction network, thereby training the third feature extraction network.

[0095] In some embodiments, the third sample set may include partitioned food images and mixed food images. The third feature extraction network is trained together with the surface training image features corresponding to the partitioned food images and the mixed food images. This allows the third feature extraction network to learn the deep image features of the mixed food images while enriching the third feature extraction capability for the mixed food images by learning the deep features of the partitioned food images, thereby improving the deep feature extraction effect of the third feature extraction network for the mixed food images.

[0096] In the process of training the third feature extraction network through the aforementioned third sample set, the third feature extraction network can be used to perform deep feature extraction processing on the surface training image features obtained by the first feature extraction network for image processing in the third sample set, to obtain second deep training image features. It should be noted that, similar to the training process of the second feature extraction network, when training the third feature extraction network, it is also necessary to fix the parameters of the first feature extraction network and only adjust the parameters of the third feature extraction network.

[0097] The adaptive network can be used to determine the feature weights corresponding to the first deep training image features output by the second feature extraction network and the second deep training image features output by the third feature extraction network, respectively, based on the surface training image features output by the first feature extraction network. In an embodiment of the present application, the adaptive network can be trained based on a fourth sample set, and the fourth sample set can include partitioned food images and mixed food images. Similarly, when training an adaptive network, it is also necessary to fix the parameters of other networks unchanged and only adjust the parameters of the adaptive network.

[0098] In the training process of the adaptive network, it is first necessary to extract surface features of the images in the fourth sample set through the first feature extraction network to obtain corresponding surface training image features. Then, the surface training image features are annotated according to the type of sample image (partitioned food image or mixed food image) to obtain annotation information corresponding to each surface training image feature, and the surface training image features are used as input of the adaptive network, and the annotation information is used as output to train the adaptive network until the preset loss function (for example, the cross entropy function) reaches a preset threshold value, and the training is terminated to obtain a trained adaptive network.

[0099] After completing the training, the adaptive network can determine the prediction probability that the corresponding food image is a partitioned food image and a mixed food image based on the surface training image features output by the first feature extraction network, and determine the feature weights corresponding to the first deep training image features output by the second feature extraction network and the second deep training image features output by the third feature extraction network according to the prediction probability.

[0100] In some embodiments of the present application, the aforementioned feature recognition model may include only one third feature extraction network, and the third feature extraction network corresponds to the deep feature extraction of mixed food images of any mixing mode (e.g., different stacking heights, different coverage areas, different occlusion angles, different mixing ratios, etc.). In this case, when the adaptive network determines the feature weights corresponding to the first depth image feature and the second depth image feature, respectively, the corresponding two-dimensional feature weights may be determined according to the surface image features corresponding to the food image to be identified, in which one dimension of the two-dimensional feature weights is the feature weight corresponding to the first depth image feature, and the other dimension is the feature weight corresponding to the second depth image feature.

[0101] In some other embodiments of the present application, the feature recognition model may include m third feature extraction networks, where m is an integer greater than 1. Each different third feature extraction network may correspond to the feature extraction of mixed food images of different mixing modes, and each third feature extraction network may be trained based on the sample image of the corresponding mixing mode. In this case, when the adaptive network determines the feature weights corresponding to the first depth image feature and the second depth image feature, the corresponding multidimensional feature weights may be determined according to the surface image features corresponding to the food image to be identified; in the multidimensional feature weights, one dimension is the weight corresponding to the first depth image feature, and the other m dimensions are the feature weights corresponding to the m third feature extraction networks. In some embodiments, the feature weights corresponding to the m third feature extraction networks may be the mean of the predicted probability of a mixed food image determined by the aforementioned adaptive network based on the surface training image features output by the first feature extraction network, that is, Ph / m, where Ph is the predicted probability of a mixed food image determined by the adaptive network based on the surface training image features output by the first feature extraction network, and m is the number of third feature extraction networks.

[0102] It can be understood that by setting multiple third feature extraction networks in the feature recognition model and using food images with different mixing methods to train the corresponding third feature extraction networks respectively, the adaptability of the feature recognition model to mixed food images with different mixing methods can be improved, and the feature extraction recognition accuracy and feature extraction capability of mixed food images can be improved.

[0103] In some embodiments of the present application, the aforementioned feature recognition model and the aforementioned diet data recognition model can be obtained through collaborative training, and the collaborative training process may include: according to the error signal between the prediction result output by the diet data recognition model and the corresponding label information, adjusting the parameters of the feature recognition model and the initial diet data recognition model at the same time, thereby reducing the error signal.

[0104] During the collaborative training process, different optimization algorithms, such as Adam (Adaptive Moment Estimation) and SGD (Stochastic Gradient Descent), can be used to accelerate convergence and improve model performance.

[0105] Specifically, in some embodiments of the present application, the error signal can be calculated based on the mean square error (MSE) formula. After obtaining the error signal, the parameters of the feature recognition model and the diet data recognition model can be updated by the back propagation algorithm. In this process, the parameter adjustment of the feature recognition model is mainly concentrated on the weights and biases corresponding to each neuron in the first feature extraction network, the second feature extraction network, the third feature extraction network and the adaptive network, while the parameter adjustment of the diet data recognition model is concentrated on its internal neural network structure. For example, in the diet data recognition model, multiple hidden layers (such as convolutional layers, activation layers, pooling layers, fully connected layers, etc.) and output layers may be included, each hidden layer containing a number of neurons, which process the input data through an activation function to extract features that are helpful in predicting dietary behavior (i.e., the type of food intake and its corresponding quantity). During the collaborative training process, the number of hidden layers in the diet data recognition model and the number of neurons in each hidden layer can be adjusted, and the parameters of these hidden layers will also be adjusted according to the error signal to optimize the prediction ability of the model.

[0106] The specific implementation method of using the back propagation algorithm to update the parameters of the feature recognition model and the dietary data recognition model can be regarded as the prior art and will not be discussed in detail in this specification.

[0107] In some embodiments, the collaborative training of the feature recognition model and the diet data recognition model can adopt a multi-task learning framework. In this framework, the two models can share part of the network structure while learning task-related features separately. For example, the feature recognition model can focus on extracting features from food images, while the diet data recognition model focuses on predicting eating behavior based on these features. In this way, the two models can promote each other, thereby improving the overall performance.

[0108] In some embodiments, in order to improve the user experience, an interactive application can be developed that allows users to upload food images and weight data, and receive the analysis results of the model, thereby realizing dietary data collection and recording. For example, in some embodiments, the user can use a mobile terminal such as a mobile phone to take a photo of the target food, and then upload the picture and the weight data corresponding to the target food to the server for processing, and then obtain the corresponding target dietary data and record it. The user can understand his or her diet by viewing the target dietary data. In some embodiments, the application can also provide dietary advice, nutritional information, and health tips based on the user's dietary data to help the user better manage his or her eating habits.

[0109] In summary, the beneficial effects that may be brought about by the embodiments of the present application include but are not limited to: (1) In the method and system for collecting dietary data of diabetic patients based on the Internet of Things provided in some embodiments of the present specification, the dietary data collection task is performed by combining the weight data corresponding to the target food and the dietary data features determined based on the food image to be identified corresponding to the target food. The weight data of the target food can be integrated on the basis of image recognition and used as a constraint condition in the model recognition process, thereby improving the accuracy and reliability of the recognition result of the dietary data recognition model; (2) In the method and system for collecting dietary data of diabetic patients based on the Internet of Things provided in some embodiments of the present specification, the user's dietary data is collected by first weight data and first state image before consumption and second weight data and second state image after consumption, which can further improve the accuracy and reliability of dietary data collection; (3) In the method and system for collecting dietary data of diabetic patients based on the Internet of Things provided in some embodiments of the present specification, the dietary data of diabetic patients based on the Internet of Things In the method and system for collecting dietary data of patients with diabetes, the surface image features of the partitioned food images are subjected to a first depth feature extraction process by using a second feature extraction network, and the surface image features of the mixed food images are subjected to a second depth feature extraction process by using a third feature extraction network, so that the feature recognition model can accurately extract both the depth features of the partitioned food images and the depth features of the mixed food images; (4) In the method and system for collecting dietary data of patients with diabetes based on the Internet of Things provided in some embodiments of the present specification, the third feature extraction network is trained together with the surface training image features corresponding to the partitioned food images and the mixed food images, so that the third feature extraction network can enrich the depth feature extraction capability of the third feature extraction network for the mixed food images by learning the depth features of the partitioned food images, thereby improving the depth feature extraction effect of the third feature extraction network for the mixed food images.

[0110] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or may be any other beneficial effects that may be obtained.

[0111] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

Claims

1. A method for collecting dietary data of diabetic patients based on the Internet of Things, characterized in that: include: Obtaining weight data of a target food and a food image to be identified corresponding to the target food; Using the first feature extraction network in the feature recognition model, performing surface feature extraction processing on the to-be-recognized food image corresponding to the target food to obtain surface image features; Performing a first deep feature extraction process on the surface image features through the second feature extraction network in the feature recognition model to obtain a first deep image feature, and performing a second deep feature extraction process on the surface image features through the third feature extraction network in the feature recognition model to obtain a second deep image feature; wherein the first deep feature extraction process focuses on extracting features from partitioned food images, and the second deep feature extraction process focuses on extracting features from mixed food images; Determine, by using an adaptive network in the feature recognition model, feature weights corresponding to the first depth image feature and the second depth image feature according to the surface image feature; Determining a dietary data feature reflected by the to-be-recognized food image according to the first depth image feature, the second depth image feature, and feature weights corresponding to the first depth image feature and the second depth image feature respectively; A diet data collection task is performed based on the weight data and the diet data characteristics.

2. The method according to claim 1, characterized in that The weight data of the target food includes first weight data before consumption and second weight data after consumption, and the to-be-recognized food image corresponding to the target food includes at least one first state image collected before consumption and at least one second state image collected after consumption; The dietary data features reflected by the food image to be identified include the first dietary data features reflected by the at least one first state image and the second dietary data features reflected by the at least one second state image; The performing of the diet data collection task based on the weight data and the diet data characteristics includes: performing the diet data collection task based on the difference between the first weight data and the second weight data, and the difference between the first diet data characteristics and the second diet data characteristics.

3. The method according to claim 2, characterized in that When the second weight data and / or the second state image are not collected, the diet data collection task is performed based on the preset second weight data and / or the preset second diet data feature.

4. The method according to claim 2, characterized in that The performing the diet data collection task based on the difference between the first weight data and the second weight data, and the difference between the first diet data feature and the second diet data feature, comprises: A first feature vector is obtained based on a difference between the first weight data and the second weight data, and a second feature vector is obtained based on a difference between a feature of the first diet data and a feature of the second diet data; fusing the first feature vector and the second feature vector to obtain a fused feature vector; The fused feature vector is input into a trained diet data recognition model for processing to obtain target diet data, wherein the target diet data includes at least one type of ingested food and its corresponding quantity.

5. The method according to claim 4, characterized in that The diet data recognition model is trained based on the following method: Acquire a sample fusion feature vector and label information corresponding to the sample fusion feature vector, wherein the label information includes the food type and food quantity corresponding to the sample fusion feature vector; Inputting the sample fusion feature vector into an initial diet data recognition model to obtain an initial prediction result; Calculating the difference between the initial prediction result and the label information to obtain an error signal, and adjusting the parameters of the initial diet data recognition model according to the error signal to reduce the error signal; When the error signal is less than a preset threshold, a trained diet data recognition model is obtained.

6. The method according to claim 5, characterized in that The first feature extraction network is trained based on a first sample set, the first sample set includes partitioned food images and mixed food images, and the first feature extraction network is used to perform surface feature extraction processing on the partitioned food images and / or mixed food images to obtain surface training image features; The second feature extraction network is trained based on a second sample set, the second sample set includes partitioned food images, and the second feature extraction network is used to perform deep feature extraction processing on the surface training image features output by the first feature extraction network to obtain first deep training image features; The third feature extraction network is trained based on a third sample set, the third sample set at least includes mixed food images, and the third feature extraction network is used to perform deep feature extraction processing on the surface training image features output by the first feature extraction network to obtain second deep training image features; The adaptive network is used to determine the feature weights corresponding to the first deep training image features output by the second feature extraction network and the second deep training image features output by the third feature extraction network, respectively, according to the surface training image features output by the first feature extraction network; The feature recognition model is constructed based on the first feature extraction network, the second feature extraction network, the third feature extraction network and the adaptive network.

7. The method according to claim 6, characterized in that The feature recognition model and the diet data recognition model are obtained through collaborative training, and the collaborative training process includes: adjusting the parameters of the feature recognition model and the initial diet data recognition model according to the error signal to reduce the error signal.

8. The method according to claim 7, characterized in that The feature recognition model includes m third feature extraction networks, where m is an integer greater than or equal to 1; when m is greater than 1, the m third feature extraction networks respectively focus on feature extraction of mixed food images with different mixing methods, and the m third feature extraction networks are trained based on a large number of mixed food images with different mixing methods; The using the adaptive network in the feature recognition model to determine the feature weights corresponding to the first depth image feature and the second depth image feature respectively according to the surface image feature includes: Through the adaptive network, multidimensional feature weights are determined according to the surface image features, wherein the multidimensional feature weights include feature weights corresponding to the first depth image features and feature weights corresponding to m second depth image features, and the m second depth image features are obtained by feature extraction through m third feature extraction networks.

9. A diabetes patient diet data collection system based on the Internet of Things, characterized in that: include: An acquisition module, used to acquire weight data of a target food and a food image to be identified corresponding to the target food; A surface feature extraction module, used to perform surface feature extraction processing on the to-be-recognized food image corresponding to the target food using the first feature extraction network in the feature recognition model, to obtain surface image features; a deep feature extraction module, configured to perform a first deep feature extraction process on the surface image features through a second feature extraction network in the feature recognition model to obtain a first deep image feature, and perform a second deep feature extraction process on the surface image features through a third feature extraction network in the feature recognition model to obtain a second deep image feature; wherein the first deep feature extraction process focuses on feature extraction of partitioned food images, and the second deep feature extraction process focuses on feature extraction of mixed food images; a feature weight determination module, configured to determine, by using an adaptive network in the feature recognition model, feature weights corresponding to the first depth image feature and the second depth image feature according to the surface image feature; a dietary data feature determination module, configured to determine the dietary data feature reflected by the to-be-recognized food image according to the first depth image feature, the second depth image feature, and feature weights corresponding to the first depth image feature and the second depth image feature respectively; A diet data collection module is used to perform a diet data collection task based on the weight data and the diet data characteristics.

10. The system according to claim 9, characterized in that The weight data of the target food includes first weight data before consumption and second weight data after consumption, and the to-be-recognized food image corresponding to the target food includes at least one first state image collected before consumption and at least one second state image collected after consumption; The dietary data features reflected by the food image to be identified include the first dietary data features reflected by the at least one first state image and the second dietary data features reflected by the at least one second state image; The dietary data collection module is specifically used for: A first feature vector is obtained based on a difference between the first weight data and the second weight data, and a second feature vector is obtained based on a difference between a feature of the first diet data and a feature of the second diet data; fusing the first feature vector and the second feature vector to obtain a fused feature vector; The fused feature vector is input into a trained diet data recognition model for processing to obtain target diet data, wherein the target diet data includes at least one type of ingested food and its corresponding quantity.

Citation Information

Patent Citations

  • Nutrition management method and system based on deep learning food image recognition model

    CN114550018A

  • Sugar-taking monitoring system for diabetics

    CN115274052A

  • Health management method and device for prediabetic patients

    CN116682532A

  • Food intake prediction method and device based on artificial intelligence, equipment and medium

    CN117456520A

  • Image recognition method, model training method and related device

    CN117612224A