A human food intake evaluation system and method based on regression analysis
Through a human food intake evaluation system based on regression analysis, using diet data acquisition sensors and FPGA sensing data processing platform, combined with YOLOv2 image recognition algorithm and nutrient database, the inefficiency and inaccurate diet data recording and evaluation in the prior art are solved. The problem has achieved efficient and safe dietary data collection and scoring, helping users improve their eating habits.
Patent Information
- Application Number
- CN202210414803.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-20
AI Technical Summary
The prior art is inefficient and inaccurate when recording and evaluating human diet data, difficult to persist for a long time, and has data security problems.
The human food intake evaluation system based on regression analysis is adopted, and real-time and accurate diet data collection, processing and scoring are achieved through diet data acquisition sensors, FPGA sensing data processing platform and backend server, combined with the YOLOv2 image recognition algorithm and nutrient database.
It improves the efficiency and accuracy of diet records, reduces the time cost of users, enhances the real-time and safety of the system, and can effectively guide users to improve their eating habits and improve their diet quality.
Smart Images

Figure CN114842949B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer processing technology and medical diet evaluation technology, and in particular to a system and method for measuring, analyzing and evaluating human food intake. Background Art
[0002] With the development of social economy and the rapid change of residents' lifestyle, the incidence of chronic diseases such as overweight and obesity, hypertension, hyperlipidemia, coronary heart disease, stroke, diabetes, and malignant tumors has shown a significant upward trend. The main factor causing these diseases is unreasonable eating patterns. Therefore, in order to help people avoid unreasonable eating patterns, it is particularly important to collect residents' dietary data and give some targeted guidance and suggestions.
[0003] Usually, if people want to understand their dietary health, they need to record their daily diet first, and then consult a nutritionist for dietary advice. However, the dietary questionnaire needs to record all the food and drinks that the user eats over a period of time, including snacks. Relying on traditional questionnaire methods to record diet is inefficient and inaccurate.
[0004] Thanks to the development and application of smartphone technology, many apps have emerged to record user diets and dietary recommendations. These apps generally use methods such as custom manual entry, voice entry, code scanning, and recipe search to record diets. However, it is difficult for users to keep recording their diets for a long time. Due to human forgetfulness and misremembering, these records may not be accurate, and they cannot achieve sufficient results for dietary evaluation. Summary of the invention
[0005] The present invention aims to propose a human food intake evaluation system and method based on regression analysis, which obtains human food intake data by collecting dietary data, and then measures and analyzes them until the evaluation of human food intake is completed.
[0006] The present invention is achieved by adopting the following technical solutions:
[0007] A human food intake evaluation system based on regression analysis, the system includes a dietary data acquisition sensor, an FPGA sensor data processing platform and a background server; wherein:
[0008] The dietary data collection sensor is used to collect the user's daily dietary data information; the dietary data collection sensor is connected to the FPGA sensor data processing platform through various interface modules, and the data is sent to the FPGA sensor data processing platform for real-time processing;
[0009] The FPGA sensor data processing platform establishes a communication connection with the backend server via a network cable, and sends the processed information to the backend server;
[0010] The backend server is used to compare the diet data with the nutrient database, so as to obtain the detailed nutrient intake data of the user and score the user's diet;
[0011] Furthermore, the dietary data collection sensor includes a pressure sensor for collecting food weight and an image sensor for collecting food pictures;
[0012] Further, the FPGA sensor data processing platform includes an HX711 data acquisition module for collecting weight data, an image acquisition module for collecting RGB565 image data of the DVP interface, a VDMA module for frame buffer control, a YOLOv2 accelerator for accelerating the YOLOv2 image recognition algorithm, an SD card for storing food pictures, and an ARM processor module for controlling the operation of the entire system;
[0013] Furthermore, the backend server includes a food nutrient database for obtaining the nutrient content of each food and a diet scoring module for evaluating the user's diet.
[0014] A method for evaluating human food intake based on regression analysis, the method specifically includes the following execution process:
[0015] The dietary data collection sensor collects the user's daily dietary data information and sends it to the FPGA sensor data processing platform for processing;
[0016] The YOLOv2 food image recognition model is trained based on the VireoFood-172 dataset; the hardware implementation of the food image recognition method implements the YOLOv2 image recognition algorithm based on the CPU+FPGA architecture, and the YOLOv2 convolutional neural network accelerator is designed using the convolution loop unrolling SIMD architecture to perform food image recognition;
[0017] The convolution operation process of the convolution loop expansion SIMD architecture is:
[0018] There are 23 convolutional layers, 5 maximum pooling layers, 2 routing layers and 1 reordering layer in the YOLOv2 network structure. Except for the routing layer, the other layers use the feature map output of the previous layer as the feature map input of the next layer, and the entire process is executed in serial order. Therefore, after the function of the routing layer is implemented through the pre-set access address offset, the YOLOv2 accelerator only needs to read the memory data, process the data and write the data back to the memory through the AXI bus under the control of the processor. When processing each convolutional layer, pooling layer and reordering layer, due to limited FPGA resources, it is necessary to divide the input feature map into several blocks and read a block of data from the memory each time;
[0019] When a block of data is processed, it must first read the data from the off-chip memory and then send the data to the on-chip cache. Then the layer category is determined and the block of data is sent to the accelerator's CONV, POOL, or REORG for processing based on the determination result. After the processing is completed, the data of the block that has completed the operation is sent to the output buffer, and finally the data is stored in the off-chip memory through the bus.
[0020] The back-end server is used to compare the dietary data with the nutrient database, thereby obtaining the user's detailed nutrient intake data and scoring the user's diet.
[0021] Compared with the prior art, the present invention has the following beneficial effects:
[0022] 1) It can directly record the user's daily diet efficiently and accurately, which is convenient and fast, and the user has almost no time cost, greatly improving the user experience;
[0023] 2) Strong real-time and security: Since the FPGA sensor data processing platform is deployed at the edge, real-time food image recognition can be achieved. However, existing technologies generally require users to upload photos to the cloud for food image recognition, which slows down data processing and poses security issues during data transmission.
[0024] 3) It can score the user's eating pattern based on the user's dietary data, help the user understand the nutrient intake, distinguish between healthy eating patterns and unreasonable eating patterns, effectively help the user change bad eating habits and dietary preferences, and gradually guide the user to change bad eating habits, thereby improving the quality of the user's diet. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the structure of a human food intake evaluation system based on regression analysis of the present invention;
[0026] Figure 2 It is a schematic diagram of the framework structure of the backend server 300;
[0027] Figure 3 YOLOv2 convolutional neural network accelerator architecture diagram for convolution loop unrolling SIMD architecture;
[0028] Figure 4 Flowchart of the YOLOv2 convolutional neural network accelerator execution schedule for SIMD architecture unrolling convolution loops.
[0029] Specific implementation method
[0030] The principles and features of the present invention will be described below with reference to the accompanying drawings.
[0031] like Figure 1 As shown, it is a schematic diagram of the structure of a human food intake evaluation system based on regression analysis of the present invention, which includes a diet data acquisition sensor 100, an FPGA sensor data processing platform 200 and a backend server 300. The diet data acquisition sensor 100 is used to collect the daily diet data information of the user; the diet data acquisition sensor and the FPGA sensor data processing platform are connected through various interface modules, and the data is sent to the FPGA sensor data processing platform 200 for processing; the FPGA sensor data processing platform 200 set at the edge end establishes a communication connection with the backend server 300 in the cloud through a network cable, and the processed information is sent to the backend server 300; the backend server 300 compares the diet data of these users with the nutrient database, obtains the detailed nutrient intake data of the user and scores the diet of the user.
[0032] Furthermore, the dietary data collection sensor 100 includes a pressure sensor 101 for collecting food weight and an image sensor (eg, a camera) 102 for collecting food pictures.
[0033] Specifically, the pressure sensor 101 is a resistance strain gauge pressure sensor with a measuring range of 5 kg, an analog power input of 2.6 V to 5.5 V, and an accuracy of 0.01% to meet the accuracy requirements.
[0034] The pressure sensor 101 also requires the HX711 module 103 for AD conversion. The analog input of the module is the analog differential input of the resistance strain gauge pressure sensor. It can be powered by a digital power supply of 2.6 V to 5.5 V. It uses serial communication to transmit data with the GPIO interface of the FPGA platform. The FPGA platform needs to provide the module with a clock, and data transmission is completed within 25 to 27 clock cycles.
[0035] The camera 102 has the following specific parameters: using the OV5640 photosensitive array to achieve 2592*1944 (i.e., 500W pixels), the fastest 15fps QSXVGA (2592*1944) or 90fps VGA (640*480) resolution image acquisition, including automatic exposure control (AEC), automatic white balance (AWB), etc. At the same time, the sensor supports LED fill light, MIPI (mobile industry processor interface) output interface and DVP (digital video parallel) output interface selection, ISP (image signal processing) and AFC (auto focus control) and other functions. Preferably, the device uses a DVP interface to output an 8-bit RGB565 format image with line and field synchronization signal control.
[0036] Furthermore, the FPGA sensor data processing platform 200 includes an HX711 data acquisition module 201 for collecting weight data, an image acquisition module 202 for collecting RGB565 image data of a DVP interface, a VDMA module 203 for frame buffer control, a YOLOv2 accelerator 204 for accelerating a YOLOv2 image recognition algorithm, an SD card 205 for storing food images, and an ARM processor module 206 for controlling the operation of the entire system.
[0037] Specifically, the HX711 data acquisition module is connected to the HX711AD conversion module through the GPIO port of the FPGA, provides a clock to the HX711AD conversion module, receives data from the module, and sends it to the ARM processor for processing.
[0038] The image acquisition module 202 receives image data in RGB565 format with line-field synchronization control from the DVP interface of the image sensor, and then converts the received image data into data in video stream format and sends it to the VDMA module 203 for frame buffering. The VDMA module 203 establishes a read-write channel in the memory to buffer each frame of the video stream, and the VDMA module needs to be configured in dynamic synchronous phase-locking mode.
[0039] The FPGA sensor data processing platform 200 implements food image recognition, that is, the YOLOv2 food image recognition model is trained based on the VireoFood-172 dataset; the food image recognition hardware implementation part implements the YOLOv2 image recognition algorithm based on the CPU+FPGA architecture, and adopts the convolution loop expansion SIMD architecture to design the YOLOv2 convolutional neural network accelerator. According to the needs, it is optimized by combining parameter reordering, ping-pong buffer and other optimization strategies.
[0040] The parameter reordering optimization strategy pre-arranges the parameters required for the convolutional layer so that the parameters transmitted each time are stored continuously in DRAM and can be transmitted with a larger burst length, effectively improving the bandwidth utilization efficiency. For weight parameters, since they are only related to the current layer, the weights of each layer are pre-arranged according to the parallelism of the input and output feature map dimensions of each layer to reduce the number of memory accesses and increase the burst length.
[0041] The ping-pong buffer optimization strategy can overlap the delay of reading data from the off-chip DRAM, the delay of on-chip data processing, and the delay of writing the processed data back to the off-chip DRAM, thereby reducing the total delay.
[0042] The YOLOv2 accelerator module 205 is responsible for the convolutional neural network acceleration tasks such as the convolution layer, pooling layer, and reordering layer assigned by the processor module. The accelerator IP core is designed using Vivado HLS, and the C language algorithm is translated into Verilog hardware description language, encapsulated as a hardware IP core and deployed on the FPGA. In this process, program optimization instructions provided by HLS are added to the algorithm, such as: what interface is used for data input / output, how much is the calculation parallelism within the program, whether pipeline design is required, etc. The YOLOv2 accelerator operates at 150Mhz, with a maximum effective bandwidth of 489MB / s.
[0043] The ARM processor 206 is responsible for a series of control tasks, including receiving data from the HX711 data acquisition module and storing it in the SD card; locking the frame buffer of the read channel stored in the VDMA module from the memory frame buffer, and grabbing a frame of image data in the channel from the memory and storing it in the SD card to complete the photo recording of the user's daily diet pictures; completing tasks such as preprocessing, post-processing, data flow control scheduling, routing layer address offset, etc. of the image recognition algorithm, and controlling the process of forward reasoning of the convolutional neural network.
[0044] Preferably, the ARM processor 206 uses an ARM main controller, adopts a dual-core Cortex-A9 32-bit MPE+FPU up to 1GB / 512MB DDR3, 32MB QSPI Flash, and a main frequency of 1Ghz. The rich resources can support the high-speed and low-power operation of the YOLOv2 image recognition algorithm and achieve accurate and reliable image recognition.
[0045] like Figure 2 , which is a schematic diagram of the framework structure of the backend server, the backend server 300 includes a food nutrient database 301 for obtaining the nutrient content of each food, and a diet scoring module 302 for evaluating the user's diet.
[0046] like Figure 3 As shown in the figure, it is a YOLOv2 convolutional neural network accelerator architecture diagram with convolution loop expansion SIMD architecture.
[0047] Among them, the YOLOv2 convolutional neural network accelerator includes an input and output buffer module, a weight buffer module, a convolution module, a pooling module, and a reordering module. The accelerator reads the input feature map and the weight parameters of each layer through the AXI bus, and then writes them into the input buffer module and the weight buffer module respectively. The convolution module, the pooling module, and the reordering module are responsible for the processing of the convolution layer (CONV), the maximum pooling layer (POOL), and the reordering layer (REORG) respectively. After processing a block of data, the output feature map pixel block in the output buffer is written back to the off-chip memory through the AXI bus. The control logic is generated by the ARM processor, which is responsible for controlling each module in the accelerator.
[0048] like Figure 4 As shown in FIG. 1 , it is a flowchart of the execution scheduling of the YOLOv2 convolutional neural network accelerator with a convolution loop unrolling SIMD architecture. The process specifically includes the following processing:
[0049] Construct a YOLOv2 convolutional neural network accelerator, including 23 convolutional layers, 5 maximum pooling layers, 2 routing layers and 1 reordering layer; except for the routing layer, the convolutional layer, the maximum pooling layer and the reordering layer use the feature map output of the previous layer as the feature map input of the next layer. The entire convolution operation process is performed in serial order. After the function of the routing layer is realized through the preset access address offset, the memory data is read, processed and written back to the memory through the AXI bus under the control of the processor; when processing each convolutional layer, the maximum pooling layer and the reordering layer, the input feature map is divided into several blocks, and a block of data is read from the memory each time;
[0050] When a block of data is processed, it first reads the data from the off-chip memory and then sends the data to the on-chip cache. Then, the layer type is determined and the block of data is sent to the accelerator for processing based on the determination result.
[0051] After the processing is completed, the data of the block that has completed the operation is sent to the output buffer, and finally the data is stored in the off-chip memory through the bus;
[0052] Furthermore, the food nutrient database includes the content of 100 types of nutrients in each of the more than 20,000 Chinese dishes. Among them, the nutrient content of each dish includes the content of 43 basic nutrients, 37 fatty acids and 20 amino acids. The basic nutrients include: cholesterol, purine, energy, protein, fat, carbohydrates, water, dietary fiber, total vitamin A, vitamin D, vitamin E, vitamin K, vitamin B1, vitamin B2, vitamin B6, vitamin B12, vitamin B5, vitamin B9, vitamin B3, choline, vitamin H, vitamin C, calcium, phosphorus, potassium, sodium, magnesium, chlorine, iron, zinc, iodine, selenium, copper, molybdenum, fluorine, manganese, chromium, energy, vitamin A1, carotene, vitamin Eα- E, vitamin Eβ+γ-E, vitamin δ-E, the number is 43; the fatty acids include: total fatty acids, saturated fatty acids, monounsaturated fatty acids, polyunsaturated fatty acids, unknown fatty acids, caproic acid, caprylic acid, caprylic acid, undecanoic acid, dodecanoic acid, tridecanoic acid, tetradecanoic acid, pentadecanoic acid, hexadecanoic acid, heptadecanoic acid, nonadecanoic acid, eicosanoic acid, docosanoic acid, tetradecanoic acid, pentadecanoic acid, hexadecanoic acid, heptadecanoic acid, octadecanoic acid, octadecanoic acid, eicosanoic acid, docosanoic acid, tetradecanoic acid, pentadecanoic acid, hexadecanoic acid, heptadecanoic acid, octadecanoic acid, eicosanoic acid, -9-enoic acid, docos-9-enoic acid, 16:2 acid, octadec-9,12-dienoic acid, octadec-9,12,15-trienoic acid, eicosa-11,14-dienoic acid, eicosa-5,8,11-trienoic acid, eicosa-5,8,11,14-tetraenoic acid, eicosa-5,8,11,14,17-pentaenoic acid, docos-13,16,19-trienoic acid, docos-7,10,13,16-tetraenoic acid, docos-7,10,1 3,16,19-pentaenoic acid, docosyl-4,7,10,13,16,19-hexaenoic acid, and polyunsaturated acids are unknown, with a total of 37; the amino acids include: isoleucine, leucine, lysine, total sulfur-containing amino acids (SAA), methionine, cysteine, total aromatic amino acids (AAA), phenylalanine, tyrosine, threonine, tryptophan, valine, arginine, histidine, alanine, aspartic acid, glutamic acid, glycine, proline, and serine, with a total of 20.
[0053] The diet scoring module 301 is based on the CHNS database, and performs big data analysis through a machine learning algorithm to extract a regression model, and analyzes and mines the user's nutrient intake data to accurately implement the user's diet evaluation. Further, the regression model is based on the CHNS database, and performs big data analysis through a machine learning algorithm to extract a regression model, and analyzes and mines the user's nutrient intake data to accurately implement the user's diet evaluation. Further, the regression model is based on 280,000 dietary records of Chinese residents in the CHNS database, and takes the 24 common nutrient intakes of Chinese residents for 3 days as feature inputs, and the diet score CHEI is used as a label. Based on the mutual pheromone method, 21 essential nutrients for the human body are feature extracted, and the 10-fold cross-validation method is used for training. After obtaining the user's actual nutrient intake, this diet scoring module can generate the intake score of each nutrient and the overall diet score, and intuitively display the user's diet evaluation in the form of a chart.
[0054] The present invention is further described above with the aid of specific embodiments. However, it should be understood that the specific description here should not be construed as limiting the essence and scope of the present invention. Various modifications made to the above embodiments by ordinary technicians in the field after reading this specification are all within the scope of protection of the present invention.
Claims
1. A human food intake evaluation system based on regression analysis, characterized in that: The system includes a dietary data collection sensor, an FPGA sensor data processing platform, and a background server; wherein: The dietary data collection sensor is used to collect the user's daily dietary data information; the dietary data collection sensor is connected to the FPGA sensor data processing platform through various interface modules, and the data is sent to the FPGA sensor data processing platform for processing; The FPGA sensor data processing platform establishes a communication connection with the backend server via a network cable, and sends the processed information to the backend server; The backend server is used to compare the diet data with the nutrient database, thereby obtaining detailed nutrient intake data of the user and scoring the user's diet, including: The dietary data collection sensor collects the user's daily dietary data information and sends it to the FPGA sensor data processing platform for processing; The YOLOv2 food image recognition model was trained based on the VireoFood-172 dataset. The hardware implementation of the food image recognition method uses the CPU+FPGA architecture to implement the YOLOv2 image recognition algorithm in hardware, and uses the convolution loop unrolling SIMD architecture to design the YOLOv2 convolutional neural network accelerator to perform food image recognition. The convolution operation process of the convolution loop expansion SIMD architecture is: Construct a YOLOv2 convolutional neural network accelerator, including 23 convolutional layers, 5 maximum pooling layers, 2 routing layers and 1 reordering layer; except for the routing layer, the convolutional layer, the maximum pooling layer and the reordering layer use the feature map output of the previous layer as the feature map input of the next layer. The entire convolution operation process is performed in serial order. After the function of the routing layer is realized through the preset access address offset, the memory data is read, processed and written back to the memory through the AXI bus under the control of the processor; when processing each convolutional layer, the maximum pooling layer and the reordering layer, the input feature map is divided into several blocks, and a block of data is read from the memory each time; When a block of data is processed, it first reads the data from the off-chip memory and then sends the data to the on-chip cache. Then, the layer type is determined and the block of data is sent to the accelerator for processing based on the determination result. After the processing is completed, the data of the block that has completed the operation is sent to the output buffer, and finally the data is stored in the off-chip memory through the bus; The backend server is used to compare the dietary data with the nutrient database, thereby obtaining detailed nutrient intake data of the user and scoring the user's diet; Furthermore, the dietary data collection sensor includes a pressure sensor for collecting food weight and a camera for collecting food pictures; Further, the FPGA sensor data processing platform includes an HX711 data acquisition module for collecting weight data, an image acquisition module for collecting RGB565 image data of the DVP interface, a VDMA module for frame buffer control, a YOLOv2 accelerator for accelerating the YOLOv2 image recognition algorithm, an SD card for storing food pictures, and an ARM processor module for controlling the operation of the entire system; Furthermore, the backend server includes a food nutrient database for obtaining the nutrient content of each food and a diet scoring module for evaluating the user's diet.
2. A human food intake evaluation system based on regression analysis as claimed in claim 1, characterized in that: The image acquisition module receives image data in RGB565 format with line-field synchronization control from the DVP interface of the camera, then converts the received image data into data in video stream format and sends it to the VDMA module for frame buffering.
3. A human food intake evaluation system based on regression analysis as claimed in claim 1, characterized in that: The VDMA module establishes a read and write channel in the memory to cache each frame of the video stream, and the VDMA module is configured in a dynamic synchronous phase-locked mode.
4. A human food intake evaluation system based on regression analysis as claimed in claim 1, characterized in that: Food image recognition is implemented in the FPGA sensor data processing platform, that is, the YOLOv2 food image recognition model is trained based on the VireoFood-172 dataset; the food image recognition hardware implementation part implements the YOLOv2 image recognition algorithm in hardware based on the CPU+FPGA architecture, and designs the YOLOv2 convolutional neural network accelerator using the convolution loop unrolling SIMD architecture.
5. The human food intake evaluation system based on regression analysis as claimed in claim 4 is characterized in that: A YOLOv2 convolutional neural network accelerator using a convolution loop unrolling SIMD architecture, wherein the YOLOv2 convolutional neural network accelerator using a convolution loop unrolling SIMD architecture: Among them, the YOLOv2 convolutional neural network accelerator includes an input and output buffer module, a weight buffer module, a convolution module, a pooling module, and a reordering module. The accelerator reads the input feature map and the weight parameters of each layer through the AXI bus, and then writes them into the input buffer module and the weight buffer module respectively; the convolution module, the pooling module and the reordering module are responsible for the processing of the convolution layer, the maximum pooling layer and the reordering layer respectively. After processing a block of data, the output feature map pixel block in the output buffer is written back to the off-chip memory through the AXI bus. The control logic is generated by the ARM processor, which is responsible for controlling each module in the accelerator.
6. A human food intake evaluation system based on regression analysis as claimed in claim 1, characterized in that: The YOLOv2 accelerator module is responsible for the convolutional layer, pooling layer, and reordering layer convolutional neural network acceleration tasks assigned by the processor module. It operates at 150Mhz with a maximum effective bandwidth of 489MB / s.
7. A human food intake evaluation system based on regression analysis as claimed in claim 1, characterized in that: The ARM processor is responsible for control, including receiving data from the HX711 data acquisition module and storing it in the SD card; locking the frame buffer of the read channel stored in the VDMA module from the memory frame buffer, and grabbing a frame of image data in the channel from the memory and storing it in the SD card to complete the photo recording of the user's daily diet pictures; completing the preprocessing, post-processing, data flow control scheduling, and routing layer address offset tasks of the image recognition algorithm, and controlling the forward reasoning process of the convolutional neural network.
