Method, device and equipment for intelligent monitoring and reminder of patient eating behavior

Abstract

The present disclosure relates to the field of image recognition and artificial intelligence technology, and provides a method, device and equipment for intelligent monitoring and reminder of patient dietary behavior, the method comprising: collecting patient dietary image data containing calibrated tableware, and preprocessing the dietary image data; extracting at least three types of local invariant feature descriptors from the preprocessed dietary image data; normalizing the extracted various feature descriptors and performing feature fusion to generate a fused feature vector; inputting the fused feature vector into a pre-trained deep neural network model for feature classification, and outputting a food category label; calculating the patient's dietary intake based on the dietary image data, the food category label and the size of the calibrated tableware; and performing intelligent personalized reminders based on the food category label, the patient's case data and the patient's dietary intake, thereby solving the problems of insufficient accuracy, high computational complexity and weak personalized services in the current process of dietary behavior recognition and intervention for cancer patients.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition and artificial intelligence technology, and in particular to a method, device and equipment for intelligently monitoring and reminding patients' dietary behavior. Background Art

[0002] Cancer patients often face challenges during treatment, such as weakened immunity, malnutrition, and difficulty eating. Studies have shown that a healthy and balanced diet significantly improves treatment tolerance, promotes postoperative recovery, and prevents tumor recurrence. However, in real life, patients often face challenges with insufficient dietary awareness, unbalanced dietary patterns, and frequent unhealthy eating habits. This makes nutritional interventions difficult to implement accurately, especially in non-medical settings lacking professional guidance.

[0003] In recent years, the rapid development of image recognition and artificial intelligence technologies has provided new opportunities for achieving "non-invasive dietary behavior recognition." By capturing dietary images through cameras or mobile devices and analyzing the image content with the help of deep learning models, it is possible to initially identify and analyze food types and nutritional structures. However, existing technologies still have the following shortcomings:

[0004] (1) Image convolution efficiency is low, and the computational pressure on edge processing equipment is high;

[0005] (2) The extraction of salient regions is unstable, which makes the recognition results susceptible to background interference;

[0006] (3) The feature expression is single and lacks multi-angle integration of the local structure and texture of food;

[0007] (4) There is a lack of an intake estimation mechanism that is combined with actual consumption and tableware specifications;

[0008] (5) It is difficult to generate personalized intervention recommendations based on the patient’s individual condition (disease type, treatment stage).

[0009] To this end, there is an urgent need for an intelligent system that integrates image denoising, saliency detection, structural feature fusion and individualized intervention recommendation. Summary of the Invention

[0010] The present disclosure provides a method, device, and equipment for intelligently monitoring and reminding patients' dietary behavior, aiming to address the current problems of insufficient accuracy, high computational complexity, and weak personalized services in the process of identifying and intervening in the dietary behavior of cancer patients.

[0011] According to a first aspect of the present disclosure, a method for intelligently monitoring and reminding a patient's dietary behavior is provided, comprising:

[0012] Collecting patient diet image data containing calibrated tableware, and preprocessing the diet image data, wherein the patient diet image data is associated with the patient case data;

[0013] Extracting at least three types of local invariant feature descriptors from the preprocessed diet image data, wherein the local invariant feature descriptors include scale-invariant features, binary high-efficiency features, and improved nonlinear scale-space features;

[0014] After normalizing the extracted feature descriptors, perform feature fusion to generate a high-dimensional fusion feature vector;

[0015] Inputting the high-dimensional fusion feature vector into a pre-trained deep neural network model for feature classification, and outputting a food category label;

[0016] Calculating the patient's intake based on the diet image data, the food category label, and the size of the calibrated tableware;

[0017] Provide intelligent personalized reminders based on the food category labels, patient case data and patient intake.

[0018] As a preferred embodiment, the preprocessing of the diet image data includes:

[0019] performing image denoising on the diet image data based on a logarithmic multiplication algorithm of extended operand decomposition;

[0020] Based on the denoised food image data, a center-periphery difference saliency map and a background contrast saliency map are constructed respectively;

[0021] The center-periphery difference saliency map and the background contrast saliency map are weightedly fused to generate a composite saliency map.

[0022] As a preferred embodiment, the logarithmic multiplication algorithm based on extended operand decomposition performs image denoising on the diet image data, including:

[0023] Decomposing the image pixel X of the diet image data and the Gaussian filter kernel weight Y into three binary operands A, B, and C;

[0024] Extract the most significant bit position of each operand and the last digit , determine the order of magnitude of each operand;

[0025] By pre-training the logarithmic table and the shifter, the operands A, B, and C are converted into logarithmic mode.

[0026] For the operands A, B, C and Perform logarithmic addition and antilogarithmic calculation to obtain the product terms between the operands;

[0027] According to the product between the operands, an approximate value of X×Y is determined to complete the image pixel filtering convolution.

[0028] As a preferred embodiment, the method of constructing a center-periphery difference saliency map and a background contrast saliency map based on the denoised food image data includes:

[0029] Based on the denoised food image data, the absolute value of the difference between the grayscale value or intensity value I(a,b) of the pixel point (a,b) and the average value S(a,b) of the neighborhood pixels around the pixel point (a,b) is calculated to obtain the center-periphery saliency map:

[0030]

[0031] Among them, the neighborhood range of pixel point (a, b) is defined by a sliding window;

[0032] Based on the denoised food image data, the absolute value of the difference between the grayscale value or intensity value I(a,b) of the pixel (a,b) and the average intensity value μ(B) of the image background area is calculated to obtain the background contrast saliency map:

[0033]

[0034] The image background area is the image edge or a preset non-target area.

[0035] As a preferred embodiment, the weighted fusion of the center-periphery difference saliency map and the background contrast saliency map to generate a composite saliency map includes:

[0036] The composite saliency map is obtained by linearly combining D(a,b) and C(a,b) with weight coefficients α and β:

[0037]

[0038] Among them, α+β=1, and the weight coefficient is adaptively adjusted according to the scenario.

[0039] As a preferred embodiment, at least three types of local invariant feature descriptors are extracted from the preprocessed diet image data, wherein the local invariant feature descriptors include scale-invariant features, binary high-efficiency features, and improved nonlinear scale space features, including:

[0040] Scale-invariant features: achieved through Gaussian scale space construction and directional gradient statistics;

[0041] Binary efficient features: Generate binary descriptors based on comparison of sampling point intensities;

[0042] Improved nonlinear scale-space features: Detect Hessian matrix feature points in nonlinear diffusion space.

[0043] As a preferred embodiment, the extracted various feature descriptors are normalized and then feature fused to generate a high-dimensional fused feature vector, including:

[0044] The scale-invariant features, binary efficient features and improved nonlinear scale space features are normalized to zero mean and unit variance to eliminate dimensional differences;

[0045] The normalized features are concatenated through the channel dimension to generate a high-dimensional fused feature vector.

[0046] As a preferred embodiment, the calculating of the patient's intake based on the diet image data, the food category labels and the size of the calibrated tableware includes:

[0047] Based on the pre-processed food image data, the food area is located through the composite saliency map and its two-dimensional projection area is extracted;

[0048] acquiring food image depth information according to the food image data;

[0049] Calculating a scaling factor based on the size of the calibrated tableware, wherein the scaling factor depends on the actual area of ​​the tableware and the pixel area of ​​the tableware image;

[0050] Calculating the volume of ingested food based on the two-dimensional projection area, depth information of the diet image, and a proportional coefficient;

[0051] Calculate the mass of food intake based on the volume of food intake and a pre-built volume and density conversion model;

[0052] Patient intake was calculated based on the mass of food ingested and a pre-constructed mass-to-calorie conversion model.

[0053] According to a second aspect of the present disclosure, there is provided a device for intelligently monitoring and reminding a patient's dietary behavior, comprising:

[0054] An image acquisition and preprocessing module, configured to acquire patient diet image data including calibrated tableware and preprocess the diet image data, wherein the patient diet image data is associated with the patient case data;

[0055] a feature extraction module, configured to extract at least three types of local invariant feature descriptors from the preprocessed diet image data, wherein the local invariant feature descriptors include scale invariance, binary efficient representation, and nonlinear structure enhancement characteristics;

[0056] The feature fusion module is used to normalize the extracted feature descriptors and then fuse them to generate a high-dimensional fused feature vector;

[0057] A food category label recognition module is used to input the high-dimensional fusion feature vector into a pre-trained deep neural network model for feature classification and output a food category label;

[0058] a patient's actual food intake module, configured to calculate the patient's food intake based on the diet image data, food category labels, and the size of the calibrated tableware;

[0059] The personalized reminder module is used to provide intelligent personalized reminders based on the food category labels, patient case data and patient intake.

[0060] According to a third aspect of the present disclosure, an electronic device, at least one processor, and a memory communicatively connected to the at least one processor are provided; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method as described in any one of the above items.

[0061] Compared with the prior art, the present disclosure achieves the following beneficial effects:

[0062] (1) This paper proposes a logarithmic multiplication algorithm based on extended operand decomposition. The algorithm optimizes the multiplication operation in traditional image filtering into addition and table lookup, effectively improving the image denoising efficiency and the computing performance of edge devices during the Gaussian filtering process, and significantly reducing the logic operation complexity and power consumption.

[0063] (2) This paper proposes a method for constructing a composite saliency map by fusing the center-periphery difference saliency map with the background contrast saliency map. This method can effectively extract the saliency information of the target area in the image, improve the positioning accuracy of foreground targets in complex scenes (such as food on a plate), and enhance the robustness of subsequent feature extraction and classification.

[0064] (3) This paper proposes an improved nonlinear scale-space feature extraction method (PKAZE), which can effectively extract high-order structural features such as edges and spots in food images through nonlinear diffusion and Hessian matrix structure enhancement operations, thereby improving the stable expression capability of highly complex food images.

[0065] (4) This paper proposes a fusion strategy of three feature descriptors: SIFT, BRISK and PKAZE. By uniformly normalizing and channel-level splicing, a high-dimensional feature vector is generated, which takes into account texture details, scale changes and rotation invariance, and achieves a more comprehensive and robust image feature representation.

[0066] (5) This disclosure proposes a method for estimating food intake, mass, and calories by combining image depth information with tableware size, introduces a volume calculation and density conversion model, and supports the inference of actual intake mass from two-dimensional images, thereby providing patients with quantitative and personalized dietary assessments and intelligent reminders to meet the nutritional intervention needs of different diseases and treatment stages.

[0067] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0069] Figure 1 A flow chart of a method for intelligently monitoring and reminding a patient's dietary behavior according to an embodiment of the present disclosure is shown;

[0070] Figure 2 A block diagram of an intelligent monitoring and reminder device for patient eating behavior according to an embodiment of the present disclosure is shown;

[0071] Figure 3 A schematic diagram of an exemplary electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0072] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0073] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0074] like Figure 1 FIG. 1 is a flow chart of a method for intelligently monitoring and reminding a patient's dietary behavior according to an embodiment of the present disclosure. The method 100 includes:

[0075] S110: Collecting patient diet image data containing calibrated tableware, and preprocessing the diet image data, wherein the patient diet image data is associated with patient case data.

[0076] In some embodiments, in order to build a scientific and personalized intelligent diet intervention system for cancer patients, multi-source data collection and labeling system construction are first carried out, mainly including the following three aspects:

[0077] (a) Demand research and indicator determination

[0078] Based on the key knowledge points summarized in the "Core Knowledge Framework of Scientific Nutrition for Cancer Prevention and Treatment," we conducted questionnaire interviews with clinical experts from nutrition, oncology, and gastroenterology departments to systematically identify the nutritional misconceptions and key demands of cancer patients during each treatment phase (such as radiotherapy, chemotherapy, and postoperative recovery). We ultimately determined a dietary monitoring indicator system, including but not limited to the following dimensions:

[0079] Food categories: cereals and potatoes, meat, eggs, milk, soy products, fruits, vegetables, nuts, dairy products, etc.;

[0080] Intake: Estimate food volume, mass, and calories based on tableware dimensions and image depth data;

[0081] Nutrients: carbohydrates, protein, fat, dietary fiber, vitamins, minerals, etc.;

[0082] Cooking methods: Nutritional retention and potential health risks associated with raw, steaming, frying, grilling, and pickling methods;

[0083] Eating behavior: eating frequency, time, speed, and whether it is accompanied by high-risk behaviors (such as eating while watching TV).

[0084] (b) Image data acquisition and standardization processing

[0085] We collected real-world dietary image data covering different disease types (such as gastric cancer, colorectal cancer, breast cancer, and lung cancer) and treatment stages. Data sources include inpatient meal photos, online patient self-reports, and hospital cafeteria electronic menu records. To ensure data quality, we adopted the following collection and processing process:

[0086] Shooting standards: standardize the acquisition distance, lighting, and background, and use calibrated tableware (bowls and plates with scales) to ensure quantifiable dimensions;

[0087] Image preprocessing: including image enhancement (denoising, brightness adjustment), image segmentation (separating food area from background), and data cleaning (eliminating blurry and non-standard images);

[0088] Labeling mechanism: Build a multi-label labeling system that combines manual and model collaboration, covering attribute labels such as food categories, the relationship between staple and non-staple foods, and unhealthy eating behaviors (such as high salt, high fat, and low fiber).

[0089] In some embodiments, preprocessing the collected dietary image data includes:

[0090] (1) The collected dietary image data is denoised using a logarithmic multiplication algorithm based on extended operand decomposition.

[0091] To improve the efficiency and noise suppression capabilities of Gaussian filtering operations in image processing, this paper proposes a logarithmic multiplication algorithm based on extended operand decomposition and applies it to the convolution operation of Gaussian filtering. By converting traditional multiplication to addition and table lookup operations, the algorithm significantly reduces logic complexity and power consumption. This algorithm is particularly suitable for edge device image processing tasks. The algorithm specifically includes the following steps:

[0092] (a) Decomposition of operands: Decompose two n-bit unsigned binary image pixels X and Y into three operands:

[0093]

[0094] Where A: the result of bitwise AND of X and Y, that is, the intersection of X and Y;

[0095] B: The bitwise AND of (the inverse of X) and Y, that is, the part of Y that is not in X;

[0096] C:X and The bitwise AND of , that is, the part of X that is not in Y;

[0097] X: the first input operand (binary image pixel or filter kernel value);

[0098] Y: second input operand (binary image pixel or filter kernel value);

[0099] : Bitwise inversion of X; : Bitwise inversion of Y; Bitwise AND operation.

[0100] (b) Locating the most significant bit: Use the first 1 detection circuit module to find the most significant bit positions of A, B, and C, which are recorded as: (the most significant bit position of A), (the most significant bit position of B), (most significant bit position of C).

[0101] (c) Extract the mantissa: All the bits to the right of the most significant bit are called the mantissa: (The last digit of A), (The last digit of B), (ending with C)

[0102] (d) Logarithmic conversion: Convert operands A, B, and C to logarithmic form using a lookup table and shifters:

[0103] ;

[0104] (e) Logarithmic addition and antilogarithmic calculation: For the product term, logarithmic addition and antilogarithmic transformation are used to obtain:

[0105] ;

[0106] Furthermore, the products of each item are obtained by looking up the antilogarithm table:

[0107]

[0108]

[0109]

[0110] ;

[0111] in, : Antilogarithmic operation; The square of itself Cross-product terms, used to recover the full product.

[0112] (f) Error processing and cumulative output:

[0113] If any operand is zero, its logarithm is set to zero by the zero detector. And ignore its product terms. Finally, add up all the legal product terms and get Approximate value of :

[0114] ;

[0115] It should be pointed out that this method can be used to accelerate the convolution operation between each pixel in the image and its kernel weight, and deployed on a low-power FPGA / ASIC accelerator to achieve efficient image preprocessing.

[0116] (2) In order to more accurately identify visually salient areas in an image, the preprocessing of the above-mentioned food image data further includes constructing a center-periphery difference saliency map and a background contrast saliency map based on the denoised food image data, and weightedly fusing the center-periphery difference saliency map and the background contrast saliency map to generate a composite saliency map, which specifically includes the following steps:

[0117] (g) Construct a center-periphery saliency map D(a,b) to measure the difference between a pixel in the image and its surrounding area, emphasizing the saliency of the central area of ​​the image:

[0118] ;

[0119] Where (a, b) is the coordinate of the current pixel; I(a, b) is the grayscale value or intensity value of the pixel (a, b) in the original image; S(a, b) is the average value of the neighboring pixels around the pixel (a, b) (usually the local mean is calculated within a certain window); D(a, b) is the difference between the current pixel and the mean of its surrounding pixels. The larger the value, the more "prominent" the pixel is in the local area.

[0120] (h) Construct a background contrast map C(a,b) to measure the overall intensity difference between the pixel and the image background, thereby highlighting areas with large contrast with the background:

[0121] ;

[0122] Wherein, μ(B) is the average intensity value of the background area (such as the edge of the image or the preset area); C(a,b) is the difference between the pixel and the background intensity. The larger the difference, the greater the difference between the pixel and the background, and the more significant it is.

[0123] (i) Constructing a composite saliency map M(a,b): The center-periphery saliency map and the background contrast map are weightedly fused to form the final saliency map:

[0124] ;

[0125] Where, D(a,b): center-periphery saliency map value; C(a,b): background contrast saliency map value;

[0126] α: weight coefficient of the center-periphery saliency map, β: weight coefficient of the background contrast saliency map, α+β=1; M(a,b): the fused composite saliency map value, which is used to identify the most attractive or most likely target area in the image.

[0127] It should be pointed out that the composite saliency map method proposed in this disclosure can adapt to the changes in the contrast intensity between foreground and background in different scenes, while retaining local mutation features and global background differences, enhancing the detectability of target areas, and providing more stable and robust saliency map support for subsequent image classification.

[0128] S120: extracting at least three types of local invariant feature descriptors from the preprocessed diet image data, wherein the local invariant feature descriptors include scale-invariant features, binary high-efficiency features, and improved nonlinear scale-space features.

[0129] In some embodiments, at least three types of local invariant feature descriptors can be extracted from the preprocessed diet image data, including scale-invariant features (SIFT), binary efficient features (BRISK), and improved nonlinear scale-space features (PKAZE).

[0130] Specific feature description extraction methods include:

[0131] (1) SIFT generates different scale spaces through Gaussian convolution to construct Gaussian difference pyramids, which divide the neighborhood of key points (usually Pixels) are divided into 4×4 sub-regions, and the gradient histograms in 8 directions are calculated for each sub-region to generate a 128-dimensional feature vector

[0132] ;

[0133] Among them, each is an 8-dimensional direction histogram.

[0134] (2) BRISK generates a binary descriptor by comparing the intensity of 58 sampling points around the key point :

[0135] ;

[0136] in, : Sampling point Pixel value, sampling point pair set: predefined 58×2 sampling point coordinate pairs.

[0137] (3) PKAZE (Improved Nonlinear Scale Space Features) detects key points in nonlinear scale space by solving the determinant of the Hessian matrix:

[0138] ;

[0139] in, : image after nonlinear diffusion;

[0140] : The determinant value of the Hessian matrix, used to detect spot-like features in the image;

[0141] : The second-order partial derivative of the image in the x direction, reflecting the intensity change rate in the x direction;

[0142] : The second-order partial derivative of the image in the y direction, reflecting the intensity change rate in the y direction;

[0143] : The image is The second-order mixed partial derivative of the direction reflects the change of the gradient direction.

[0144] Key points to be met:

[0145] Threshold and

[0146] : The trace of the Hessian matrix, equal to ;

[0147] Where r is the characteristic response ratio (preset parameter, usually 10), which is used to filter edge responses.

[0148] Sample 4×4 subregions around the keypoint in logarithmic polar coordinates, each subregion contains 8 sampling points. Rotate the sampling point coordinates to the main direction:

[0149] ;

[0150] Among them, (x, y): original sampling point coordinates (relative to the key point center), Coordinates after rotation (with the main direction as the positive direction of the x-axis) The main direction of the keypoint (in radians), determined by the peak of the gradient histogram.

[0151] In the rotated coordinate system, calculate the gradient of each sampling point :

[0152] ;

[0153] in, : The gradient vector of the i-th sampling point, : The coordinates of the sampling points after rotation and normalization.

[0154] The gradient vector in each sub-region is projected to the horizontal and vertical directions to generate a 4×4×2=32-dimensional vector, and then nonlinear enhancement and L2 normalization are performed:

[0155] ;

[0156] in, ;

[0157] Subregion i: gradient statistics vector of the i-th 4×4 subregion;

[0158] : Gaussian weight of the jth sampling point (decreasing with the distance from the center of the key point);

[0159] : The gradient x component of the j-th sampling point in sub-region i;

[0160] : The gradient y component of the j-th sampling point in sub-region i.

[0161] S130: After normalizing the extracted feature descriptors, perform feature fusion to generate a high-dimensional fused feature vector.

[0162] In some embodiments, the above three types of features are normalized and concatenated according to the channel dimension to form a unified description vector:

[0163] ;

[0164] in, : SIFT feature vector; : BRISK feature vector; : PKAZE feature vector; concat: vector concatenation operation; : The fused feature vector integrates the features of three types of key points and has higher discriminative ability and robustness.

[0165] S140: Input the high-dimensional fusion feature vector into a pre-trained deep neural network model for feature classification, and output a food category label.

[0166] The fused feature vector is fused based on the trained deep neural network model and the pre-built dietary knowledge graph. Perform classification and output probability vector:

[0167] ;

[0168] in, : Predict the category probability vector (such as the probability of "red meat", "white meat", "cancer-preventing vegetables", etc.); : The fused feature vector obtained in step S130; : The weight matrix of the fully connected layer; : bias vector; Softmax: normalization function used to convert the output into a probability distribution.

[0169] In some embodiments, the graph nodes of the above-mentioned pre-constructed dietary knowledge graph include food categories, nutritional components, contraindicated diseases, and alternative suggestions. The graph association relationships include: food-category, category-recommended intake, disease-contraindicated food. The graph can be used to provide a label system for neural network classification.

[0170] Assume that the category set is Red meat, white meat, refined rice and flour, whole grains, cancer-preventing vegetables, high-sugar beverages , the final category is:

[0171] ;

[0172] in, : predicted probability of category c; arg max: find the category index corresponding to the maximum value; class: final classification result.

[0173] S150: Calculating the patient's intake based on the diet image data, food category labels and the size of the calibrated tableware.

[0174] In some embodiments, calculating the patient's intake amount based on the dietary image data, the food category labels, and the size of the calibrated tableware includes:

[0175] First, based on the preprocessed food image data, the food area is located through the composite saliency map and its two-dimensional projection area G is extracted;

[0176] Acquire food image depth information R according to the food image data;

[0177] Calculating a proportionality coefficient φ based on the size of the calibrated tableware, wherein the proportionality coefficient depends on the actual area of ​​the tableware and the pixel area of ​​the tableware image;

[0178] The volume of ingested food is calculated based on the two-dimensional projection area, the depth information of the diet image and the proportional coefficient, using the formula:

[0179] ;

[0180] Where Q is the volume of the food; R is the depth information of the food image; G is the two-dimensional projection area of ​​the food in the image; φ is the actual scale factor (calibrated by the size of the tableware, such as the known diameter of the bowl / plate).

[0181] Furthermore, based on the volume of food intake and the pre-built volume and density conversion model, the mass of food intake is calculated. The specific formula is:

[0182] ;

[0183] Where J is the estimated mass of food (g); ρ is the density of food (which can be found in the table, such as red meat is approximately , lettuce is about ).

[0184] The patient's intake is further calculated based on the quality of the food consumed and the pre-built mass and calorie conversion model to estimate the food energy:

[0185] ;

[0186] Where, E: energy (kcal); M: mass (g); : Calories per unit mass (can be found in the table, such as red meat is about ).

[0187] S160: Provide intelligent personalized reminders based on the food category labels, patient case data, and patient intake.

[0188] To achieve a closed-loop management system of "perception-understanding-intervention," the system needs to integrate recognition results, knowledge rules, and medical records to implement personalized dietary behavior reminders. This mainly includes the following three types of rules:

[0189] (a) Positive dietary behavior recommendations

[0190] When the recognition result meets the healthy eating standards, the system will proactively push reinforcement encouragement information to help patients develop good eating habits. The rule examples are as follows:

[0191] If you consume foods rich in dietary fiber such as "whole grains, beans, green leafy vegetables, and fruits", the system will prompt: "This diet is rich in dietary fiber, which is beneficial to intestinal health. It is recommended to continue."

[0192] If you identify the combination of "low GI carbohydrates (such as brown rice, purple sweet potatoes)" and "high-quality protein (such as eggs, tofu, fish)", the system recommends: "Low GI combined with high-quality protein helps stabilize blood sugar and enhance immunity."

[0193] (b) Warning of high-risk eating behaviors

[0194] Based on case data, once the system detects unhealthy foods related to cancer patients' dietary taboos (such as high-salt foods, fried foods, barbecued foods, and alcoholic beverages), it immediately issues a risk warning, supplemented by alternative suggestions and popular science content:

[0195] Example of warning rule: "A combination of pickled foods (pickles, salted fish) and fried foods has been detected, posing a high risk of cancer."

[0196] Substitution suggestion form: The system automatically pushes "animated skit + expert interview" videos (such as "Eat the golden combination of cancer-preventing foods") and substitution lists (such as "Replace fried potato chips with steamed vegetables").

[0197] (c) Personalized intervention rules

[0198] Introduce patient medical history information (disease type, stage, treatment method, previous nutritional assessment) to build a personalized reminder model, for example:

[0199] Chemotherapy patients: Identifying "too little total amount of food + single color + no liquid", the system prompts: "Nausea and vomiting are common during chemotherapy. It is recommended to adopt a warm, light liquid diet and try to eat small meals frequently."

[0200] Postoperative patients: If no foods rich in zinc or vitamin C (such as nuts, animal liver, kiwi) are detected continuously, the system will push: "During the postoperative recovery period, the intake of foods containing zinc and vitamin C should be increased to help wound healing."

[0201] Patients with diabetes and tumors: When consuming high-sugar desserts, a prompt will appear: "The current sugar intake is too high. It is recommended to choose xylitol or erythritol as a substitute."

[0202] According to the above embodiments of the present disclosure, the following technical effects are achieved:

[0203] (1) This paper proposes a logarithmic multiplication algorithm based on extended operand decomposition. The algorithm optimizes the multiplication operation in traditional image filtering into addition and table lookup, effectively improving the image denoising efficiency and the computing performance of edge devices during the Gaussian filtering process, and significantly reducing the logic operation complexity and power consumption.

[0204] (2) This paper proposes a method to fuse the center-periphery difference saliency map and the background contrast saliency map. By constructing a composite saliency map, the saliency information of the target area in the image is effectively extracted, the positioning accuracy of foreground targets in complex scenes (such as food on a plate) is improved, and the robustness of subsequent feature extraction and classification is enhanced.

[0205] (3) This paper proposes an improved nonlinear scale-space feature extraction method (PKAZE), which effectively extracts high-order structural features such as edges and spots in food images through nonlinear diffusion and Hessian matrix structure enhancement operations, thereby improving the stable expression capability of highly complex food images.

[0206] (4) This paper proposes a fusion strategy of three feature descriptors: SIFT, BRISK and PKAZE. By uniformly normalizing and channel-level splicing, a high-dimensional feature vector is generated, which takes into account texture details, scale changes and rotation invariance, and achieves a more comprehensive and robust image feature representation.

[0207] (5) This disclosure proposes a method for estimating food intake, mass, and calories by combining image depth information with tableware size, introducing a volume calculation and density conversion model to support the inference of actual intake mass from two-dimensional images, thereby providing patients with quantitative and personalized dietary assessments and intelligent reminders to meet the nutritional intervention needs of different diseases and treatment stages.

[0208] It should be noted that, for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0209] The above is an introduction to the method embodiment. The following is a further explanation of the solution disclosed in the present disclosure through an apparatus embodiment.

[0210] Figure 2 FIG. 1 shows a block diagram of a patient's dietary behavior intelligent monitoring and reminder device according to an embodiment of the present disclosure. Figure 2 As shown, the apparatus 200 includes:

[0211] An image acquisition and preprocessing module 210 is configured to acquire patient diet image data including calibrated tableware and preprocess the diet image data, wherein the patient diet image data is associated with the patient case data;

[0212] a feature extraction module 220 for extracting at least three types of local invariant feature descriptors from the preprocessed diet image data, wherein the local invariant feature descriptors include scale invariance, binary efficient representation, and nonlinear structure enhancement characteristics;

[0213] The feature fusion module 230 is used to normalize the extracted feature descriptors and then perform feature fusion to generate a high-dimensional fused feature vector;

[0214] A food category label recognition module 240 is configured to input the high-dimensional fusion feature vector into a pre-trained deep neural network model for feature classification and output a food category label;

[0215] The patient's actual food intake module 250 is used to calculate the patient's food intake based on the diet image data, food category labels and the size of the calibrated tableware;

[0216] The personalized reminder module 260 is used to provide intelligent personalized reminders based on the food category labels, patient case data and patient intake.

[0217] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0218] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0219] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0220] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0221] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0222] Multiple components in the electronic device 300 are connected to the I / O interface 305, including an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0223] Computing unit 301 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed onto electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by computing unit 301, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, computing unit 301 may be configured to perform method 100 in any other suitable manner (e.g., via firmware).

[0224] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0226] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0228] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0229] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0230] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0231] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for intelligently monitoring and reminding patients' dietary behavior, characterized in that: include: Collecting patient diet image data containing calibrated tableware, and preprocessing the diet image data, wherein the patient diet image data is associated with the patient case data; Extracting at least three types of local invariant feature descriptors from the preprocessed diet image data, wherein the local invariant feature descriptors include scale-invariant features, binary high-efficiency features, and improved nonlinear scale-space features; After normalizing the extracted feature descriptors, perform feature fusion to generate a high-dimensional fusion feature vector; Inputting the high-dimensional fusion feature vector into a pre-trained deep neural network model for feature classification, and outputting a food category label; Calculating the patient's intake based on the diet image data, the food category label, and the size of the calibrated tableware; Provide intelligent personalized reminders based on the food category labels, patient case data, and patient intake; The preprocessing of the diet image data includes: performing image denoising on the diet image data based on a logarithmic multiplication algorithm of extended operand decomposition; Based on the denoised food image data, a center-periphery difference saliency map and a background contrast saliency map are constructed respectively; The center-periphery difference saliency map and the background contrast saliency map are weightedly fused to generate a composite saliency map.

2. The method according to claim 1, characterized in that The logarithmic multiplication algorithm based on extended operand decomposition performs image denoising on the diet image data, comprising: Decomposing the image pixel X of the diet image data and the Gaussian filter kernel weight Y into three binary operands A, B, and C; Extract the most significant bit position of each operand and the last digit , determine the order of magnitude of each operand; By pre-training the logarithmic table and the shifter, the operands A, B, and C are converted into logarithmic mode. ; For the operands A, B, C and Perform logarithmic addition and antilogarithmic calculation to obtain the product terms between the operands; According to the product between the operands, an approximate value of X×Y is determined to complete the image pixel filtering convolution.

3. The method according to claim 1, characterized in that The method of constructing a center-periphery difference saliency map and a background contrast saliency map based on the denoised food image data includes: Based on the denoised food image data, the absolute value of the difference between the grayscale value or intensity value I(a,b) of the pixel point (a,b) and the average value S(a,b) of the neighborhood pixels around the pixel point (a,b) is calculated to obtain the center-periphery saliency map: Among them, the neighborhood range of pixel point (a, b) is defined by a sliding window; Based on the denoised food image data, the absolute value of the difference between the grayscale value or intensity value I(a,b) of the pixel (a,b) and the average intensity value μ(B) of the image background area is calculated to obtain the background contrast saliency map: The image background area is the image edge or a preset non-target area.

4. The method according to claim 3, characterized in that The weighted fusion of the center-periphery difference saliency map and the background contrast saliency map to generate a composite saliency map includes: The composite saliency map is obtained by linearly combining D(a,b) and C(a,b) with weight coefficients α and β: Among them, α+β=1, and the weight coefficient is adaptively adjusted according to the scenario.

5. The method according to claim 1, wherein At least three types of local invariant feature descriptors are extracted from the preprocessed diet image data, wherein the local invariant feature descriptors include scale-invariant features, binary high-efficiency features, and improved nonlinear scale-space features, including: Scale-invariant features: achieved through Gaussian scale space construction and directional gradient statistics; Binary efficient features: Generate binary descriptors based on comparison of sampling point intensities; Improved nonlinear scale-space features: Detect Hessian matrix feature points in nonlinear diffusion space.

6. The method according to claim 5, characterized in that The extracted feature descriptors are normalized and then feature fused to generate a high-dimensional fused feature vector, including: The scale-invariant features, binary efficient features and improved nonlinear scale space features are normalized to zero mean and unit variance to eliminate dimensional differences; The normalized features are concatenated through the channel dimension to generate a high-dimensional fused feature vector.

7. The method according to claim 1, characterized in that The calculating of the patient's intake amount according to the diet image data, the food category label and the size of the calibrated tableware includes: Based on the pre-processed food image data, the food area is located through the composite saliency map and its two-dimensional projection area is extracted; acquiring food image depth information according to the food image data; Calculating a scaling factor based on the size of the calibrated tableware, wherein the scaling factor depends on the actual area of ​​the tableware and the pixel area of ​​the tableware image; Calculating the volume of ingested food based on the two-dimensional projection area, depth information of the diet image, and a proportional coefficient; Calculate the mass of food intake based on the volume of food intake and a pre-built volume and density conversion model; Patient intake was calculated based on the mass of food ingested and a pre-constructed mass-to-calorie conversion model.

8. An intelligent monitoring and reminder device for patient eating behavior, characterized in that: include: An image acquisition and preprocessing module is used to acquire patient diet image data including calibrated tableware and preprocess the diet image data, wherein the patient diet image data is associated with the patient's case data; the preprocessing of the diet image data includes: performing image denoising on the diet image data based on a logarithmic multiplication algorithm of extended operand decomposition; Based on the denoised food image data, a center-periphery difference saliency map and a background contrast saliency map are constructed respectively; weighted fusion of the center-periphery difference saliency map and the background contrast saliency map to generate a composite saliency map; a feature extraction module, configured to extract at least three types of local invariant feature descriptors from the preprocessed diet image data, wherein the local invariant feature descriptors include scale invariance, binary efficient representation, and nonlinear structure enhancement characteristics; The feature fusion module is used to normalize the extracted feature descriptors and then fuse them to generate a high-dimensional fused feature vector; A food category label recognition module is used to input the high-dimensional fusion feature vector into a pre-trained deep neural network model for feature classification and output a food category label; a patient's actual food intake module, configured to calculate the patient's food intake based on the diet image data, food category labels, and the size of the calibrated tableware; The personalized reminder module is used to provide intelligent personalized reminders based on the food category labels, patient case data and patient intake.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

Images