Clinical meal intake evaluation method, system and device based on image recognition
By converting pre- and post-meal images into three-dimensional voxel grids and constructing an interactive potential energy model, the interaction between foods is quantified, solving the problem of inaccurate assessment in existing technologies and achieving higher-precision assessment of food intake.
Patent Information
- Application Number
- CN202511002628.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-10-31
AI Technical Summary
Existing technologies have low accuracy in assessing clinical food intake due to variations in imaging conditions and neglect of food interactions. Traditional methods are highly subjective, time-consuming, labor-intensive, and difficult to accurately assess the intake of mixed foods and irregularly shaped foods.
An image recognition-based approach is used to convert pre-meal and post-meal images into three-dimensional voxel grid data. Spatial alignment and interactive potential models are used to quantify the interactions between foods, and visual features are combined to correct calorie calibration, thereby improving the accuracy of the assessment.
It effectively overcomes the assessment errors caused by factors such as perspective, changes in lighting, and food displacement, improves the accuracy and precision of meal intake assessment, and reduces the subjectivity and workload of manual estimation.
Smart Images
Figure CN120878073A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical meal intake assessment technology, and in particular to a clinical meal intake assessment method, system and device based on image recognition. Background Technology
[0002] In clinical settings, accurate monitoring of patients' nutritional status is crucial, directly impacting their recovery speed and treatment outcomes. Daily dietary intake assessment is a core component of nutritional monitoring. Traditional methods for recording clinical meal intake rely heavily on manual observation and estimation by medical staff or patients' families. For example, visually recording that a patient has consumed "half a bowl of rice" or "two pieces of pork ribs." This method has obvious drawbacks: First, it is highly subjective; different recorders may have vastly different interpretations of "half a bowl" or "one piece," leading to inconsistent and error-prone results. Second, it is time-consuming and labor-intensive, increasing the workload of medical staff, especially in departments with a large number of patients. Third, for mixed foods (such as soup or mixed dishes) and irregularly shaped foods, manually estimating their intake and nutritional composition is extremely difficult, making accuracy nearly impossible.
[0003] With the development of computer vision technology and artificial intelligence, some solutions have emerged that attempt to use image recognition technology to assist in dietary assessment. These solutions typically involve users taking photos of their food before and after meals through mobile apps, and then using background image recognition algorithms to identify food types and estimate portions. However, existing technologies still face many challenges when applied to clinical scenarios requiring high accuracy.
[0004] In related technologies, processing is mostly performed at the two-dimensional image level, such as estimating consumption by comparing changes in the pixel area of food regions in pre-meal and post-meal images. This method is extremely sensitive to changes in shooting angle, lighting conditions, and food placement. Slight changes in viewing angle can cause significant changes in the projected area of food in the image, thus introducing huge errors. In addition, food may shift, deform, or be obscured by other food during the meal, and two-dimensional pixel comparison is difficult to accurately handle these complex situations.
[0005] In related technologies, each food item is typically analyzed as an independent entity, neglecting the interactions between foods. In actual cooking and dining, ubiquitous physical and chemical interactions occur between foods; for example, broth soaks into rice and vegetables, and oils from dishes adhere to adjacent foods. These interactions significantly alter the local nutritional composition and caloric density of food. Simply multiplying the identified food items by their values in a standard nutritional database fails to reflect this real-world complexity, leading to discrepancies between the assessment results and the actual nutritional value ingested. Summary of the Invention
[0006] This application provides a clinical meal intake assessment method, system, and device based on image recognition, which aims to improve the technical problem of low assessment accuracy caused by changes in shooting conditions and neglect of food interactions in related technologies.
[0007] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:
[0008] In a first aspect, this application provides a clinical meal intake assessment method based on image recognition, comprising: acquiring a pre-meal image and a post-meal image containing food; generating a first voxel grid based on the pre-meal image, wherein each first voxel in the first voxel grid corresponds to a food region in the pre-meal image, and each first voxel is assigned a food category identifier and an initial potential value; generating a second voxel grid based on the post-meal image, wherein each second voxel in the second voxel grid corresponds to a food region in the post-meal image; and spatially aligning the second voxel grid with the first voxel grid. Obtain the aligned target voxel grid; identify the set of consuming voxels in the first voxel grid that are not matched in the target voxel grid; based on the food category identifier and initial potential energy value of each consuming voxel in the set of consuming voxels, and combined with the spatial proximity relationship between the consuming voxel and other consuming voxels, obtain the interaction potential energy correction value characterizing the interaction between foods; determine the effective consumption value of each consuming voxel according to the initial potential energy value of each consuming voxel in the set of consuming voxels and the corresponding interaction potential energy correction value; sum up the effective consumption values of all consuming voxels to obtain the total food intake.
[0009] In one possible implementation of the first aspect, the step of generating a first voxel grid based on the pre-meal image includes: segmenting the pre-meal image to obtain contour information and texture information of multiple food regions; setting a three-dimensional coordinate system for the pre-meal image and constructing a first voxel grid covering the entire dining area; mapping the contour information and texture information of the multiple food regions to the first voxel grid, and activating the first voxels that overlap with the food regions; and assigning the food category identifier and the initial potential value to each activated first voxel.
[0010] In one possible implementation of the first aspect, the initial potential energy value is a combination of the volume of the food region corresponding to the first voxel and the standard nutrient density value.
[0011] In one possible implementation of the first aspect, the step of spatially aligning the second voxel grid with the first voxel grid includes: extracting the edge or specific mark of the tableware as stable feature points from the first voxel grid and the second voxel grid; constructing an affine transformation matrix from the coordinate system of the second voxel grid to the coordinate system of the first voxel grid; and applying the affine transformation matrix to the second voxel grid to generate the target voxel grid.
[0012] In one possible implementation of the first aspect, the step of obtaining the interaction potential correction value characterizing the interaction between foods includes: for any target consuming voxel in the set of consuming voxels, obtaining other neighboring consuming voxels in its neighborhood; querying a preset food interaction matrix to obtain the interaction coefficient between the food category of the target consuming voxel and the food category of each of the neighboring consuming voxels; calculating a distance attenuation factor based on the spatial distance between the target consuming voxel and each of the neighboring consuming voxels; combining the interaction coefficient corresponding to each of the neighboring consuming voxels with the distance attenuation factor to obtain multiple local interaction components; and aggregating the multiple local interaction components to obtain the interaction potential correction value of the target consuming voxel.
[0013] In one possible implementation of the first aspect, the step of determining the effective consumption value of each consumed voxel is to obtain the effective consumption value by combining the initial potential energy value and the interaction potential energy correction value.
[0014] In one possible implementation of the first aspect, the method further includes: after determining the total food intake, performing a calorie calibration correction on the total food intake; the calorie calibration correction step includes: extracting image texture features of the food regions corresponding to the voxels of different food categories in the consumed voxel set in the pre-meal image; querying a preset texture-calorie density mapping relationship based on the image texture features to obtain a calorie correction factor for the corresponding food category; and applying the calorie correction factor to the total food intake to obtain the corrected final food intake.
[0015] Secondly, this application also provides a clinical meal intake assessment system based on image recognition, comprising: an image acquisition module for acquiring pre-meal and post-meal images containing food; a raster generation module for generating a first voxel raster based on the pre-meal image and a second voxel raster based on the post-meal image, wherein each first voxel in the first voxel raster corresponds to a food region in the pre-meal image, and each first voxel is assigned a food category identifier and an initial potential energy value; and a spatial alignment module for spatially aligning the second voxel raster with the first voxel raster to obtain... The aligned target voxel grid; the consumption identification module, used to identify the set of consumption voxels in the first voxel grid that is not matched in the target voxel grid; the intake calculation module, used to obtain the interaction potential energy correction value characterizing the interaction between foods based on the food category identifier and initial potential energy value of each consumption voxel in the set of consumption voxels, combined with the spatial proximity relationship between the consumption voxel and other consumption voxels; determine the effective consumption value of each consumption voxel according to the initial potential energy value and the interaction potential energy correction value; and accumulate the effective consumption values of all the consumption voxels to obtain the total food intake.
[0016] Thirdly, this application also provides a clinical meal intake assessment device based on image recognition, including a processor, a memory, and a communication bus for communicatively connecting the processor and the memory; the memory stores a computer program, and the processor is configured to implement the method as described in any one of the first aspects when executing the computer program.
[0017] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any one of the first aspects. Attached Figure Description
[0018] Figure 1 A flowchart illustrating an image recognition-based clinical meal intake assessment method provided for some embodiments of this application;
[0019] Figure 2 A schematic diagram of the structure of an image recognition-based clinical meal intake assessment system provided for some embodiments of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.
[0021] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0022] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.
[0023] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "electrical connection" can refer to the manner in which an electrical connection is used to achieve signal transmission.
[0024] As used herein, “about,” “approximately,” or “approximately” includes the stated value and a reference value within an acceptable range of deviation from the given value, characterized in that the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement method).
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that, in order to make the specification of this patent concise and easy to understand, detailed descriptions of some known functions and components have been omitted in the embodiments of this invention.
[0026] This application provides a method, system, and apparatus for assessing clinical meal intake based on image recognition. The method converts pre- and post-meal two-dimensional images into structured three-dimensional voxel grid data, and performs alignment, comparison, and calculation within this grid space. This effectively overcomes assessment errors caused by factors such as viewing angle, lighting changes, and food displacement and deformation in traditional two-dimensional image processing. In particular, this application constructs an interactive potential energy model to quantify the mutual influences (such as the effects of broth infiltration and flavor transfer on nutritional value) of different foods placed adjacent to each other, thereby improving the accuracy of meal intake assessment.
[0027] refer to Figure 1The diagram illustrates a flowchart of a clinical meal intake assessment method based on image recognition, provided by an embodiment of the present invention. This method can be applied to computer devices, such as servers, personal computers, or applications integrated into mobile terminals (such as smartphones and tablets).
[0028] S100: Obtain pre-meal and post-meal images containing food.
[0029] Users (such as patients or caregivers) operate the system using devices equipped with cameras (such as specific apps on smartphones). Before the meal, a photo is taken of the plate or bowl with the food set, creating a pre-meal image. After the meal, take another photo of the same plate or bowl containing the remaining food to obtain a post-meal image. .
[0030] To ensure consistent image quality and subsequent processing, the application can provide an auxiliary focus frame and level to guide the user to take photos at similar angles and distances. Optionally, a reference object of known size (e.g., a standard-sized card) can be placed next to the plate for subsequent size calibration and image correction. The acquired images are typically RGB color images, containing color, texture, and spatial information.
[0031] S200. Based on the pre-meal image, generate a first voxel grid. .
[0032] Acquire pre-meal images Subsequently, this method does not perform direct analysis at the pixel level, but instead transforms it into structured data, the first voxel raster. The purpose is to transform two-dimensional, unstructured pixel information into three-dimensional, discretized spatial occupancy information. This step can be further divided into the following sub-steps:
[0033] S210, Segment the pre-meal image This yields the contour and texture information of multiple food regions.
[0034] The system first processes the pre-meal image. Perform image segmentation. The segmentation here aims to identify each individual food category in the image and delineate its boundaries. For example, a dish may contain rice, broccoli, and chicken nuggets; the segmentation algorithm needs to accurately distinguish these three parts. This can be achieved using an instance segmentation model from deep learning, which can not only identify that a pixel belongs to "food" but also distinguish which "food instance" it belongs to. To enable those skilled in the art to implement this invention, the method for obtaining the instance segmentation model is briefly described here.
[0035] For example, an image dataset specifically designed for clinical dining scenarios is collected and constructed. This dataset contains food images under various lighting conditions, shooting angles, and tableware types. Each image in the dataset is annotated at the pixel level. For each individual food item, the annotator needs to use specialized tools to delineate its precise outline, generating a pixel-level mask. Crucially, this annotation not only assigns a food category label (e.g., "rice," "chicken") to each mask but also assigns a unique instance ID to different individuals within the same category. For example, two pieces of chicken in a plate would be labeled "chicken-instance 1" and "chicken-instance 2," respectively. A mature instance segmentation model architecture is selected, such as, but not limited to, MaskR-CNN, YOLACT, or SOLOv2. The annotated dataset is then input into the selected model architecture for training. During training, the model continuously learns the complex mapping relationship from raw image pixels to the "category + instance" mask. The difference between the model's predictions and the manually labeled true values is quantified by defining a loss function, and an optimizer (such as Adam) is used to continuously adjust the parameters inside the model to minimize this loss.
[0036] Through the above training process, a pre-trained model capable of performing food instance segmentation tasks can be obtained. This model is then embedded and integrated into the evaluation system of this invention. Therefore, when executing the method of this invention, the model can provide accurate food region, category, and instance information as input for subsequent steps.
[0037] This model segmentation yields a set of food region masks, each mask corresponding to a specific food. For each mask, its contour information (such as the boundary point coordinate sequence) and internal texture information (such as feature vectors extracted using methods like Gray-Level Co-occurrence Matrix (GLCM) and Local Binary Pattern (LBP)) can be extracted. Simultaneously, a food classifier (such as the classification function embedded in the instance segmentation model) identifies each segmented food region and assigns it a food category label. .
[0038] For example, suppose a pre-meal image There is a bowl of rice and several pieces of braised pork inside. After segmentation, the system will generate two masks. and At the same time, the classifier identified The corresponding food category is "rice". The corresponding food category is "braised pork".
[0039] S220. Establish a three-dimensional coordinate system for the pre-meal image and construct the first voxel grid covering the entire dining area. .
[0040] For example, a virtual three-dimensional coordinate system is established for the pre-meal image. .in, shaft and The axis is parallel to the plane coordinates of the image, while The axis is perpendicular to the image plane and represents height. The plane containing the plate or bowl can be defined as... The reference plane.
[0041] A three-dimensional mesh, namely the first voxel grid, is constructed in this coordinate system. The grid consists of numerous tiny cubic units—voxels. The grid needs to be large enough to completely encompass all the space that food in the image might occupy. Each voxel is identified by its index within the grid. Unique identifier. The physical size of the voxel (e.g., each voxel represents...). The space (of the evaluation) can be pre-defined, which determines the accuracy of the evaluation.
[0042] S230, Map the contour information and texture information of the multiple food regions onto the first voxel grid. Activate the first voxel that overlaps with the food region.
[0043] This step is used to reconstruct the food regions segmented from the 2D image and their occupancy in 3D space. To achieve this, the present invention constructs a Hierarchical Volume Reconstruction Model. This model can be used to generate the 3D height distribution of food based on its category identifier and the geometric features of its 2D mask.
[0044] For example, for each food category A morphology category identifier is configured in a pre-defined food morphology profile library. This library classifies foods into pre-defined morphology categories such as "discrete solids," "accumulated particles," and "fluids within a container." For foods with specific morphology categories obtained from S210... Two-dimensional food region mask The model is from the mask Extract a set of geometric descriptors, including their area. Centroid coordinates and the shortest side length of the smallest bounding rectangle. Query the food morphology feature database to obtain information related to the food category. The associated shape category identifier. Based on this shape category identifier, a corresponding height calculation function is called to obtain the mask. Each pixel within Calculate its three-dimensional height value .
[0045] The height calculation function is directly related to the shape category. If the shape category is "discrete solid" (such as chicken nuggets or cauliflower), a uniform height function is used. All elements located in the mask... Each pixel within the range is assigned a uniform height value, which is a direct function of its geometric description: .in, It is read from a food morphology feature database and is related to food categories. The inherent aspect ratio of the bound element.
[0046] If the morphology is classified as "accumulated particles" or "fluid within a container" (such as rice or porridge in a bowl), a radial basis parabolic function is used. Its height distribution is modeled as a convex surface that smoothly increases in height from the edge of the food region towards the center. height From it to the center of mass normalized distance Decide: Among them, the maximum height It is related to the container depth and the food surface area, while the curvature coefficient These are inherent parameters obtained from a food morphology feature database that characterize the stacking properties of this type of food.
[0047] The system traverses the first voxel grid. Each voxel in Its index is The system projects the center point coordinates of the voxel onto a two-dimensional image plane. If the projection point... Mask that falls in a certain food area Internally, and the height of the voxel (by index) The value determined is less than or equal to the height calculated at that point. Then the voxel Activated and inherits the food category. .
[0048] Through the process of the above-described hierarchical volume reconstruction model, this invention transforms the two-dimensional segmentation results into a set of voxels with a three-dimensional shape, providing a data foundation for subsequent volume comparison and interactive calculation.
[0049] S240, Assign the food category identifier to each activated first voxel. and the initial potential energy value .
[0050] For each activated voxel The system will assign it two core attributes: food category identifier. and initial potential energy value .
[0051] Food category labeling Inherited from the food region of its mapping source. For example, all food regions derived from... The activated voxels all have the food category label "rice".
[0052] Initial potential energy value This is used to quantify the "potential nutritional value" or "potential mass" contained in the food micro-element represented by each voxel. It is not merely volume. In this embodiment, the initial potential energy value is defined as a combination of the voxel volume and the standard nutrient density value for that food category. The calculation method is as follows:
[0053]
[0054] in, It is the standard volume of a single voxel (e.g.) ). It is a food category The standard nutrient density is a value retrieved from a pre-defined food nutrient database. For example, the database might store the standard caloric density of "rice" as... "Broccoli" is This value can also be mass density, such as... .
[0055] After all the steps of S200, the original two-dimensional image It was successfully converted into a structured data body. Each element (voxel) in this data volume explicitly carries a spatial location, food identity, and basic quantitative value.
[0056] S300. Based on the post-meal image, generate a second voxel grid. .
[0057] right Perform the same segmentation, classification, and 3D mapping as described above to generate a second voxel grid representing the spatial distribution of leftover food after a meal. . Each active voxel in the system also has a food category identifier and an initial potential energy value.
[0058] S400, the second voxel grid With the first voxel grid Spatial alignment is performed to obtain the aligned target voxel grid. .
[0059] Because users cannot perfectly match the phone's position, angle, and distance when taking pictures before and after a meal, this can lead to... and The coordinate system is biased. Directly comparing at the voxel level will produce huge errors. Therefore, spatial alignment is necessary. The S400 includes:
[0060] S410, from the first voxel grid and the second voxel grid In this process, the edges or specific markers of the tableware are extracted as stable feature points.
[0061] The key to the alignment described above is finding a stable reference object that remains unchanged throughout both images. Understandably, the edges of tableware (plates, bowls) are an ideal choice. In the segmentation step, in addition to identifying the food, the outlines of the tableware are also identified. Edge detection algorithms (such as Canny edge detection) can extract the circular or square outline of the plate. A series of stable feature points are sampled from these outlines, for example, the four orthogonal points on the circular plate outline, or the four corner points of the square plate. These points have definite features in the 3D voxel grid. coordinate.
[0062] S420. Construct an affine transformation matrix from the coordinate system of the second voxel grid to the coordinate system of the first voxel grid. .
[0063] Suppose The stable feature point set extracted is ,exist The corresponding feature point set extracted is The goal of the system is to find an optimal affine transformation matrix. , making Acting on The subsequent point set and The difference is minimized. That is, minimize the difference. Understandably, this is essentially a least squares problem, solvable using methods such as Singular Value Decomposition (SVD) to obtain a unique solution. or Affine transformation matrix (when using homogeneous coordinates) The matrix It contains transformation information such as rotation, translation, and scaling.
[0064] S430, Apply the affine transformation matrix In the second voxel grid Generate the target voxel grid .
[0065] System traversal Each activator voxel Read its coordinates Then, the obtained transformation matrix is applied. Transform the coordinates to obtain its position. New coordinates in the coordinate system Due to the transformed coordinates It might not be an integer; the system will use interpolation (such as nearest neighbor interpolation) to map it to... At the nearest integer index in the raster. All transformed... The voxels in the matrix together form the aligned target voxel grid. .
[0066] At this point, the distribution of food before and after a meal is unified under the same coordinate system, allowing for precise comparison of individual components.
[0067] S500, Identify the first voxel grid The target voxel grid is not in the specified range. The set of consumed voxels matched in .
[0068] The system traverses the first voxel grid. All activating voxels For each Inspect the aligned target voxel grid Does an active voxel exist at the same coordinate position?
[0069] if The corresponding location has no activated voxel, or although there is a voxel, its food category identifier is not... The difference (for example, where the rice was originally covered by broth) is considered... The food micro-element represented by this voxel is consumed during the meal.
[0070] Should Added to the "Consumable Voxel Set" middle.
[0071] After the traversal is complete, The set precisely contains information about all the food micro-elements (voxels) that have been eaten.
[0072] S600, Based on the consumed voxel set For each consuming voxel, the food category identifier and initial potential energy value are used, combined with the spatial proximity relationship between the consuming voxel and other consuming voxels, to obtain the interaction potential energy correction value characterizing the interaction between foods. .
[0073] Traditional assessment methods simply add up the amount of food consumed, but they overlook an important physical and chemical phenomenon: different foods affect each other when cooked and plated. For example, vegetables soaked in broth have a higher calorie content than stir-fried vegetables; a piece of rice placed next to greasy fried chicken will absorb some of the oil. This effect is significant from a nutritional perspective.
[0074] In-depth research aimed at addressing the issue of localized changes in nutritional value caused by food mixing and broth soaking during clinical meal intake assessment has revealed that each food micro-element (voxel) can be considered a particle with a basic energy (initial potential energy value). When multiple different types of particles are closely arranged in space, they interact with each other through a "field," thereby modifying their respective apparent energies. To this end, this application constructs the following model to describe this interaction. This step can be specifically broken down as follows:
[0075] S610, For the consumed voxel set Any target consumption voxel Obtain other neighboring consumed voxels within its neighborhood. .
[0076] System traversal Each voxel in the set is designated as the "target consumption voxel" for the current processing. Then, the system in Define a neighborhood around This neighborhood can be a or The system searches for all cube-shaped regions within this neighborhood. The other voxels in the set are called "nearby consumption voxels". .
[0077] S620, Query the preset food interaction matrix The target consumed voxel is obtained. Food categories With each of the adjacent consumed voxels Food categories Interaction coefficients between .
[0078] The system pre-builds and stores a "food interaction matrix". This is a symmetric matrix where the rows and columns are category identifiers for various foods. The elements in the matrix... ,Right now , representing food and food When adjacent, this coefficient represents the strength and nature of the interaction per unit distance. It is dimensionless and can be positive (enhancing effect, such as vegetables absorbing broth), negative (inhibiting effect, which is less common but acceptable in the model), or zero (no interaction). These coefficient values are derived from extensive analysis of culinary and nutritional data.
[0079] For example, a simplified interaction matrix It can be represented as:
[0080]
[0081] Here This indicates that there is a strong positive interaction between rice and fried chicken when they are adjacent (the rice absorbs the oil).
[0082] S630, Based on the target consumed voxel With each of the adjacent consumed voxels Spatial distance between Calculate the distance attenuation factor .
[0083] The strength of the interaction is distance-dependent; the greater the distance, the weaker the effect. This invention uses a distance attenuation factor to characterize this phenomenon. The attenuation function can take various forms; this embodiment uses an exponential attenuation form:
[0084]
[0085] in, It is a voxel and The Euclidean distance between the center points can be calculated using voxel indexing. It is a positive decay constant used to control the decay rate, and its value can be calibrated experimentally.
[0086] S640, Each of the adjacent consumed voxels The corresponding interaction coefficient With the distance attenuation factor By combining them, multiple local interactive components are obtained.
[0087] for Each neighboring voxel Its effect on The resulting local interaction effect is determined by both the food type and distance between the participants. This effect, or local interaction component, is calculated as:
[0088]
[0089] S650. Aggregate the multiple local interaction components to obtain the target consumption voxel. The interaction potential energy correction value .
[0090] Target voxel The perceived total interaction is the sum of the local interaction components exerted on it by all other consuming voxels in its neighborhood. Therefore, the interaction potential energy correction value... The calculation is as follows:
[0091]
[0092]
[0093] It is a dimensionless correction coefficient used for quantization. The comprehensive impact of the microenvironment on its potential nutritional value.
[0094] S700, according to the consumed voxel set Each consumed voxel initial potential energy value and the corresponding interaction potential energy correction value Determine the effective consumption value of each consumed voxel. .
[0095] After obtaining the interaction potential energy correction value, the final effective consumption value of each consumed voxel can be calculated. This is the process of correcting its initial potential energy value. In this embodiment, a multiplicative correction method is used:
[0096]
[0097] Here, "1" indicates that its base value is retained. If A positive value indicates that its effective consumption value has been enhanced by surrounding food; a negative value indicates that it has been weakened.
[0098] For example, suppose a "rice" voxel The initial potential energy value is It was right next to a "fried chicken" voxel. Assume the interaction coefficient between them is After distance decay, it becomes .So The interaction potential correction value (considering only this one neighbor) is .but The effective consumption value will be This demonstrates that the rice's effective calories (or equivalent mass) increase due to absorbing the chicken's oil.
[0099] S800: Accumulate the effective consumption values of all the aforementioned consumed voxels to obtain the total food intake. .
[0100] The system will consume voxel sets The final evaluation result is obtained by summing the effective consumption values of all voxels:
[0101]
[0102] this This is the final quantitative assessment of the intake from this meal. Its unit depends on the initial potential energy value. The unit. If The unit is kilocalories (kcal), then The unit is also kilocalories. If the unit is grams (g), the result will also be in grams. This value can be recorded in the clinical management system for nutritional monitoring, medication order adjustments, etc.
[0103] Based on the above embodiments, in order to further improve the accuracy of the evaluation, especially considering that the same food may have drastically different caloric densities due to different cooking methods (such as steaming, frying, and boiling), this application also includes:
[0104] S900, Regarding the total amount of food consumed Perform calorie calibration correction. This step aims to fine-tune the calorie assessment based on the visual characteristics of the food. Specifically, S900 may include:
[0105] S910, Regarding the consumed voxel set Voxels of different food categories were extracted from the pre-meal image. Image texture features of the corresponding food region .
[0106] For each type of food consumed (e.g., all the "chicken" voxels consumed), the system will find their original pre-meal image. The system then extracts a set of predefined texture features from the corresponding food regions in the image. These features may include:
[0107] Gloss characteristics measure the glossiness of food surfaces by analyzing the distribution and intensity of highlight areas, such as the proportion of pixels with brightness values above a certain threshold within an image area.
[0108] Color saturation features are extracted by averaging or histograms of the S (saturation) component in a color space (such as HSV). High saturation is often associated with fresh or sauce-rich foods.
[0109] Texture complexity features, such as calculating the average value or entropy of an image gradient. Fried foods typically have more complex and sharper textures on their surfaces than steamed foods.
[0110] Using the method described above, for each type of food consumed The system will obtain a texture feature vector. .
[0111] S920, Based on the image texture features Query the preset texture-heat density mapping relationship to obtain the heat correction factor for the corresponding food category. .
[0112] The system can pre-store a mapping model (which could be a lookup table, a regression model, or a simple set of rules) that establishes the relationship between texture feature vectors and calorie correction factors. This model can be learned and calibrated from a large number of food sample images with known cooking methods and calorie content.
[0113] For example, the rules can be as follows:
[0114] If the food category is "chicken" and its gloss characteristic value is greater than the threshold... Then its heat correction factor (This suggests that the cooking method may be deep-frying or using a lot of oil).
[0115] If the food category is "chicken" and the gloss characteristic value is less than... Meanwhile, the texture complexity is low, (This suggests that the meat may have been boiled or poached.)
[0116] If the eigenvalues lie between the two, then the factor is... (Standard cooking method).
[0117] S930, Apply the heat correction factor The corrected final meal intake is obtained from the total meal intake. .
[0118] The correction here doesn't directly operate on the total input, but rather weights the contributions of different food categories. First, it's necessary to calculate the contribution of each food category. In total input Contributions to:
[0119]
[0120] Then, the corresponding calorie correction factor is applied to the contribution of each food, and the results are summed to obtain the final result:
[0121]
[0122] this It is a more accurate assessment of meal intake after taking into account visual cues from cooking methods.
[0123] The system and apparatus provided in this invention serve as the physical carrier for implementing the above-described method. The system may include: a front-end application (APP) responsible for image acquisition and simple user interaction; and a back-end server responsible for performing computationally intensive tasks, such as image segmentation, voxelization, spatial alignment, interactive potential energy calculation, and thermal correction. The processor executes program instructions stored in memory to complete each step of the method.
[0124] like Figure 2 As shown, in one example, the evaluation system includes: an image acquisition module for acquiring pre-meal and post-meal images containing food; a raster generation module for generating a first voxel raster based on the pre-meal image and a second voxel raster based on the post-meal image, wherein each first voxel in the first voxel raster corresponds to a food region in the pre-meal image, and each first voxel is assigned a food category identifier and an initial potential value; and a spatial alignment module for spatially aligning the second voxel raster with the first voxel raster to obtain an aligned target volume. A voxel grid; a consumption identification module, used to identify the set of consumption voxels in the first voxel grid that is not matched in the target voxel grid; an intake calculation module, used to obtain an interaction potential energy correction value characterizing the interaction between foods based on the food category identifier and initial potential energy value of each consumption voxel in the set of consumption voxels, combined with the spatial proximity relationship between the consumption voxel and other consumption voxels; determine the effective consumption value of each consumption voxel according to the initial potential energy value and the interaction potential energy correction value; and accumulate the effective consumption values of all consumption voxels to obtain the total food intake.
[0125] In summary, this invention addresses the technical pain point of inaccurate evaluation in existing technologies by constructing a voxel grid data structure and proposing a spatial interaction potential energy model to quantify the interaction between foods, supplemented by a heat calibration correction based on visual features.
[0126] It should be noted that the above embodiments are merely preferred embodiments of the present invention. Those skilled in the art can make various combinations, modifications, and equivalent substitutions based on the ideas presented in this invention, and all such variations should fall within the protection scope of this invention. For example, the shape of the voxel grid is not limited to a cube, but can also be other geometric shapes that can be closely packed. The distance decay function in the interaction potential model can also take other forms, such as a Gaussian function or a piecewise function. The coefficients of the food interaction matrix can be automatically learned from the data using machine learning methods. These variations do not depart from the core technical concept of this invention.
[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0128] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0129] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Depending on actual needs, some or all of the units can be selected to achieve the purpose of this embodiment.
[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware.
[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A clinical meal intake assessment method based on image recognition, characterized in that, include: Obtain pre-meal and post-meal images containing food; Based on the pre-meal image, a first voxel grid is generated, where each first voxel in the first voxel grid corresponds to a food region in the pre-meal image, and each first voxel is assigned a food category identifier and an initial potential value. Based on the post-meal image, a second voxel grid is generated, and each second voxel in the second voxel grid corresponds to a food region in the post-meal image. The second voxel grid is spatially aligned with the first voxel grid to obtain the aligned target voxel grid. Identify the set of consumed voxels in the first voxel grid that are not matched in the target voxel grid; Based on the food category identifier and initial potential energy value of each consuming voxel in the set of consuming voxels, and combined with the spatial proximity relationship between the consuming voxel and other consuming voxels, the interaction potential energy correction value characterizing the interaction between foods is obtained. The effective consumption value of each voxel is determined based on the initial potential energy value of each voxel in the set of voxels and the corresponding interaction potential energy correction value. The total food intake is obtained by summing the effective consumption values of all the aforementioned voxels.
2. The method according to claim 1, characterized in that, The step of generating the first voxel grid based on the pre-meal image includes: The pre-meal image is segmented to obtain contour and texture information of multiple food regions; A three-dimensional coordinate system is set for the pre-meal image, and a first voxel grid covering the entire dining area is constructed; The contour and texture information of the multiple food regions are mapped to the first voxel grid, and the first voxel overlapping with the food region is activated; Assign the food category identifier and the initial potential energy value to each activated first voxel.
3. The method according to claim 2, characterized in that, The initial potential energy value is a combination of the volume of the food region corresponding to the first voxel and the standard nutrient density value.
4. The method according to claim 1, characterized in that, The step of spatially aligning the second voxel grid with the first voxel grid includes: Extract the edges or specific markers of the tableware as stable feature points from the first voxel grid and the second voxel grid; Construct an affine transformation matrix from the coordinate system of the second voxel grid to the coordinate system of the first voxel grid; The target voxel grid is generated by applying the affine transformation matrix to the second voxel grid.
5. The method according to claim 1, characterized in that, The step of obtaining the interaction potential energy correction value characterizing the interaction between food items includes: For any target consumption voxel in the set of consumption voxels, obtain other neighboring consumption voxels in its neighborhood; Query the preset food interaction matrix to obtain the interaction coefficient between the food category of the target consuming voxel and the food category of each neighboring consuming voxel; Based on the spatial distance between the target consumption voxel and each of the neighboring consumption voxels, calculate the distance attenuation factor; The interaction coefficient corresponding to each of the neighboring consumed voxels is combined with the distance decay factor to obtain multiple local interaction components; By aggregating the multiple local interaction components, the interaction potential energy correction value of the target consumed voxel is obtained.
6. The method according to claim 1 or 5, characterized in that, The step of determining the effective consumption value of each consumed voxel is based on a combined calculation of the initial potential energy value and the interaction potential energy correction value to obtain the effective consumption value.
7. The method according to claim 1, characterized in that, The method further includes: After determining the total food intake, the step of performing a calorie calibration correction on the total food intake; The steps for heat calibration correction include: For voxels of different food categories in the consumed voxel set, extract the image texture features of the corresponding food regions in the pre-meal image; Based on the image texture features, a preset texture-heat density mapping relationship is queried to obtain the heat correction factor for the corresponding food category; The calorie correction factor is applied to the total meal intake to obtain the corrected final meal intake.
8. A clinical meal intake assessment system based on image recognition, characterized in that, include: The image acquisition module is used to acquire pre-meal and post-meal images containing food. A raster generation module is used to generate a first voxel raster based on the pre-meal image and a second voxel raster based on the post-meal image. Each first voxel in the first voxel raster corresponds to a food region in the pre-meal image, and each first voxel is assigned a food category identifier and an initial potential value. A spatial alignment module is used to spatially align the second voxel grid with the first voxel grid to obtain an aligned target voxel grid. Consumption identification module, used to identify the set of consumption voxels in the first voxel grid that are not matched in the target voxel grid; The intake calculation module is used to obtain an interaction potential energy correction value that characterizes the interaction between foods based on the food category identifier and initial potential energy value of each consuming voxel in the set of consuming voxels, combined with the spatial proximity relationship between the consuming voxel and other consuming voxels; determine the effective consumption value of each consuming voxel according to the initial potential energy value and the interaction potential energy correction value; and accumulate the effective consumption values of all consuming voxels to obtain the total food intake.
9. A clinical meal intake assessment device based on image recognition, characterized in that, Includes a processor, a memory, and a communication bus for communicating with the processor and the memory; The memory stores a computer program, and the processor is configured to execute the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.