A dietary quantity quantification analysis system based on multi-modal large model fine-tuning and a use method thereof

Abstract

The present application provides a kind of dietary quantity analysis system based on multimodal large model fine-tuning and use method, including dietary semantic decoupling identification module, for obtaining initial image containing target meal;Multi-source adaptive quantization reconstruction module, reference feature is extracted based on the initial image to calculate scale factor;Evidence-based decision feedback module, the total amount sequence of purine intake of user continuous multiple days is time series correlation modeling with synchronous physiological data, the index deviation prediction value in future preset time period is calculated, and when the index deviation prediction value is greater than preset threshold, data is retrieved in clinical knowledge base using search enhancement generation architecture, individualized health diet management reference suggestion is generated and output, the reference suggestion is only used to provide dietary information reference, not as direct clinical medical diagnosis conclusion.The present application can improve the decoupling identification fineness of dish variety, raw material composition and cooking process of system in complex background.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a dietary quantitative analysis system and its usage method based on multimodal large model fine-tuning. Background Technology

[0002] With the improvement of socioeconomic levels, the demand for precise personal health and dietary management is increasing. For those who require long-term, refined dietary intake management, dynamic and objective data monitoring is the core management method. Currently, industry solutions for dietary recording and analysis typically use pre-trained general convolutional neural networks (CNNs) for basic image classification and food recognition.

[0003] Existing conventional techniques typically involve: capturing two-dimensional dietary images with a mobile device and uploading them to the cloud; using object detection frameworks such as YOLO or Faster R-CNN in the cloud to locate bounding boxes and output a single semantic label; and then performing mechanical key-value pair matching in a static food composition table to retrieve calorie or macronutrient data.

[0004] However, when faced with data processing scenarios requiring extremely high quantitative accuracy, existing technologies have revealed the following serious limitations:

[0005] First, the recognition precision is severely insufficient, and there is a fundamental lack of semantic decoupling capability for cooking processes. Existing visual models have difficulty decoupling food categories from surface cooking representations. For example, under different physical cooking modes such as "stewed", "fried" and "thick soup", the extraction rate of soluble substances varies greatly. The flat classification mapping logic of existing technologies leads to unacceptable huge biases introduced at the input end in subsequent calculations.

[0006] Second, the deep reliance on static constant mapping leads to huge nonlinear quantitative errors in dynamic estimation. Conventional terminals lack effective geometric reconstruction methods for the three-dimensional physical volume of food in two-dimensional images. They can only perform fuzzy calculations based on the average absolute content per 100g, ignoring the nonlinear relationship between the actual physical intake volume and food density, making accurate quantification impossible.

[0007] Third, the data processing logic is fragmented, lacking high-dimensional time-series analysis and prediction capabilities. Existing solutions mostly consist of discrete, stream-of-consciousness data records, lacking the ability to perform time-series correlation modeling of long-term dietary intake data with dynamically fluctuating physiological data. The system cannot perform nonlinear predictions based on data evolution, and the generated suggestions are mostly template-based and have low reference value.

[0008] Fourth, there is an imbalance between limited local computing power and the need for personal privacy protection. Traditional applications typically send un-anonymized images to the cloud for complex inference, resulting in extremely high latency when the network is limited. At the same time, photography often inevitably captures highly sensitive privacy data such as environmental and facial images. In the context of lightweight mobile terminals, existing systems lack efficient local pre-cleaning and data anonymization mechanisms, posing a risk of privacy leaks. Summary of the Invention

[0009] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a dietary quantitative analysis system and method based on multimodal large model fine-tuning, which is used to solve the problem of excessive quantitative error caused by the lack of image recognition dimension in the prior art, the problem of weak time series data correlation modeling ability, and the problems of computing power bottleneck and privacy imbalance when deployed on lightweight mobile terminals.

[0010] To achieve the above and other related objectives, this invention provides a dietary quantification and analysis system based on multimodal large model fine-tuning, comprising:

[0011] The dietary semantic decoupling and recognition module is used to acquire an initial image containing the target meal, and to extract global features and local high-frequency texture features from the initial image using a multimodal large language model with efficient parameter fine-tuning. It performs semantic decoupling on the type of dish, ingredient composition and cooking process, and outputs the probability distribution vector of cooking method and the corresponding purine kinetic migration coefficient.

[0012] The multi-source adaptive quantization reconstruction module is communicatively connected to the dietary semantic decoupling recognition module. Based on the initial image, it extracts reference object features to calculate the scaling factor, uses a monocular depth estimation network to predict pixel depth and generate a three-dimensional physical height field, combines semantic prior constraints to perform adaptive thickness compensation on the physical height field, calculates the three-dimensional physical volume of food through discrete spatial integration, and maps the actual total amount of purines ingested in a single meal.

[0013] The evidence-based decision feedback module is communicatively connected to the multi-source adaptive quantization reconstruction module. It performs time-series correlation modeling on the user's total purine intake sequence over multiple consecutive days and synchronized physiological data, calculates the deviation of the indicator from the predicted value within a preset time period, and when the deviation of the indicator from the predicted value is greater than a preset threshold, it uses a retrieval-enhanced generation architecture to retrieve data from the clinical knowledge base, generate and output personalized health diet management reference suggestions. The reference suggestions are only used to provide dietary information reference and should not be used as direct clinical medical diagnosis conclusions.

[0014] The edge-cloud collaborative singleton resource pooling scheduling module is connected to the above modules. It is configured to build a singleton resource management pool in the local computing environment to perform unified asynchronous orchestration of memory resources. After performing feature desensitization locally, it transmits the structured data to the cloud computing power cluster to perform deep inference tasks.

[0015] By adopting the above technical solution, and through the joint deployment of the dietary semantic decoupling recognition module and the multi-source adaptive quantification reconstruction module, the divergent quantitative errors caused by the flat classification and two-dimensional area estimation of traditional visual models are overcome, and three-dimensional physical-level accurate calculation of objective substance intake is achieved. Through the combination of the edge-cloud collaborative single-instance resource pooling scheduling module and the evidence-based decision feedback module, not only is the isolation of daily image privacy data guaranteed at the terminal physical level, but also a nonlinear data prediction and auxiliary management closed-loop system with high evidence-based reference value is established using the time series modeling engine.

[0016] In one embodiment of the present invention, the dietary semantic decoupling recognition module adopts an instruction fine-tuning architecture. Without changing the pre-training weights of the multimodal large language model, it trains proprietary neuron parameters specifically for the physical representation of cooking by injecting a low-rank decomposition matrix into the self-attention layer. The module includes a dual-tower feature alignment unit and a thermodynamic representation analysis unit. It achieves multi-dimensional feature alignment by calculating the cosine similarity loss function, and decouples and outputs the probability distribution vector of the cooking method by detecting the color gradient and edge charring degree of the high-frequency texture on the surface of the food.

[0017] In one embodiment of the present invention, the multi-source adaptive quantization reconstruction module calculates the scaling factor. At the same time, it supports intelligent switching between standard object calibration mode and adaptive prior calibration mode; when using adaptive prior calibration mode, it extracts the semantic recognition results of inherent objects in the scene and matches their physical constants, combines the focal length metadata of the mobile terminal camera, and obtains the scale factor by back-calculation through perspective projection model. .

[0018] In one embodiment of the present invention, the multi-source adaptive quantization reconstruction module generates the physical height field using a monocular depth estimation network of a cloud-based inference engine. Furthermore, nonlinear corrections are made based on semantic prior constraints: if the food is identified as thin slices, the bottom cutoff threshold of the height field is adaptively lowered; if the food is identified as stacked, a preset volume compensation coefficient is enabled. .

[0019] In one embodiment of the present invention, the volume of the discrete space integration algorithm The calculation formula is: ,in, Area per unit pixel;

[0020] The mapping formula for the total objective amount of purines P is: ,in, The physical density of this ingredient. Based on basal purine concentration, is the purine kinetic migration coefficient.

[0021] In one embodiment of the present invention, the evidence-based decision feedback module utilizes the self-attention mechanism of the Transformer model architecture to temporally concatenate continuous dietary sequences and physiological data, fit a nonlinear mapping relationship, determine the risk status through a task orchestration agent, and drive the retrieval enhancement generation architecture to extract literature paragraphs from the FAISS vector knowledge base, and inject the results into the large language model through prompt word engineering.

[0022] In one embodiment of the present invention, the edge-cloud collaborative single-instance resource pooling scheduling module is configured to perform a de-identification operation on the mobile terminal, including the cropping of the region of interest in the image, and after removing the environmental background and facial privacy data, only the de-identified feature vector and environmental parameters are asynchronously transmitted to the cloud.

[0023] A method for using a dietary quantification system based on multimodal large model fine-tuning, applied to the system described above, includes the following steps:

[0024] S1. Obtain the region of interest of the dietary image through the edge-cloud collaborative single-instance resource pooling scheduling module, perform physical truncation and privacy data desensitization, and send the desensitized features back to the cloud;

[0025] S2. In the cloud, high-frequency texture parameters of features are extracted through a dietary semantic decoupling recognition module. A multimodal large language model with an injected low-rank decomposition matrix is ​​used for dual-tower feature alignment, decoupling and outputting the food type and its corresponding purine kinetic migration coefficient. ;

[0026] S3, the multi-source adaptive quantization reconstruction module calculates the physical scaling factor using a priori models. The height field is generated by combining a depth estimation network and a compensation coefficient is introduced under semantic constraints. Physical volume is calculated through discrete spatial integration. The objective total purine content of a single meal was obtained. ;

[0027] S4. Input the objective total amount of purines into the evidence-based decision feedback module, use the self-attention mechanism to construct the time-series mapping feature between the long-term dietary sequence and physiological indicators, and output the deviation of the indicators from the predicted value for a future preset time period.

[0028] S5. When the predicted value meets the conditions, drive the retrieval enhancement generation architecture to match external knowledge base data, use a multimodal large language model to generate and distribute reference results containing objective data feedback and dietary adjustment strategies.

[0029] By adopting the above technical solutions, and through the composite infinitesimal integral of semantic decoupling probability vectors with deep model fine-tuning in the cloud and spatial thickness compensation coefficients generated by deep networks, a data processing leap from coarse qualitative image recognition to precise three-dimensional quantitative analysis of matter was achieved. Finally, by combining temporal mapping mechanisms and RAG map retrieval mechanisms, discrete and disordered eating records were upgraded to continuous, personalized dietary intervention strategy information supported by authoritative literature sources, significantly improving the objectivity and accuracy of the computer system's output results and their practical application reference value.

[0030] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the dietary quantitative analysis and auxiliary diagnosis and treatment method based on multimodal large model fine-tuning as described in claim 8.

[0031] As described above, the dietary quantitative analysis system and its usage method based on multimodal large model fine-tuning of the present invention have the following beneficial effects:

[0032] 1. By using PEFT instruction fine-tuning combined with thermodynamic characterization algorithm, the semantic confusion problem of the model in cooking process is solved, and the robustness of the system in recognition under complex lighting background is significantly improved.

[0033] 2. By fusing multi-source adaptive geometric scale calibration with monocular depth estimation, the pixel array is transformed into a continuous physical height field. Rigorous discrete spatial integration is used to make the input calculation results approximate the real physical mass.

[0034] 3. By leveraging a temporal feature attention mechanism and a RAG retrieval enhancement architecture, the "illusion" weakness easily generated by large model generation is overcome. A non-linear link between multi-dimensional data is established, and an authoritative knowledge base is retrieved through a scheduling agent to generate highly targeted data refining route guidance.

[0035] 4. By using a singleton resource pooling mechanism to make concurrency management asynchronous, program crashes caused by overload are eliminated. By using a local truncation and desensitization operator, the flow of sensitive privacy data to the cloud is blocked at the physical level, ensuring that the system achieves high security isolation while maintaining low power consumption and fast response. Attached Figure Description

[0036] Figure 1 The diagram shown is a schematic representation of the overall process disclosed in an embodiment of the present invention. Detailed Implementation

[0037] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0038] Example 1:

[0039] This invention provides a dietary quantification and analysis system based on multimodal large model fine-tuning, comprising:

[0040] The edge-cloud collaborative singleton resource pooling scheduling module builds a singleton resource management pool in the local computing environment to perform unified asynchronous orchestration of memory resources. After performing feature desensitization locally, it transmits structured data to the cloud computing power cluster to perform deep inference tasks.

[0041] In lightweight windows (such as WeChat Mini Program environments), this module uses the Singleton pattern to construct a globally unique resource management pool object. Upon image acquisition, the Singleton pool immediately intercepts heavy computations on the main thread, waking up the local computing power matrix (such as NPU-accelerated micro-operators) to perform preprocessing.

[0042] The core of the system lies in local-level privacy filtering: within milliseconds, the system uses a rectangular bounding box to quickly capture the region of interest (ROI) containing food and size reference objects (such as the edge of a plate). The system determines all pixels outside the ROI boundary—including privacy noise such as faces and environmental backgrounds—and performs destructive data masking and truncation in local memory to prevent them from entering the persistent stack. The desensitized and condensed feature tensor is asynchronously pushed to the cloud public GPU cluster to offload the heavy-load model inference.

[0043] The dietary semantic decoupling and recognition module is used to acquire an initial image containing the target meal, and to extract global features and local high-frequency texture features from the initial image using a multimodal large language model with efficient parameter fine-tuning. It performs semantic decoupling on the type of dish, ingredient composition and cooking process, and outputs the probability distribution vector of cooking method and the corresponding purine kinetic migration coefficient.

[0044] After the cloud engine receives the desensitized tensor, it triggers semantic decoupling. This module integrates the low-rank adaptive (LoRA) algorithm in the parameter efficient fine-tuning (PEFT) technique. The system freezes the common weights of the basic model and injects a trainable low-rank matrix into the self-attention layers via parallel bypass.

[0045] The system deploys a dual-tower feature alignment algorithm, which uses cosine similarity to align the "color-texture" characteristics of the image to the semantic text space of the process name. The built-in thermodynamic characterization analysis logic specializes in extracting high-frequency textures: the charred area ratio of the food surface is captured by the edge operator (to evaluate the frying depth), and the oil reflection flare matrix of the liquid surface is extracted to determine the concentration of the soup.

[0046] The model decouples and outputs a multidimensional probability distribution vector of cooking methods. It also performs nonlinear interpolation in the built-in database to match the core dynamic migration loss coefficient of the corresponding material transfer process. ;

[0047] The multi-source adaptive quantization reconstruction module is communicatively connected to the dietary semantic decoupling recognition module. Based on the initial image, it extracts reference object features to calculate the scaling factor, uses a monocular depth estimation network to predict pixel depth and generate a three-dimensional physical height field, and performs adaptive thickness compensation on the physical height field using semantic prior constraints. It then calculates the three-dimensional physical volume of the food through discrete spatial integration and maps it to the actual total purine intake of a single meal.

[0048] To map two-dimensional vision to three-dimensional physical properties, this module constructs a multi-source adaptive calibration (ScaleCalibration) engine.

[0049] Multi-source adaptive scale calibration first preprocesses the image via a mobile terminal (such as a WeChat mini-program), extracts reference object features, and calculates the scaling factor between pixel distance and physical distance. .

[0050] The calibration logic has two modes:

[0051] Standard object calibration mode: The system identifies manually preset reference objects in the image (such as a standard 20cm diameter plate or a standard coin) and calculates the scale factor. .

[0052] Adaptive Prior Calibration Mode: If artificial reference objects are missing in the image, the system initiates prior-based size estimation logic. It uses semantic recognition results to match the physical constants of inherent objects in the scene (e.g., using chopsticks with a standard length of 23cm, or rice grains with an average vertical axis length of 7mm), and combines this with the focal length metadata of the mobile terminal's camera to inversely deduce the scale factor using a perspective projection model. .

[0053] Get Then, a monocular depth estimation network with a self-attention mechanism is loaded in the cloud to decode the feature tensor, predict the relative depth of each pixel, and combine it with... Render the physical height field matrix that describes the surface undulations. Furthermore, semantic prior nonlinear compensation is introduced: if identified as "thin sheet-like", the depth truncation threshold is forcibly lowered to eliminate shadow overhangs; if identified as "stacked", volume compensation correction coefficients are injected into the integral equation according to granularity. .

[0054] The algorithm performs integral calculations on all pixel units within the semantic segmentation mask region generated in the cloud, and the volume is calculated. The calculation formula is expressed as:

[0055]

[0056] in, This represents the area per unit pixel.

[0057] Ultimately, the actual total purine intake from a single meal The mapping formula is:

[0058]

[0059] in, This refers to the average physical density of this type of food. Based on basal purine concentration, is the purine kinetic migration coefficient.

[0060] The evidence-based decision feedback module is communicatively connected to the multi-source adaptive quantization reconstruction module. It performs time-series correlation modeling on the user's total purine intake sequence over multiple consecutive days and synchronized physiological data, calculates the deviation of the indicator from the predicted value within a preset time period, and when the deviation of the indicator from the predicted value is greater than a preset threshold, it uses a retrieval-enhanced generation architecture to retrieve data from the clinical knowledge base, generate and output personalized health diet management reference suggestions. The reference suggestions are only used to provide dietary information reference and should not be used as direct clinical medical diagnosis conclusions.

[0061] In this embodiment, the system achieves the logical leap from data to medical opinions through a multi-agent collaborative architecture:

[0062] Temporal correlation modeling: Utilizing the self-attention mechanism in the Transformer model, it sequentially associates user data. Daily Dietary Intake Sequence Simultaneously entered blood uric acid levels By splicing the data, the model learns the nonlinear mapping relationship between fluctuations in intake and the lag of physiological indicators.

[0063] Task Orchestrator: Automatically assesses the current risk status. If the predicted risk of the user's uric acid level exceeding the threshold in the next 24 hours is greater than the threshold, then a search command is activated.

[0064] RAG Evidence Retrieval: The system uses "current cuisine + user's disease stage" as the retrieval vector to search authoritative clinical knowledge bases in FAISS (Facebook AI Similarity Search, an efficient vector retrieval library) to obtain targeted medical evidence.

[0065] Prompt Engineering: The identification results, quantitative data, and retrieved evidence-based medicine evidence are injected into the fine-tuned LLM to generate a three-part expert opinion that includes "risk warning + pitfall avoidance guide + metabolic suggestions".

[0066] Example 2:

[0067] This embodiment provides a method for using the dietary quantitative analysis system based on multimodal large model fine-tuning as described in Embodiment 1, including the following steps:

[0068] S1. Terminal local environment preprocessing and asynchronous desensitization: The region of interest of the dietary image is obtained through the terminal-cloud collaborative single-instance resource pooling scheduling module, and physical truncation and privacy data desensitization are performed. The desensitized features are then sent back to the cloud.

[0069] The scheduling module of the mobile terminal intercepts data at the operating system level, runs edge truncation operators within a very small amount of RAM, peels off the ROI matrix containing core reference objects from the original image, uses masking technology to destroy the bitmap containing the surrounding environment such as faces, and pushes the low-dimensional structured desensitized features and metadata to the cloud service center for asynchronous processing via an encrypted network.

[0070] S2. Semantic Deep Deconstruction and Dynamic Coefficient Matching: High-frequency texture parameters of features are extracted in the cloud through a dietary semantic decoupling recognition module. A multimodal large language model with an injected low-rank decomposition matrix is ​​used for dual-tower feature alignment, decoupling and outputting the food type and its corresponding purine kinetic transfer coefficient. ;

[0071] The cloud-based large model engine takes over the data stream, using a dual-tower model and a specialized high-frequency texture perception neural network to extract thermodynamic and physical state features. The algorithm deconstructs a clear discrete probability vector of cooking process and matches it with the corresponding dynamic heat migration loss coefficient. .

[0072] S3. Three-dimensional topological space reconstruction and quantitative normalization: The multi-source adaptive quantization reconstruction module calculates the physical scaling factor using a priori models. The height field is generated by combining a depth estimation network and a compensation coefficient is introduced under semantic constraints. Physical volume is calculated through discrete spatial integration. The objective total purine content of a single meal was obtained. ;

[0073] The computational geometry algorithm initiates a monocular depth network to estimate the Z-axis depth distribution, and calculates the physical scaling constant through perspective projection adaptive scaling. Perform point cloud mapping and inject porosity / thickness compensation factors using semantic prior constraints. The discrete infinitesimal space volume integral engine is executed to accurately calculate the true volume basis. Substituting into the nonlinear mapping formula, the total objective mass increment of the current input sample is rigorously calculated. .

[0074] S4. Spatiotemporal feature sequence mapping and trend deviation prediction: The total objective amount of purines is input into the evidence-based decision feedback module, and the temporal mapping features between the long-term dietary sequence and physiological indicators are constructed using the self-attention mechanism. The predicted value of the indicator deviation for the future preset time period is output.

[0075] The calculated incremental values ​​are concatenated into the system's long-term historical sliding time window tensor array and aligned with objective biochemical network indicators for multidimensional feature analysis. The Transformer model performs forward propagation prediction and outputs a deviation warning factor for indicator imbalance within a preset time period. .

[0076] S5. Information retrieval and strategy delivery based on RAG architecture: When the predicted value meets the conditions, the retrieval enhancement generation architecture is driven to match external knowledge base data, and reference results containing objective data feedback and dietary adjustment strategies are generated and delivered using a multimodal large language model.

[0077] When deviating from the warning factor When the safety threshold is exceeded, the system task agent triggers the underlying RAG pipeline. It uses the current high-dimensional feature vectors to anchor authoritative dehydrated literature in the vector knowledge graph, and, in conjunction with the fine-tuned language model, translates and assembles a logically sound adjustment strategy report with traceable sources under strict constraints of prompt words. The report is then presented to the client interface for user reference through a non-blocking animation on the front end.

[0078] Example 3:

[0079] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the dietary quantitative analysis and auxiliary diagnosis and treatment method based on multimodal large model fine-tuning as described in Embodiment 2.

[0080] In summary, this invention overcomes the shortcomings of existing technologies, such as missing identification dimensions, large quantization errors, and weak clinical relevance, through vertical domain large model fine-tuning, three-dimensional geometric quantization reconstruction, and evidence-driven temporal correlation algorithms. Specific technical effects are as follows:

[0081] 1. Achieved refined dietary analysis based on deep semantic understanding.

[0082] Compared to general dataset solutions like Food-101, this invention employs Supervised Fine-Tuning (SFT) to enable the model to perceive the complex culinary systems and physical properties of Chinese cuisine. Through thermodynamic characterization analysis, the system accurately identifies cooking techniques such as "stewing," "stir-frying," and "thick soup," thereby precisely matching the corresponding purine kinetic transfer coefficients. Its recognition precision is significantly improved compared to traditional classification methods. Furthermore, this invention addresses the robustness issue of general large-scale models when facing complex backgrounds and overlapping ingredients, providing highly reliable feature inputs for subsequent medical-grade data processing.

[0083] 2. A technological leap from two-dimensional visual perception to three-dimensional physical quantization has been achieved.

[0084] This invention physically reconstructs the quantitative dimension, overcoming the bottleneck of existing dietary management applications that can only perform static tag retrieval and cannot quantify total intake. Through geometric scaling calibration and a monocular depth estimation integral algorithm, it achieves a precise mapping from pixel area to physical volume, enabling dietary monitoring to move from "qualitative identification" to "quantitative analysis." Dynamic correction coefficients improve calculation accuracy, and prior knowledge of cuisines (such as the thickness compensation factor for rice noodle rolls and the migration ratio factor for broth) is used to nonlinearly correct the calculation results, making the calculated purine intake more consistent with clinical metabolic realities.

[0085] 3. A closed-loop diagnosis and treatment system based on evidence-based medicine, encompassing "sensing-tracking-intervention," has been established.

[0086] By combining authoritative clinical medical knowledge bases with large-scale model generation capabilities through the RAG (Retrieval-Augmented Generation) architecture, the system enables the precise delivery of expert opinions. It can generate expert-level intervention suggestions for specific cuisines (such as hydration strategies for hot pot with a rich broth base). By temporally correlating dietary intake with physiological indicators such as blood uric acid, the system significantly improves the self-management adherence and treatment continuity of patients with chronic diseases in daily life.

[0087] 4. This invention addresses the operational bottleneck of highly complex multi-agent cooperation logic in computationally limited environments such as mobile terminals (e.g., lightweight front-end windows like WeChat mini-programs) through a Singleton Resource Pooling management mechanism. It demonstrates efficient resource scheduling: the singleton pool handles lightweight image preprocessing and Region of Interest (ROI) extraction locally, ensuring smooth front-end interaction through asynchronous task orchestration and preventing mini-program crashes or delays caused by heavy computational tasks. Multi-level privacy protection: Preliminary cleaning and data masking of sensitive medical privacy data are completed locally. The system only uploads the masked dietary feature vectors and environmental parameters to the public cloud, effectively blocking leakage paths containing privacy information such as facial features and home environment, meeting the dual requirements of data security and high real-time performance in long-term chronic disease tracking scenarios. Response performance optimization: Through the Edge-Cloud Collaboration architecture, the high-energy-consuming deep inference is offloaded to the cloud, while the local singleton pool caching mechanism is used to reduce duplicate requests, realizing low-power and fast-response deployment of medical-grade dietary analysis on mobile devices.

[0088] Therefore, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0089] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A dietary quantitative analysis system based on multimodal large model fine-tuning, characterized in that, include: The dietary semantic decoupling and recognition module is used to acquire an initial image containing the target meal, and to extract global features and local high-frequency texture features from the initial image using a multimodal large language model with efficient parameter fine-tuning. It performs semantic decoupling on the type of dish, ingredient composition and cooking process, and outputs the probability distribution vector of cooking method and the corresponding purine kinetic migration coefficient. The multi-source adaptive quantization reconstruction module is communicatively connected to the dietary semantic decoupling recognition module. Based on the initial image, it extracts reference object features to calculate the scaling factor, uses a monocular depth estimation network to predict pixel depth and generate a three-dimensional physical height field, combines semantic prior constraints to perform adaptive thickness compensation on the physical height field, calculates the three-dimensional physical volume of food through discrete spatial integration, and maps the actual total amount of purines ingested in a single meal. The evidence-based decision feedback module is communicatively connected to the multi-source adaptive quantization reconstruction module. It performs time-series correlation modeling on the user's total purine intake sequence over multiple consecutive days and synchronized physiological data, calculates the deviation of the indicator from the predicted value within a preset time period, and when the deviation of the indicator from the predicted value is greater than a preset threshold, it uses a retrieval-enhanced generation architecture to retrieve data from the clinical knowledge base, generate and output personalized health diet management reference suggestions. The reference suggestions are only used to provide dietary information reference and should not be used as direct clinical medical diagnosis conclusions. The edge-cloud collaborative singleton resource pooling scheduling module is connected to the above modules. It builds a singleton resource management pool in the local computing environment to perform unified asynchronous orchestration of memory resources. After performing feature desensitization locally, it transmits the structured data to the cloud computing power cluster to perform deep inference tasks.

2. The dietary quantitative analysis system based on multimodal large model fine-tuning according to claim 1, characterized in that: The dietary semantic decoupling recognition module adopts an instruction fine-tuning architecture. Without changing the pre-training weights of the multimodal large language model, it trains proprietary neuron parameters specifically for the physical representation of cooking by injecting a low-rank decomposition matrix into the self-attention layer. This module includes a dual-tower feature alignment unit and a thermodynamic representation analysis unit. It achieves multi-dimensional feature alignment by calculating the cosine similarity loss function and decouples and outputs the probability distribution vector of the cooking method by detecting the color gradient and edge charring degree of high-frequency textures on the surface of the ingredients.

3. The dietary quantitative analysis system based on multimodal large model fine-tuning according to claim 1, characterized in that: The multi-source adaptive quantization reconstruction module calculates the scaling factor. At the same time, it supports intelligent switching between standard object calibration mode and adaptive prior calibration mode; when using adaptive prior calibration mode, it extracts the semantic recognition results of inherent objects in the scene and matches their physical constants, combines the focal length metadata of the mobile terminal camera, and obtains the scale factor by back-calculation through perspective projection model. .

4. The dietary quantitative analysis system based on multimodal large model fine-tuning according to claim 1, characterized in that: The multi-source adaptive quantization reconstruction module generates the physical height field using the monocular depth estimation network of the cloud inference engine. Furthermore, nonlinear corrections are made based on semantic prior constraints: if the food is identified as thin slices, the bottom cutoff threshold of the height field is adaptively lowered; if the food is identified as stacked, a preset volume compensation coefficient is enabled. .

5. The dietary quantitative analysis system based on multimodal large model fine-tuning according to claim 1, characterized in that: Volume of the discrete space integration algorithm The calculation formula is: ,in, Area per unit pixel; The mapping formula for the total objective amount of purines P is: ,in, The physical density of this ingredient. Based on basal purine concentration, is the purine kinetic migration coefficient.

6. The dietary quantitative analysis system based on multimodal large model fine-tuning according to claim 1, characterized in that: The evidence-based decision feedback module utilizes the self-attention mechanism of the Transformer model architecture to temporally concatenate continuous dietary sequences with physiological data, fit a nonlinear mapping relationship, determine the risk status through a task orchestration agent, and drive the retrieval enhancement generation architecture to extract literature paragraphs from the FAISS vector knowledge base. The results are generated by injecting prompt word engineering into the large language model.

7. The dietary quantitative analysis system based on multimodal large model fine-tuning according to claim 1, characterized in that: The edge-cloud collaborative single-instance resource pooling scheduling module is configured to perform a de-identification operation on the mobile terminal, which includes cropping the region of interest of the image. After removing the environmental background and facial privacy data, only the de-identified feature vector and environmental parameters are asynchronously transmitted to the cloud.

8. A method of using a dietary quantitative analysis system based on multimodal large model fine-tuning, applied to the system as described in any one of claims 1-7, characterized in that, Includes the following steps: S1. Obtain the region of interest of the dietary image through the edge-cloud collaborative single-instance resource pooling scheduling module, perform physical truncation and privacy data desensitization, and send the desensitized features back to the cloud; S2. In the cloud, high-frequency texture parameters of features are extracted through a dietary semantic decoupling recognition module. A multimodal large language model with an injected low-rank decomposition matrix is ​​used for dual-tower feature alignment, decoupling and outputting the food type and its corresponding purine kinetic migration coefficient. ; S3, the multi-source adaptive quantization reconstruction module calculates the physical scaling factor using a priori models. The height field is generated by combining a depth estimation network and a compensation coefficient is introduced under semantic constraints. ; Physical volume is calculated using discrete space integration. The objective total purine content of a single meal was obtained. ; S4. Input the objective total amount of purines into the evidence-based decision feedback module, use the self-attention mechanism to construct the time-series mapping feature between the long-term dietary sequence and physiological indicators, and output the deviation of the indicators from the predicted value for a future preset time period. S5. When the predicted value meets the conditions, drive the retrieval enhancement generation architecture to match external knowledge base data, use a multimodal large language model to generate and distribute reference results containing objective data feedback and dietary adjustment strategies.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method of using the dietary quantitative analysis system based on multimodal large model fine-tuning as described in claim 8.

Images