Food packaging information identification and comprehensive analysis method based on multi-modal large model

Abstract

The invention relates to the technical field of information identification and analysis, and provides a multi-modal large model-based food packaging information identification and comprehensive analysis method, which comprises the following steps of: 1, acquiring basic information of a user and a commodity packaging image uploaded by the user; 2, character recognition is carried out on the content of the commodity packaging image, a core information field and an auxiliary information field of a product are extracted, and structured information is obtained; step 3, performing multi-path analysis; 4, inputting the structured information and the analysis result into a locally deployed large language model, performing semantic fusion and knowledge modeling, and constructing a user-oriented knowledge graph; 5, performing multi-modal information fusion on the core information and the auxiliary information by adopting an embedded model to obtain multi-modal semantic information; and step 6, generating personalized consumption suggestions. According to the method, the product packaging information can be extracted and analyzed by combining the OCR technology with the multi-modal model, and the recognition and acquisition speed of the product information is increased.

Description

Technical Field

[0001] This invention relates to the field of information recognition and analysis technology, and in particular to a method for food packaging information recognition and comprehensive analysis based on a multimodal large model. Background Technology

[0002] With the rise of e-commerce and consumption upgrades, consumers are increasingly relying on information on product packaging when making purchasing decisions. However, product packaging information is often just a superficial description, and consumers need more comprehensive information, such as the scientific names of ingredients, negative news about the product, market prices, and purchase channels.

[0003] Traditional food packaging information can lead to consumers having an incomplete understanding of the information; the way product information is obtained relies on manual search, the sources of information are scattered, the search efficiency is low, and it is easily influenced by biases that may mislead consumers. Summary of the Invention

[0004] This invention primarily addresses the technical problems of traditional food packaging information being merely a superficial description, leading to incomplete understanding of packaging information by consumers; and the reliance on manual searches for product information, resulting in scattered information sources, low search efficiency, and susceptibility to bias that can mislead consumers. It proposes a multimodal large-scale model-based method for food packaging information identification and comprehensive analysis. By combining OCR technology with a multimodal model, product packaging information is extracted and analyzed, improving the speed of product information identification and acquisition, enabling comprehensive market data analysis, and optimizing consumer purchasing decisions.

[0005] This invention provides a method for identifying and comprehensively analyzing food packaging information based on a multimodal large model, comprising the following processes:

[0006] Step 1: Obtain basic user information and product packaging images uploaded by the user;

[0007] Step 2: Perform text recognition on the content of the product packaging image to extract the core information fields and auxiliary information fields of the product, and perform structured processing through entity recognition and field classification to obtain structured information;

[0008] Step 3: Perform multipath analysis on the structured information obtained in Step 2;

[0009] Step 4: Input the structured information obtained in Step 2 and the analysis results in Step 3 into the locally deployed large language model to perform semantic fusion and knowledge modeling, and build a user-oriented knowledge graph.

[0010] Step 5: Using an embedding model, the structured core information and auxiliary information obtained in Step 2 are fused in a multimodal manner to obtain multimodal semantic information;

[0011] Step 6: Generate personalized consumption suggestions based on the locally deployed large language model.

[0012] Furthermore, following step 6, the following steps are also included:

[0013] Step 7: Collect user feedback on consumption suggestions and store the consumption suggestions and feedback information in the database.

[0014] Furthermore, the user's basic information includes population type, health status, and dietary preferences;

[0015] Furthermore, step 2 includes the following steps 201 to 202:

[0016] Step 201: Based on the OCR engine and multimodal large model, perform text recognition on the product packaging image to extract the core information fields and auxiliary information fields of the product;

[0017] The multimodal large model adopts the qwen2.5vl-72b model and uses the pre-trained knowledge of the qwen2.5vl-72b model for recognition.

[0018] The core information includes product name, ingredients, product standard number, production license number, manufacturer name, place of origin, net content, and brand.

[0019] The auxiliary information includes ingredient icons, warning signs, and certification badges;

[0020] Step 202: Perform entity recognition and field classification on the core information and auxiliary information extracted in step 201 to obtain structured information.

[0021] Furthermore, step 3 includes the following steps 301 to 304:

[0022] Step 301: Identify the ingredients and components information in the core information and perform ingredient and component analysis;

[0023] Step 302: Identify the brand and manufacturer information in the core information and perform brand and manufacturer analysis;

[0024] Step 303: Identify public opinion information in the core information and conduct public opinion analysis;

[0025] Step 304: Identify the price information in the core information and perform price analysis.

[0026] Furthermore, the embedding model is the embedding-3 model.

[0027] Furthermore, the consumer advice includes whether to recommend purchasing, whether it is suitable for consumption, recommendations for applicable groups, potential health and ingredient risks, brand and manufacturer credibility, disputes and compliance risks, recommendations for consumption scenarios and dosages, and analysis of price reasonableness.

[0028] The present invention provides a method for food packaging information identification and comprehensive analysis based on a multimodal large model, which has the following advantages compared with the prior art:

[0029] Improving Information Acquisition Efficiency: This invention utilizes OCR technology combined with a multimodal large model to extract and analyze product packaging information, thereby increasing the speed of product information recognition and acquisition. It enables comprehensive market data analysis and optimizes consumer purchasing decisions. Through image recognition, OCR technology, natural language processing, knowledge graph construction, and multi-channel information fusion, it comprehensively collects, extracts, analyzes, and evaluates product information, ultimately outputting clear and accurate consumer recommendations.

[0030] Multi-dimensional and comprehensive market analysis: This invention goes beyond information on packaging, incorporating factors such as negative news about the product, manufacturer, parent company, and capital relationships, as well as investigating market prices, e-commerce prices, and other purchasing channels. The comprehensive analysis provides purchase recommendations, consumption suggestions, and suitable target groups to help users make more comprehensive purchasing decisions.

[0031] RAG support: This invention can collect and store user feedback on consumption suggestions, allowing users to provide evaluations and ratings. After data accumulation, it is RAG-ized to support subsequent surveys and analyses. With the accumulation of data, a robust data-supported survey and analysis platform can be gradually formed, supporting deeper market trend analysis. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the implementation of the food packaging information identification and comprehensive analysis method based on a multimodal large model provided by the present invention. Detailed Implementation

[0033] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of them.

[0034] like Figure 1 As shown in the figure, the present invention provides a method for food packaging information identification and comprehensive analysis based on a multimodal large model, which includes the following processes:

[0035] Step 1: Obtain basic user information and product packaging images uploaded by the user.

[0036] The user's basic information includes, but is not limited to, demographic type, health status, dietary preferences, etc. Demographic types include pregnant women, children, people trying to lose weight, etc.

[0037] Users can take photos of product packaging using their mobile phones to obtain images of the product packaging.

[0038] After acquiring the product packaging image uploaded by the user, the process further includes: preprocessing the original product packaging image; the preprocessing includes, but is not limited to, image enhancement, noise reduction, and resolution normalization. This preprocessing provides clear input for subsequent OCR recognition.

[0039] Step 2 involves performing text recognition on the product packaging image to extract the core and auxiliary information fields. This information is then structured using entity recognition and field classification to obtain structured information. Step 2 includes the following steps 201 to 202:

[0040] Step 201: Based on the OCR engine and multimodal large model, perform text recognition on the product packaging image to extract the core information fields and auxiliary information fields of the product.

[0041] This step employs an OCR (Optical Character Recognition) engine that supports complex layouts and a multimodal large model for multilingual text recognition. Combined with layout analysis and table parsing techniques, it ensures high-precision recognition of different packaging styles.

[0042] It uses pre-trained knowledge for recognition and supports characters in multiple languages ​​such as Chinese, English, Japanese, and Korean.

[0043] The layout analysis Mask R-CNN input channel includes RGB image channels, text density heatmaps (from OCR intermediate output), and edge enhancement feature maps to distinguish sections such as main label areas and sub-label areas.

[0044] The table analysis uses the YOLOv8 model to enhance the detection capabilities for dashed-line tables and borderless tables.

[0045] The core information includes, but is not limited to: product name, ingredients, product standard number, production license number, manufacturer name, place of origin, net content, brand, etc.

[0046] The auxiliary information includes, but is not limited to: ingredient icons, warning signs, and certification badges.

[0047] Step 202: Perform entity recognition and field classification on the core information and auxiliary information extracted in step 201 to obtain structured information.

[0048] Named entity recognition (NER) and semantic classification are performed on the core information extracted in step 201 to transform the unstructured OCR results into structured field data.

[0049] Field classification rules include tag semantic similarity matching, contextual association analysis, and knowledge base-based field mapping.

[0050] Structured information can be presented in a standardized product information table format, providing a unified input for subsequent multi-path analysis.

[0051] Step 3: Perform multipath analysis on the structured information obtained in Step 2.

[0052] The multi-path analysis includes: ingredient and component analysis, brand and manufacturer analysis, public opinion analysis, and price analysis. Step 3 includes the following steps 301 to 304:

[0053] Step 301: Identify the ingredients and components information in the core information and perform ingredient and component analysis.

[0054] The ingredient and component analysis process involves querying the scientific name, chemical properties, potential risks, and usage instructions of the identified ingredients via the internet.

[0055] Step 302: Identify the brand and manufacturer information in the core information and perform brand and manufacturer analysis.

[0056] The brand and manufacturer analysis process involves using a knowledge graph to match manufacturer backgrounds, extracting manufacturer operational information, and determining brand credibility. This manufacturer operational information includes, but is not limited to, their parent group, capital relationships, and historical events.

[0057] Step 303: Identify public opinion information in the core information and conduct public opinion analysis.

[0058] The public opinion analysis process involves retrieving records of disputes and consumer feedback related to the product or brand from public news sources, e-commerce review databases, and negative event databases.

[0059] Step 304: Identify the price information in the core information and perform price analysis.

[0060] The price analysis process involves capturing information such as the current selling price of products, recent fluctuations, and comparative prices of similar products from major e-commerce platforms.

[0061] Step 4: Input the structured information obtained in Step 2 and the analysis results in Step 3 into the locally deployed large language model to perform semantic fusion and knowledge modeling, and build a user-oriented knowledge graph.

[0062] The operational mechanism of locally deployed large language models typically employs industry-knowledge-enhanced LLM models (such as Llama, ChatGLM, and deepseek, which are open-source models) and deployed on enterprise local servers or private servers after quantization and compression. Then, pre-trained general semantic understanding capabilities are loaded, while integrating knowledge from product standard databases (such as GB national labeling standards) and chemical substance safety databases. Finally, LoRA fine-tuning technology is used to optimize tasks such as product information comparison and risk factor extraction while maintaining basic NLP capabilities.

[0063] Step 5: Using an embedding model, the structured core information and auxiliary information obtained in Step 2 are fused in a multimodal manner to obtain multimodal semantic information.

[0064] The embedding model is the embedding-3 model. The characteristics of the embedding model are as follows:

[0065] The embedding model maps core information, auxiliary information, and product packaging image features to the same high-dimensional semantic space of 768 dimensions or more.

[0066] The embedding model is modeled using a GNN graph neural network to construct a graph structure containing nodes of core product information elements. The edge weights between nodes are dynamically adjusted based on semantic similarity and risk correlation.

[0067] The obtained multimodal semantic information uses 2048 arrays to reflect the semantic feature distribution, contextual relationships, conceptual abstraction level, and potential similarity with other samples in high-dimensional space.

[0068] This invention employs an embedding model that can simultaneously learn the potential associations between the semantic layer (text) and the visual layer (image), achieving a consistent representation of cross-modal information; and providing high-precision multimodal knowledge support for consumer suggestion generation.

[0069] Step 6: Generate personalized consumption suggestions based on the locally deployed large language model.

[0070] The multimodal semantic information fused in step 5 is input into the locally deployed large language model, and personalized consumption suggestions are automatically generated based on the user's basic information and the knowledge graph obtained in step 4.

[0071] The large language model is based on the Transformer architecture and incorporates pre-defined prompt templates, such as "Generate purchase and consumption suggestions based on the following ingredients and brand information." Through contextual learning and semantic alignment mechanisms, the model can automatically determine the product's risk level, target audience, consumption suggestions, and price reasonableness analysis.

[0072] The generated consumer recommendations include, but are not limited to: whether to purchase, whether it is suitable for consumption, recommendations for suitable populations, potential health and ingredient risks, brand and manufacturer credibility, disputes and compliance risks, consumption scenarios and dosage suggestions, and price reasonableness analysis. The generated consumer recommendations are presented to the user interface in natural language.

[0073] Step 7: Collect user feedback on consumption suggestions and store the consumption suggestions and feedback information in the database.

[0074] The system receives user feedback on consumption suggestions, including ratings, text reviews, and suggested corrections. All consumption suggestions and feedback are stored together in a local database for structured storage and to provide data support for subsequent model and knowledge optimization.

[0075] During the knowledge update phase, the system uses the RAG (Retrieval-Augmented Generation) mechanism to associate user feedback with the original knowledge base, enabling dynamic model optimization. During the retrieval phase, it retrieves similar historical data and recommended cases based on user input. During the generation phase, it generates updated knowledge representations based on retrieval results and new feedback. The optimization goal is to continuously improve the adaptability of the large language model to user habits and the accuracy of judging consumption scenarios, forming a continuously evolving knowledge enhancement generation system.

[0076] The above technical solutions can effectively improve users' ability to perceive and judge product information, avoid risks such as mispurchase and incorrect consumption, and significantly improve consumer safety and satisfaction.

[0077] The invention will be described in detail below with examples: The method of the present invention is applicable to fast-moving consumer goods, health products, and other products that are highly sensitive to ingredients, brand, and reputation. The actual usage process is as follows:

[0078] A user takes a photo of a brand's ready-to-eat seaweed product via a mobile app. The system automatically completes image acquisition, OCR recognition, information extraction, and knowledge enhancement. In the OCR stage, the system successfully extracted the following fields:

[0079] Product Name: Seasoned Ready-to-Eat Seaweed

[0080] Ingredients: Seaweed, sesame oil, salt, monosodium glutamate

[0081] Manufacturer: XXX Food (Qingdao) Co., Ltd.

[0082] Product standard number: GB / T 23596

[0083] License No.: SC12237021300231

[0084] Net weight: 50g

[0085] Place of origin: Qingdao, Shandong Province

[0086] After entering the external knowledge enhancement module, the system, relying on its built knowledge graph and networked database, analyzes "monosodium glutamate" and identifies its scientific name as "MSG," classifying it as a "synthetic seasoning." Given the potential controversy surrounding its effects on children's nervous system development, the system automatically marks it as "recommended for moderate consumption." Simultaneously, by identifying the manufacturer's background, the system finds that the manufacturer was previously notified by market regulators in 2022 for "non-compliant labeling," which the system identifies as a "mild negative event" and assigns it a "medium brand risk" rating.

[0087] The system also retrieved price data for the product from major e-commerce platforms over the past 30 days, analyzed its average selling price of 15.8 yuan, and found that the lowest price through group buying channels could be as low as 12.5 yuan, inferring that there is room for price fluctuation and it is suitable for centralized procurement.

[0088] Based on the above structured information and semantic fusion results, the system automatically generates the following content using a prompt-word-driven language model:

[0089] Purchase Recommendation: Suitable for general consumption, but pregnant women and children under 3 years old are advised to reduce their intake; brand credibility is generally low, it is recommended to check the production batch.

[0090] Consumption recommendations: It is recommended to limit daily intake to less than 5g and consume it with staple foods;

[0091] Suitable for: Students, office workers, and people trying to lose weight can consume in moderation;

[0092] Price recommendation: We suggest placing orders through group buying or e-commerce promotional activities to avoid purchasing individually during peak periods.

[0093] Users can rate and comment on the system's suggestions, and their feedback data will be recorded in the local RAG database for future analysis of similar products.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for food packaging information identification and comprehensive analysis based on a multimodal large model, characterized in that, Includes the following processes: Step 1: Obtain basic user information and product packaging images uploaded by the user; Step 2: Perform text recognition on the content of the product packaging image to extract the core information fields and auxiliary information fields of the product, and perform structured processing through entity recognition and field classification to obtain structured information; Step 3: Perform multipath analysis on the structured information obtained in Step 2; Step 4: Input the structured information obtained in Step 2 and the analysis results in Step 3 into the locally deployed large language model to perform semantic fusion and knowledge modeling, and build a user-oriented knowledge graph. Step 5: Using an embedding model, the structured core information and auxiliary information obtained in Step 2 are fused in a multimodal manner to obtain multimodal semantic information; Step 6: Generate personalized consumption suggestions based on the locally deployed large language model.

2. The method for food packaging information identification and comprehensive analysis based on a multimodal large model according to claim 1, characterized in that, Following step 6, the following is also included: Step 7: Collect user feedback on consumption suggestions and store the consumption suggestions and feedback information in the database.

3. The method for food packaging information identification and comprehensive analysis based on a multimodal large model according to claim 1, characterized in that, The user's basic information includes population type, health status, and dietary preferences.

4. The method for food packaging information identification and comprehensive analysis based on a multimodal large model according to claim 1, characterized in that, Step 2 includes the following steps 201 to 202: Step 201: Based on the OCR engine and multimodal large model, perform text recognition on the product packaging image to extract the core information fields and auxiliary information fields of the product; The multimodal large model adopts the qwen2.5vl-72b model and uses the pre-trained knowledge of the qwen2.5vl-72b model for recognition. The core information includes product name, ingredients, product standard number, production license number, manufacturer name, place of origin, net content, and brand. The auxiliary information includes ingredient icons, warning signs, and certification badges; Step 202: Perform entity recognition and field classification on the core information and auxiliary information extracted in step 201 to obtain structured information.

5. The method for food packaging information identification and comprehensive analysis based on a multimodal large model according to claim 1, characterized in that, Step 3 includes the following steps 301 to 304: Step 301: Identify the ingredients and components information in the core information and perform ingredient and component analysis; Step 302: Identify the brand and manufacturer information in the core information and perform brand and manufacturer analysis; Step 303: Identify public opinion information in the core information and conduct public opinion analysis; Step 304: Identify the price information in the core information and perform price analysis.

6. The method for food packaging information identification and comprehensive analysis based on a multimodal large model according to claim 1, characterized in that, The embedding model is the embedding-3 model.

7. The method for food packaging information identification and comprehensive analysis based on a multimodal large model according to claim 1, characterized in that, The consumer advice includes whether to recommend purchasing, whether it is suitable for consumption, recommendations for the target population, potential health and ingredient risks, brand and manufacturer credibility, disputes and compliance risks, recommendations for consumption scenarios and dosages, and analysis of price reasonableness.

Images