Dish Analysis Method, Device, Equipment, and Medium Based on Multimodal Knowledge Graph

Abstract

The present disclosure provides a dish analysis method, device, equipment, and medium based on a multimodal knowledge graph, belonging to the technical field of data processing. The method includes: matching dish image data and dish text data, and constructing multimodal dish data based on the matched data; constructing a multimodal knowledge graph based on the multimodal dish data; the multimodal knowledge graph includes a full-link data inference structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain; clustering the dish multimodal data set to obtain multiple dish data subsets; annotating the search priority of each dish data subset based on the data volume; in response to receiving input information, searching for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset. The present disclosure can achieve precise matching between local regions of images and ingredients in dish analysis, and improve the fine-grained dish analysis ability.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of data processing, and more specifically, relates to a dish analysis method, device, equipment, and medium based on a multimodal knowledge graph. Background Art

[0002] With the digital transformation of the catering industry, food ingredient detection technology based on image recognition has been widely used in fields such as catering recommendations. Existing methods mainly rely on the extraction of overall image features, such as extracting global semantic features, for simple overall positioning and classification of food ingredients. In this context, some solutions combine text descriptions to improve detection accuracy. In addition, the introduction of knowledge graph technology further integrates structured information such as food ingredient attributes and cuisine classifications, improving the comprehensiveness of dish analysis.

[0003] However, existing dish analysis based on knowledge graphs is limited to the representation learning of a single dimension of food, relying on overall image features, with insufficient fine-grained food ingredient analysis capabilities, and it is difficult to handle food ingredient occlusion and fragmentation problems in complex dishes, resulting in inaccurate recommendation results and unable to meet the diverse needs of users. For example, the matching relationship between local regions of an image and food ingredients is blurred, prone to false correlations, and the accuracy is low. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a dish analysis method, device, equipment, and medium based on a multimodal knowledge graph to achieve precise matching between local regions of an image and food ingredients in dish analysis and improve the fine-grained dish analysis ability.

[0005] In the first aspect of the embodiments of the present disclosure, a dish analysis method based on a multimodal knowledge graph is provided, including:

[0006] Match dish image data and dish text data, and construct multimodal dish data based on the matched data; construct a multimodal knowledge graph based on the multimodal dish data; the multimodal knowledge graph includes a full-link data inference structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain;

[0007] Cluster the dish multimodal data set to obtain multiple dish data subsets; label the search priority of each dish data subset based on the data volume;

[0008] In response to receiving input information, search for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset.

[0009] In the second aspect of the embodiments of the present disclosure, a dish analysis device based on a multimodal knowledge graph is provided, including:

[0010] A data construction module for matching dish image data and dish text data, constructing multi-modal dish data based on the matched data; constructing a multi-modal knowledge graph based on the multi-modal dish data; the multi-modal knowledge graph includes a full-link data inference structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain;

[0011] A priority division module for clustering the dish multi-modal data set to obtain multiple dish data subsets; annotating the search priority of each dish data subset based on the data volume;

[0012] A search analysis module for, in response to receiving input information, searching for target dish information from the multi-modal knowledge graph based on the input information and the search priority of each dish data subset.

[0013] In a third aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above-mentioned dish analysis method based on a multi-modal knowledge graph are implemented.

[0014] In a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned dish analysis method based on a multi-modal knowledge graph are implemented.

[0015] The beneficial effects of the dish analysis method, device, equipment, and medium based on a multi-modal knowledge graph provided by the embodiments of the present disclosure are as follows: By deeply integrating image and text data, the present disclosure constructs a three-dimensional knowledge network containing visual features and semantic information, significantly improving the accuracy and efficiency of dish analysis.

[0016] On the one hand, the multi-modal data fusion of the present disclosure effectively solves the problem of the limitation of single-modal information. Through the cross-validation of image local features and ingredient text descriptions, accurate parsing of fine-grained attributes is achieved. The full-link inference structure supports complex relationship reasoning, such as the association mining of "ingredient - cooking technique - flavor feature", providing a decision basis for scenarios such as intelligent recipe generation and nutritional calculation.

[0017] On the other hand, the dynamic priority clustering mechanism of the present disclosure enables high-frequency dish data to respond quickly. Combining the weight decay algorithm of the weighted relationship chain, the retrieval efficiency is improved while maintaining the analysis depth. The present disclosure supports cross-modal semantic alignment. In the case of fuzzy query scenarios, missing information is automatically supplemented through multi-round iterative reasoning, promoting the evolution of dish analysis from simple classification to knowledge reasoning. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 Schematic flowchart of a dish analysis method based on a multi-modal knowledge graph provided by an embodiment of the present disclosure;

[0020] Figure 2 Schematic flowchart of the actual application process of a dish analysis method based on a multi-modal knowledge graph provided by an embodiment of the present disclosure;

[0021] Figure 3 Block diagram of the structure of a dish analysis device based on a multi-modal knowledge graph provided by an embodiment of the present disclosure;

[0022] Figure 4 Schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0023] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present disclosure. However, those skilled in the art should clearly understand that the present disclosure can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present disclosure.

[0024] To make the objectives, technical solutions, and advantages of the present disclosure clearer, the following will be described through specific embodiments in conjunction with the accompanying drawings.

[0025] Please refer to Figure 1 , Figure 1 Schematic flowchart of a dish analysis method based on a multi-modal knowledge graph provided by an embodiment of the present disclosure. The method may include S101 to S103.

[0026] S101: Match the dish image data and the dish text data, and construct multi-modal dish data based on the matched data. Construct a multi-modal knowledge graph based on the multi-modal dish data. The multi-modal knowledge graph includes a full-link data inference structure in which multiple entity nodes are interconnected pairwise, and each two entity nodes correspond to a weighted relationship chain.

[0027] In this embodiment, the dish image data includes an image set that integrates visual information such as the appearance, color, and shape of the dish. The dish text data may include information described in words such as dish attributes, ingredients, cooking methods, nutritional components, applicable populations, taboos, related diseases, and suitable eating scenarios. The multimodal dish data refers to the comprehensive data in which the vision and semantic information are aligned after the image and text are matched and integrated. The multimodal knowledge graph is a graph-structured knowledge base that integrates dish images, dish texts, and structured data. The multimodal knowledge graph contains entity nodes and relationship edges (i.e., weighted relationship chains).

[0028] The entity node is the basic component element in the multimodal knowledge graph and can represent various entities in the real world. For example, the entity node types can include dishes, diseases, recipes, ingredients, nutrition, and scenarios, etc. Among them, the scenario refers to the suitable eating scenario, applicable population, etc.

[0029] The weighted relationship chain is the relationship link connecting two entity nodes in the knowledge graph and assigns a certain weight to this relationship, that is, the edge weight, which is used to quantify the association strength between the two entity nodes. The edge weight can be calculated based on the co-occurrence frequency, matching degree, semantic similarity, etc.

[0030] The full-link data inference structure refers to a complete data structure formed by the mutual connection and correlation of all entity nodes in the multimodal knowledge graph. The full-link data inference structure allows starting from any entity node and reaching any other entity node through a series of relationship chains, so as to achieve comprehensive data inference and information retrieval. The full-link data inference structure supports complex logical inferences based on multimodal data. For example, according to the ingredients and scenarios input by the user, the suitable dishes and recipes can be inferred.

[0031] In this embodiment, by integrating the visual features of the dish image and the semantic information of the text data, a multimodal association relationship is established, including: extracting features from the image and text respectively, realizing cross-modal matching through similarity calculation, and forming multimodal dish data containing visual-semantic dual information; extracting entity features such as dishes, ingredients, nutritional components, etc. and relationship attributes such as "contains", "belongs to", "taboo", etc. from the multimodal data. Based on the correlation between entities, weights are assigned to the relationship chains to construct a knowledge graph that supports full-link reasoning.

[0032] Exemplarily, OpenCV is used to crop (retain the main area), scale (unify the size to 224×224), and denoise (Gaussian filtering) the dish image to improve the quality of feature extraction. Based on the convolutional neural network ResNet50, global visual features such as ingredient distribution, color distribution, and overall shape are extracted, and then combined with local descriptors, such as using SIFT / SURF to extract ingredient textures, tableware details, etc., to jointly generate an image feature vector.

[0033] Use NLP tools such as Jieba word segmentation and spaCy named entity recognition to extract entity features from dish text data, and construct text semantic vectors using TF-IDF or BERT embeddings. Match the global features of the image with the text semantic vectors through cosine similarity to screen the preliminarily matched texts; further generate "image-text" matching pairs through the fine-grained association between local features and texts. Integrate the matched image features and text entities into multi-modal data entries.

[0034] Exemplarily, extract core entity types from multi-modal data: dish nodes, ingredient nodes, nutrition nodes, disease nodes, scenario nodes, etc. Attach attributes to each node. For example, dish nodes include "cuisine", "taste", "production time", and ingredient nodes include "origin", "season", "calorie".

[0035] Establish various basic relationship types such as "dish-ingredient" inclusion relationship, "ingredient-nutrition" attribute relationship, "disease-taboo" association relationship, etc. Count the co-occurrence times of "Kung Pao Chicken" and "chicken" to generate the weight of the "inclusion" relationship. Calculate the semantic distance between "high fat" and "hyperlipidemia" through Word2Vec to generate the weight of the "taboo" relationship.

[0036] Use the Neo4j graph database to store entities and relationships, construct a radial graph centered on dishes, and support multi-hop queries such as "dish→ingredient→nutrition→disease". Eliminate duplicate nodes through entity alignment algorithms such as TF-IDF-based synonym table matching to ensure the consistency of the graph. Ensure that there is at least one inference path between any two nodes. Dynamically adjust the relationship chain weight through user search logs to enhance the inference efficiency.

[0037] Through the above steps, a complete technical link covering "image-text matching→multi-modal data construction→knowledge graph empowerment" can be formed, providing a structured data basis for subsequent intelligent analysis.

[0038] S102: Cluster the dish multi-modal data set to obtain multiple dish data subsets. Mark the search priority of each dish data subset based on the data volume.

[0039] In this embodiment, the dish data subsets are obtained by clustering the multi-modal data set into several subsets with similar features. The dishes within each subset have high consistency in image vision, text semantics, or structured attributes. The search priority is the preferential retrieval order given to them during the search process according to the scale of the data subset or the high-frequency needs of users. The subsets with large data volume or high frequency are preferentially matched to improve the query efficiency.

[0040] In this embodiment, the dish data is clustered through multi-modal feature fusion to form subsets with semantic similarity; a priority function is designed based on the subset data volume, and each dish data subset is labeled with a priority to ensure that high-frequency and large-scale data subsets are preferentially retrieved during user queries, reducing the computational overhead.

[0041] Exemplarily, the K-means++ or DBSCAN algorithm is adopted to determine the optimal number of clusters K according to the elbow method. The cosine similarity is used to calculate the distance between multi-modal vectors to ensure that visual and semantic similarity are measured simultaneously. The multi-modal feature matrix is input into the clustering model to obtain the cluster label of each data point. The data volume within each cluster is counted to generate subset metadata containing the cluster center, entity label, and data volume.

[0042] After clustering, multiple dish data subsets are obtained. The priority score of each subset is calculated, and a target search sequence is generated in descending order. For example, high-priority subsets preferentially include "home-cooked dishes" and "popular dishes". The priority metadata is stored in the knowledge graph and associated with entity nodes. When the user inputs a query, the entity nodes within the high-priority subset are preferentially retrieved. When the user does not specify the dish type, the search defaults to starting from the "home-cooked dishes" subset. Additionally, the subset data volume needs to be recalculated regularly. When the data volume of a certain subset increases by more than a preset ratio, its priority is automatically increased to ensure that the search strategy is dynamically optimized as the data distribution changes.

[0043] S103: In response to receiving the input information, search for the target dish information from the multi-modal knowledge graph based on the input information and the search priority of each dish data subset.

[0044] In this embodiment, the input information refers to the query content input by the user, which can include audio question information, text questions, or dish pictures, etc., for triggering the knowledge graph search. The input information can also include the user's historical relevant data obtained by the system. The target dish information refers to the dish details matching the input information obtained through the search, which can include multi-dimensional data such as ingredients, nutritional components, suitable populations, cooking steps, etc., and can also generate a visual complete reasoning process.

[0045] Such as Figure 2As shown, exemplarily, the user inputs: the user inputs a question and a picture, for example: "Is this dish suitable for people with three highs?" plus a picture of the dish. Entity recognition, intent analysis, and picture retrieval: Use a large model to recognize the key entities in the user input question, such as "people with three highs". Conduct intent analysis on the entities to match similar nodes in the knowledge graph, such as "hyperlipidemia", "hypertension", and "hyperglycemia". For the picture retrieval process, it includes: Picture preprocessing: Preprocess the picture uploaded by the user, including operations such as resizing the picture, denoising, and cropping, to ensure picture quality and consistency. Feature extraction: Use deep learning models such as convolutional neural networks to extract the features in the picture. The convolutional neural network performs convolutional operations on the picture through multiple convolutional kernels to extract features at different levels, such as edges, textures, and shapes. Feature matching and similarity calculation: Match the extracted picture features with the features of the dish category pictures in the knowledge graph. Determine the most similar dish picture by calculating the cosine similarity, Euclidean distance, etc. between the features.

[0046] Exemplarily, in this embodiment, the input information is parsed: For text information, large models such as BERT can be used to perform entity recognition on the input information, extract keywords, and determine the type of user demand to generate a query target. For image data, a convolutional neural network (CNN) can be used to extract global visual features and local features to generate feature vectors. Generate query conditions based on the obtained keywords, entity features, or feature vectors, etc., and determine a subset of dish data that meets the conditions to narrow the retrieval scope. Retrieve the multi-modal knowledge graph in descending order of the priority of the dish data subset, find associated nodes through breadth-first search, and perform logical operations on all the retrieved results, such as merging the common taboo nodes ("high salt") of "diabetes" and "hypertension".

[0047] Generate an inference path, such as "user input → entity node → subset priority → matching dish", and label the knowledge source. Display the inference trajectory in the form of a mind map, such as "diabetes → taboo high sugar → matching dish 'cold spinach' → low GI value", and additionally supplement and label the dish attributes and associated scenarios. If the retrieval results are insufficient, prompt the user to supplement the input through a large model, such as "Do you need to query local specialty dishes?".

[0048] It can be concluded from the above that in this embodiment, by deeply integrating image and text data, a three-dimensional knowledge network containing visual features and semantic information is constructed, significantly improving the accuracy and efficiency of dish analysis.

[0049] On the one hand, the multi-modal data fusion in this embodiment effectively solves the problem of limited single-modal information. Through the cross-validation of the local features of the image and the text description of the ingredients, the accurate parsing of fine-grained attributes is achieved. The full-link inference structure supports complex relationship reasoning, such as the association mining of "ingredients - cooking techniques - flavor characteristics", providing a decision-making basis for scenarios such as intelligent recipe generation and nutritional calculation.

[0050] On the other hand, the dynamic priority clustering mechanism in this embodiment enables the high-frequency dish data to respond quickly. Combining the weight decay algorithm of the weighted relationship chain, the retrieval efficiency is improved while maintaining the analysis depth. This disclosure supports cross-modal semantic alignment. In the fuzzy query scenario, the missing information is automatically completed through multi-round iterative reasoning, promoting the evolution of dish analysis from simple classification to knowledge reasoning.

[0051] In an embodiment of the present disclosure, the dish image data and the dish text data are matched, and multi-modal dish data is constructed based on the matched data, including:

[0052] Extract the global semantic features and local semantic features of the dish image data.

[0053] Match the global semantic features with the dish text data to obtain the first text description that matches the dish image data successfully.

[0054] Match the local semantic features with the dish text data to obtain the second text description that matches the dish image data successfully.

[0055] Construct multi-modal dish data based on the dish image data, the first text description, and the second text description.

[0056] In this embodiment, matching the global semantic features with the dish text data to obtain the first text description that matches the dish image data successfully includes:

[0057] Calculate the first matching degree between the global semantic features and the dish text data.

[0058] If the first matching degree is greater than or equal to the first matching threshold, it is determined that the dish text data matches the global semantic features successfully, and the dish text data is used as the first text description.

[0059] In this embodiment, matching the local semantic features with the dish text data to obtain the second text description that matches the dish image data successfully includes:

[0060] Calculate the third matching degree between the local semantic features and the dish text data.

[0061] If the third matching degree is greater than or equal to the third matching threshold, it is determined that the dish text data matches the local semantic features successfully, and the dish text data is used as the second text description.

[0062] In this embodiment, the global semantic features may include abstract features representing the overall visual semantics of the dish image, such as macroscopic visual information like color distribution, shape contour, composition structure, etc., which are used to describe the overall category or main attributes of the dish. The local semantic features may include the detailed visual features of key regions in the dish image, such as ingredient textures, specific ingredients, cooking states, etc., which are used to precisely match the specific descriptions in the text. The first matching degree refers to the similarity between the global semantic features and the text semantic vector, which is used to measure the matching degree between the overall visual of the image and the macroscopic description of the text. The third matching degree refers to the similarity between the local features and the text detail semantics, which is used to measure the similarity between the detailed visual of the image and the specific description of the text. The first matching threshold and the third matching threshold are preset similarity critical values, which are used to determine whether the global matching is successful.

[0063] In this embodiment, the overall features of the image can be extracted using a CNN, and the similarity with the global semantic vector of the text is calculated to screen out the text that matches the macroscopic category of the image, such as macroscopic semantics like "Kung Pao Chicken", "Sichuan cuisine", "spicy", etc. The key regions such as ingredients and seasonings in the image are located through object detection, and after extracting the detailed features, they are matched with the specific descriptions in the text to supplement and refine the information. The original image data, the macroscopic description obtained from the global matching, and the detailed description obtained from the local matching are integrated to form structured multi-modal data containing "overall category + detailed attributes", providing hierarchical input for the knowledge graph.

[0064] Exemplarily, the image is cropped to the main body region using OpenCV, and the cropped image is preprocessed. The preprocessed image is input into the pre-trained ResNet50 model to obtain the 2048-dimensional global features output by the avg_pool layer. It is reduced to 1024 dimensions through a fully connected layer, and L2 normalization is performed to obtain the global feature vector v_global_image.

[0065] Use the YOLOv5s model to detect the ingredient / condiment areas in the image and generate a list of bounding boxes: including class labels such as "chicken", "peanut", "chili pepper". Crop the sub-images within each bounding box as local feature extraction units. Extract Scale-Invariant Feature Transform (SIFT) features for each sub-image: construct a Gaussian pyramid, detect the extreme points of the Difference of Gaussians (DOG) to locate key points; calculate the 8-direction gradient histogram centered on the key points to generate a 128-dimensional descriptor; sort by response value and retain L = 500 strongest descriptors to form the local feature set D_local_image.

[0066] Use spaCy for named entity recognition and parse the dish text into three types of entities: global entities: dish name, cuisine, dish type; local entities: ingredients, condiments, cooking status; attribute entities: taste, nutrition, preparation time.

[0067] Input the global entity list into the BERT-base model to obtain the 768-dimensional embedding vector v_global_text at the CLS position, which represents the overall semantics of the text. For each local entity, use the Word2Vec model to generate 300-dimensional word vectors; calculate the weighted average according to the entity appearance frequency to obtain the local semantic vector set V_local_text.

[0068] In this embodiment, calculating the first matching degree between the global semantic features and the dish text data includes:

[0069] Calculate the first matching degree between the global semantic features and the dish text data based on the global matching degree function.

[0070] The global matching degree function is:

[0071]

[0072] Among them, Sim_global represents the first matching degree, v_global_image represents the global semantic features, and v_global_text represents the dish text data.

[0073] In this embodiment, calculating the third matching degree between the local semantic features and the dish text data includes:

[0074] For each local semantic feature: find the nearest text vector in the dish text data.

[0075] Calculate the third matching degree between the local semantic features and the nearest text vector based on the local matching degree function.

[0076] The local matching degree function is as follows:

[0077]

[0078]

[0079] Among them, Sim_local represents the third matching degree, Dist represents the Euclidean distance between the local semantic feature and the nearest text vector, represents the i-th local semantic feature, represents the i-th nearest text vector.

[0080] In this embodiment, through the hierarchical matching of global semantic features and local semantic features, cross-modal alignment of macro classification and micro refinement is achieved, avoiding the ambiguity of single-feature matching and improving the matching accuracy. This embodiment combines the original image features, macro text descriptions, and detailed text descriptions to form a standardized data structure including overall categories and detailed attributes, providing multi-dimensional input for the knowledge graph and supporting subsequent in-depth reasoning. In this embodiment, global matching is first used to filter out irrelevant texts, reducing the amount of local matching calculations; local matching realizes precise screening in the remaining text data, improving the matching efficiency.

[0081] In an embodiment of the present disclosure, the dish image data and the dish text data are matched, and multi-modal dish data is constructed based on the matched data, including:

[0082] Extract the semantic features of the dish text data, and divide the dish text data into multiple dish units based on the semantic features. Each dish unit includes multiple entity data.

[0083] Match each dish unit with the dish image data, and construct multi-modal dish data based on the matched data.

[0084] In this embodiment, the multiple entity data includes ingredient data, recipe data, dish data, nutrition data, and scenario data.

[0085] Match each dish unit with the dish image data, and construct multi-modal dish data based on the matched data, including:

[0086] For each dish unit:

[0087] Calculate the second matching degree of the dish data, recipe data, and dish image data.

[0088] If the second matching degree is greater than or equal to the second matching threshold, the dish image data and the dish unit are successfully matched, and multi-modal dish data is constructed based on the dish image data and the dish unit.

[0089] In this embodiment, the dish unit is an independent information set divided from the dish text data according to semantic features, and each dish unit contains entity data such as ingredients and recipes. The second matching degree is used to measure the similarity of the dish data, recipe data, and dish image data in the dish unit. The second matching threshold is a preset critical value used to determine whether the matching is successful.

[0090] Exemplarily, semantic analysis is performed on the dish text data and divided into multiple dish units, and each unit represents the relevant information of a dish. For each dish unit, the matching degrees of the dish data, recipe data, and dish image data therein are calculated. If the matching degree reaches the threshold, it indicates that the dish unit matches the image data successfully. Finally, the two are fused to construct multi-modal dish data.

[0091] In this embodiment, calculating the second matching degrees of the dish data, recipe data, and dish image data includes:

[0092] Calculating the fourth matching degree between the dish data and the dish image data, and calculating the fifth matching degree between the recipe data and the dish image data.

[0093] Calculating the second matching degree based on the dish unit matching function, the fourth matching degree, and the fifth matching degree.

[0094] The dish unit matching function is:

[0095]

[0096]

[0097]

[0098] Among them, α and β represent weight coefficients, α + β = 1, Sim_unit represents the second matching degree, Sim_dish represents the fourth matching degree, and Sim_recipe represents the fifth matching degree.

[0099] In this embodiment, the text is divided into independent dish units, and the matching degrees of the dish data and the recipe data are calculated respectively, avoiding the ambiguity of traditional single matching and improving the matching accuracy. In this embodiment, the matching priorities of the dish and the recipe are dynamically adjusted through weight coefficients to meet diverse requirements. This embodiment constructs structured data including "dish - recipe - image", provides fine-grained input for the knowledge graph, supports cross-modal reasoning of "ingredient → cooking → nutrition", solves the problem of single dimension in the prior art, and improves the comprehensiveness of analysis. This embodiment combines the hierarchical matching of image features and text semantics, and can still effectively match fuzzy input through the weighting strategy, with strong robustness.

[0100] This embodiment mainly focuses on text data and searches for matching image data. The previous embodiment targeted image data and searched for matching text data. This two-way matching method can improve the integrity and accuracy of multi-modal dish data. Searching for images by text can accurately locate visual information, and searching for text by images can supplement detailed descriptions. The combination of the two can more comprehensively explore the relationships between data, provide users with richer and more three-dimensional dish information, and enhance the practicality of the knowledge graph and the effectiveness of intelligent recommendations.

[0101] In one embodiment of the present disclosure, the multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes.

[0102] Constructing a multi-modal knowledge graph based on multi-modal dish data, including:

[0103] Performing a correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes in the multi-modal dish data to obtain the correlation coefficient between every two entity nodes.

[0104] Generating a weighted relationship chain for every two entity nodes based on the correlation coefficient.

[0105] In this embodiment, the ingredient node can include attributes such as name, category, and food taboos; the recipe node can include attributes such as cooking methods, steps, tools, and time consumption; the dish node can include attributes such as dish name, cuisine, taste, and production difficulty; the nutrition node can include attributes such as nutrition component name, content, and health impact; the scenario node can include attributes such as applicable population tags, occasions, and seasons. The correlation coefficient is a numerical value that quantifies the association strength between two entity nodes, and the higher the value, the stronger the association. The weighted relationship chain refers to a directed edge connecting two entity nodes, and the edge weight is the correlation coefficient.

[0106] Exemplarily, for the ingredient node: Extract the name from the dish text ingredient list and the image ingredient detection results, and supplement the nutrition attributes;

[0107] For the dish node: Extract the name from the image retrieval results and the text title, and associate attributes such as cuisine and taste;

[0108] Merge duplicate nodes through a custom synonym table and TF-IDF similarity to ensure entity uniqueness, such as unifying "potato" as "potato".

[0109] For the correlation analysis between ingredients and dishes, and between dishes and recipes, the co-occurrence frequency method can be used to count the number of times an ingredient appears in a dish, calculate the normalized co-occurrence coefficient, and the co-occurrence coefficient = the number of times the ingredient appears in the dish / the total number of times the ingredient appears.

[0110] For the correlation analysis of nutrition - scenario and ingredient - nutrition, the cosine similarity of entity vectors can be calculated using BERT. For example, the similarity between "high - fat" and "hyperlipidemia" is 0.85, which is used as the weight of the "nutrition - taboo - disease" relationship.

[0111] For the correlation analysis of disease - scenario and recipe - nutrition, strong association weights can be set according to relevant dietary guidelines. For example, the weight of "hypertension - recommended diet - low - salt" = 1.0, and the weight of "high - sugar - taboo - diabetes" = 0.95.

[0112] In this embodiment, a weighted relationship chain is constructed through co - occurrence frequency, semantic similarity, and domain rules to achieve cross - dimensional association of five types of entities: ingredients, recipes, dishes, nutrition, and scenarios, covering the entire "ingredient - dish - scenario" link, solving the problem of single - dimensional analysis in the prior art, and improving the reasoning accuracy. The weighted relationship chain in this embodiment supports weighted path search and can also dynamically adjust the reasoning path in combination with the user's historical information to make the recommendation results more in line with personalized needs. In this embodiment, a synonym table and TF - IDF are used to remove duplicates to ensure entity uniqueness and avoid data redundancy; the combination of domain rules and data - driven methods improves the scientific basis of the knowledge graph and reduces reasoning deviation. The weighted relationship chain clearly marks the association strength, and the reasoning process is transparent, which can improve user trust.

[0113] In an embodiment of the present disclosure, the search priority of each dish data subset is marked based on the data volume, including:

[0114] Determine the search priority of each dish data subset based on the search priority function and the data volume.

[0115] The search priority function is:

[0116]

[0117] Wherein, represents the search priority of the dish data subset , represents the data volume of the data subset , N represents the total data volume of all dish data subsets, represents the historical search frequency of the data subset , F represents the total global search frequency, and γ represents the weight coefficient.

[0118] Mark all dish data subsets based on the search priority.

[0119] In this embodiment, the search priority function combines two factors: the amount of data and the historical search frequency. The proportion of the amount of data reflects the scale of the subset in the overall data, and the proportion of the historical search frequency reflects the degree of attention of the subset. By performing a weighted sum of the two using a weight coefficient, the search priority of each subset can be obtained. All subsets are labeled and sorted according to the priority. When performing a dish search, the dish data subsets are searched in sequence according to the target search sequence, and the subsets with higher priority are processed first. This is to improve the search efficiency and accuracy.

[0120] In this embodiment, through the weighted fusion of the amount of data and the historical search frequency, the data scale and user preferences are dynamically balanced, enabling high-frequency subsets and large-scale subsets to be retrieved first, thereby improving the search efficiency.

[0121] Corresponding to the dish analysis method based on a multi-modal knowledge graph in the above embodiment, Figure 3 is a structural block diagram of a dish analysis device based on a multi-modal knowledge graph provided by an embodiment of the present disclosure. For ease of illustration, only parts related to the embodiments of the present disclosure are shown. Refer to Figure 3 The dish analysis device 20 based on the multi-modal knowledge graph includes: a data construction module 21, a priority division module 22, and a search analysis module 23.

[0122] Among them, the data construction module 21 is configured to match dish image data and dish text data, and construct multi-modal dish data based on the matched data. A multi-modal knowledge graph is constructed based on the multi-modal dish data. The multi-modal knowledge graph includes a full-link data inference structure in which all pairs of entity nodes communicate with each other, and each pair of entity nodes corresponds to a weighted relationship chain.

[0123] The priority division module 22 is configured to cluster the dish multi-modal data set to obtain multiple dish data subsets. The search priority of each dish data subset is labeled based on the amount of data.

[0124] The search analysis module 23 is configured to, in response to receiving input information, search for target dish information from the multi-modal knowledge graph based on the input information and the search priority of each dish data subset.

[0125] In an embodiment of the present disclosure, the data construction module 21 is specifically configured to extract the global semantic features and local semantic features of the dish image data.

[0126] Based on the global semantic features and the dish text data, a match is performed to obtain a first text description that successfully matches the dish image data.

[0127] Based on the local semantic features and the dish text data, a match is performed to obtain a second text description that successfully matches the dish image data.

[0128] Construct multimodal dish data based on dish image data, the first text description, and the second text description.

[0129] In one embodiment of the present disclosure, the data construction module 21 is further specifically configured to calculate the first matching degree between the global semantic feature and the dish text data.

[0130] If the first matching degree is greater than or equal to the first matching threshold, it is determined that the dish text data matches the global semantic feature successfully, and the dish text data is used as the first text description.

[0131] In one embodiment of the present disclosure, the data construction module 21 is further specifically configured to match the dish image data and the dish text data, and construct multimodal dish data based on the matched data, including:

[0132] Extract the semantic features of the dish text data, and divide the dish text data into multiple dish units based on the semantic features. Each dish unit includes multiple entity data.

[0133] Match each dish unit with the dish image data, and construct multimodal dish data based on the matched data.

[0134] In one embodiment of the present disclosure, the multiple entity data includes ingredient data, recipe data, dish data, nutrition data, and scene data. The data construction module 21 is further specifically configured to, for each dish unit:

[0135] Calculate the second matching degree between the dish data, the recipe data, and the dish image data.

[0136] If the second matching degree is greater than or equal to the second matching threshold, the dish image data matches the dish unit successfully, and multimodal dish data is constructed based on the dish image data and the dish unit.

[0137] In one embodiment of the present disclosure, the multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scene nodes. The data construction module 21 is further specifically configured to perform a correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scene nodes in the multimodal dish data to obtain the correlation coefficient between each two entity nodes.

[0138] Generate a weighted relationship chain for each two entity nodes based on the correlation coefficient.

[0139] In one embodiment of the present disclosure, the priority division module 22 is specifically configured to determine the search priority of each dish data subset based on the search priority function and the data volume.

[0140] The search priority function is:

[0141]

[0142] Among them, represents the search priority of the dish data subset , represents the data volume of the data subset , N represents the total data volume of all dish data subsets, represents the historical search frequency of the data subset , F represents the total global search frequency, and α represents the weight coefficient.

[0143] Label all dish data subsets based on the search priority.

[0144] See Figure 4 , Figure 4 which is the schematic block diagram of the electronic device provided by an embodiment of the present disclosure. As Figure 4 shown, the electronic device 300 in this embodiment may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The above-mentioned processors 301, input devices 302, output devices 303, and memories 304 communicate with each other through the communication bus 305. The memory 304 is used to store computer programs, and the computer programs include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. Among them, the processor 301 is configured to call the program instructions to execute the functions of each module in the above-mentioned device embodiments, such as Figure 3 the functions of the data construction module 21, the priority division module 22, and the search analysis module 23 shown.

[0145] It should be understood that in the embodiments of the present disclosure, the so-called processor 301 may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0146] The input device 302 may include a touchpad, a fingerprint acquisition sensor (for acquiring the fingerprint information and the direction information of the fingerprint) of the user, a microphone, etc., and the output device 303 may include a display (such as an LCD), a speaker, etc.

[0147] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A part of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0148] In a specific implementation, the processor 301, the input device 302, and the output device 303 described in the embodiments of the present disclosure may implement the implementation manners described in the first and second embodiments of the dish analysis method based on a multimodal knowledge graph provided by the embodiments of the present disclosure, and may also implement the implementation manner of the electronic device 300 described in the embodiments of the present disclosure, which will not be elaborated herein.

[0149] In another embodiment of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, all or part of the processes in the methods of the above embodiments are implemented. It can also be completed by instructing related hardware through the computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0150] The computer-readable storage medium may be an internal storage unit of the electronic device in any of the foregoing embodiments, such as the hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store the data that has been output or will be output.

[0151] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this disclosure.

[0152] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described electronic devices and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0153] In several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces or units, or can also be electrical, mechanical, or other forms of connection.

[0154] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of this disclosure.

[0155] In addition, the functional units in various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0156] The above is only the specific implementation manner of this disclosure, but the protection scope of this disclosure is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by this disclosure, and these modifications or substitutions should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A dish analysis method based on a multi-modal knowledge graph, characterized in that, Including: Matching dish image data and dish text data, and constructing multimodal dish data based on the matched data; Constructing a multimodal knowledge graph based on the multimodal dish data; The multimodal knowledge graph includes a full-link data inference structure in which multiple entity nodes are interconnected pairwise, and each two entity nodes correspond to a weighted relationship chain; Clustering the dish multimodal data set to obtain multiple dish data subsets; Determining the search priority of each dish data subset based on a search priority function and the data volume; The search priority function is: Among them, represents the search priority of the subset of dish data , represents the data volume of the data subset , N represents the total data volume of all subsets of dish data, represents the historical search frequency of the data subset , F represents the total global search frequency, and α represents the weight coefficient; Annotating all dish data subsets based on the search priority; In response to receiving input information, searching for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset; The multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes; The constructing a multimodal knowledge graph based on the multimodal dish data includes: Performing correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes in the multimodal dish data to obtain a correlation coefficient between each two entity nodes; Generating a weighted relationship chain for each two entity nodes based on the correlation coefficient.

2. The dish analysis method based on the multi-modal knowledge graph according to claim 1, characterized in that The matching dish image data and dish text data, and constructing multimodal dish data based on the matched data includes: Extracting the global semantic feature and local semantic feature of the dish image data; Matching based on the global semantic feature and the dish text data to obtain a first text description that successfully matches the dish image data; Matching based on the local semantic feature and the dish text data to obtain a second text description that successfully matches the dish image data; Constructing multimodal dish data based on the dish image data, the first text description, and the second text description.

3. The dish analysis method based on a multi-modal knowledge graph according to claim 2, wherein The matching based on the global semantic feature and the dish text data to obtain a first text description that successfully matches the dish image data includes: Calculating a first matching degree between the global semantic feature and the dish text data; If the first matching degree is greater than or equal to a first matching threshold, it is determined that the dish text data successfully matches the global semantic feature, and the dish text data is used as the first text description.

4. The dish analysis method based on the multi-modal knowledge graph according to claim 1, characterized in that The matching dish image data and dish text data, and constructing multimodal dish data based on the matched data includes: Extracting the semantic feature of the dish text data, and dividing the dish text data into multiple dish units based on the semantic feature; each dish unit includes multiple entity data; Matching each dish unit with the dish image data, and constructing multimodal dish data based on the matched data.

5. The dish analysis method based on the multi-modal knowledge graph according to claim 4, wherein, The multiple entity data includes ingredient data, recipe data, dish data, nutrition data, and scenario data; The matching each dish unit with the dish image data, and constructing multimodal dish data based on the matched data includes: For each dish unit: Calculating a second matching degree between the dish data, the recipe data, and the dish image data; If the second matching degree is greater than or equal to the second matching threshold, the dish image data and the dish unit are successfully matched, and multimodal dish data is constructed based on the dish image data and the dish unit.

6. A dish analysis device based on a multi-modal knowledge graph, characterized in that, Including: A data construction module, configured to match dish image data and dish text data, and construct multimodal dish data based on the matched data; Construct a multimodal knowledge graph based on the multimodal dish data; the multimodal knowledge graph includes a full-link data reasoning structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain; the multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes; The data construction module is specifically configured to perform correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes in the multimodal dish data to obtain a correlation coefficient between each two entity nodes; generate a weighted relationship chain for each two entity nodes based on the correlation coefficient; A priority division module, configured to cluster the dish multimodal data set to obtain multiple dish data subsets; Determine the search priority of each dish data subset based on the search priority function and the data volume; The search priority function is: Among them, represents the search priority of the subset of dish data , represents the data volume of the data subset , N represents the total data volume of all subsets of dish data, represents the historical search frequency of the data subset , F represents the total global search frequency, and α represents the weight coefficient; Label all dish data subsets based on the search priority; A search analysis module, configured to, in response to receiving input information, search for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset.

7. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images