Dish analysis method and device based on multi-modal knowledge graph, equipment and medium

Abstract

The invention provides a dish analysis method and device based on a multi-modal knowledge graph, equipment and a medium, and belongs to the technical field of data processing.The method comprises the steps that dish image data and dish text data are matched, and multi-modal dish data are constructed based on the matched data; constructing a multi-modal knowledge graph based on the multi-modal dish data; the multi-modal knowledge graph comprises a full-link data reasoning structure with pairwise intercommunication among a plurality of entity nodes, and every two entity nodes correspond to one weighting relation chain; clustering the dish multi-modal data set to obtain a plurality of dish data subsets; marking the search priority of each dish data subset based on the data volume; and in response to receiving the input information, searching target dish information from the multi-modal knowledge graph based on the input information and the search priority of each dish data subset. According to the invention, accurate matching between the local area of the image and the food material in dish analysis can be realized, and the fine-grained dish analysis capability is improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of data processing, and more specifically, relates to a dish analysis method, device, equipment, and medium based on a multimodal knowledge graph. Background Art

[0002] With the digital transformation of the catering industry, food ingredient detection technology based on image recognition has been widely used in fields such as catering recommendations. Existing methods mainly rely on the extraction of overall image features, such as extracting global semantic features, for simple overall positioning and classification of food ingredients. In this context, some solutions combine text descriptions to improve detection accuracy. In addition, the introduction of knowledge graph technology further integrates structured information such as food ingredient attributes and cuisine classifications, improving the comprehensiveness of dish analysis.

[0003] However, the existing dish analysis based on knowledge graphs is limited to the representation learning of a single dimension of food, relying on overall image features, with insufficient fine-grained food ingredient analysis capabilities, and it is difficult to handle food ingredient occlusion and fragmentation problems in complex dishes, resulting in inaccurate recommendation results and inability to meet the diverse needs of users. For example, the matching relationship between local regions of an image and food ingredients is blurred, prone to false correlations, and the accuracy is low. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a dish analysis method, device, equipment, and medium based on a multimodal knowledge graph to achieve precise matching between local regions of an image and food ingredients in dish analysis and improve the fine-grained dish analysis ability.

[0005] In the first aspect of the embodiments of the present disclosure, a dish analysis method based on a multimodal knowledge graph is provided, including: Matching dish image data and dish text data, and constructing multimodal dish data based on the matched data; constructing a multimodal knowledge graph based on the multimodal dish data; the multimodal knowledge graph includes a full-link data inference structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain; Clustering the dish multimodal data set to obtain multiple dish data subsets; annotating the search priority of each dish data subset based on the data volume; In response to receiving input information, searching for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset.

[0006] In the second aspect of the embodiments of the present disclosure, a dish analysis device based on a multimodal knowledge graph is provided, including: A data construction module, configured to match dish image data and dish text data, construct multimodal dish data based on the matched data; construct a multimodal knowledge graph based on the multimodal dish data; the multimodal knowledge graph includes a full-link data inference structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain; A priority division module, configured to cluster the dish multimodal data set to obtain multiple dish data subsets; label the search priority of each dish data subset based on the data volume; A search and analysis module, configured to, in response to receiving input information, search for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset.

[0007] In a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above-mentioned dish analysis method based on a multimodal knowledge graph are implemented.

[0008] In a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned dish analysis method based on a multimodal knowledge graph are implemented.

[0009] The beneficial effects of the dish analysis method, device, equipment, and medium based on a multimodal knowledge graph provided by the embodiments of the present disclosure are as follows: By deeply integrating image and text data, the present disclosure constructs a three-dimensional knowledge network including visual features and semantic information, significantly improving the accuracy and efficiency of dish analysis.

[0010] On the one hand, the multimodal data fusion of the present disclosure effectively solves the problem of limited single-modal information. Through the cross-validation of image local features and ingredient text descriptions, accurate parsing of fine-grained attributes is achieved. The full-link inference structure supports complex relationship reasoning, such as the association mining of "ingredient - cooking technique - flavor feature", providing a decision basis for scenarios such as intelligent recipe generation and nutritional calculation.

[0011] On the other hand, the dynamic priority clustering mechanism of the present disclosure enables high-frequency dish data to respond quickly. Combining the weight decay algorithm of the weighted relationship chain, the retrieval efficiency is improved while maintaining the analysis depth. The present disclosure supports cross-modal semantic alignment. In the case of fuzzy query scenarios, missing information is automatically completed through multiple rounds of iterative reasoning, promoting the evolution of dish analysis from simple classification to knowledge reasoning. Description of the Drawings

[0012] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 It is a schematic flowchart of a dish analysis method based on a multi-modal knowledge graph provided by an embodiment of the present disclosure; Figure 2 It is a schematic diagram of the actual application process of a dish analysis method based on a multi-modal knowledge graph provided by an embodiment of the present disclosure; Figure 3 It is a structural block diagram of a dish analysis device based on a multi-modal knowledge graph provided by an embodiment of the present disclosure; Figure 4 It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0014] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present disclosure. However, those skilled in the art should clearly understand that the present disclosure can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present disclosure.

[0015] To make the purpose, technical solutions, and advantages of the present disclosure clearer, the following will be described through specific embodiments in conjunction with the accompanying drawings.

[0016] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of a dish analysis method based on a multi-modal knowledge graph provided by an embodiment of the present disclosure, and this method may include S101 to S103.

[0017] S101: Match the dish image data and the dish text data, and construct multi-modal dish data based on the matched data. Construct a multi-modal knowledge graph based on the multi-modal dish data. The multi-modal knowledge graph includes a full-link data reasoning structure in which multiple entity nodes are interconnected pairwise, and each two entity nodes correspond to a weighted relationship chain.

[0018] In this embodiment, the dish image data includes an image set that integrates visual information such as the appearance, color, and shape of the dish. The dish text data may include information described in words such as dish attributes, ingredients, cooking methods, nutritional components, applicable populations, taboos, related diseases, and suitable eating scenarios. The multimodal dish data refers to the comprehensive data formed by matching and integrating images and texts, with the visual and semantic information aligned. The multimodal knowledge graph is a graph-structured knowledge base that integrates dish images, dish texts, and structured data. The multimodal knowledge graph contains entity nodes and relationship edges (i.e., weighted relationship chains).

[0019] Entity nodes are the basic components in the multimodal knowledge graph and can represent various entities in the real world. For example, the entity node types can include dishes, diseases, recipes, ingredients, nutrition, and scenarios, etc. Among them, scenarios refer to suitable eating scenarios, applicable populations, etc.

[0020] The weighted relationship chain is the relationship bond connecting two entity nodes in the knowledge graph, and a certain weight, i.e., edge weight, is assigned to this relationship to quantify the association strength between the two entity nodes. The edge weight can be calculated based on the co-occurrence frequency, matching degree, semantic similarity, etc.

[0021] The full-link data inference structure refers to a complete data structure formed by the mutual connection and association of all entity nodes in the multimodal knowledge graph. The full-link data inference structure allows starting from any entity node and reaching any other entity node through a series of relationship chains, thereby realizing comprehensive data inference and information retrieval. The full-link data inference structure supports complex logical inferences based on multimodal data. For example, according to the ingredients and scenarios input by the user, it can infer suitable dishes and recipes.

[0022] In this embodiment, by integrating the visual features of the dish image and the semantic information of the text data, a multimodal association relationship is established, including: extracting features from the image and text respectively, achieving cross-modal matching through similarity calculation, and forming multimodal dish data containing visual-semantic dual information; extracting entity features such as dishes, ingredients, nutritional components, etc. and relationship attributes such as "contains", "belongs to", "taboo", etc. from the multimodal data. Based on the correlation between entities, weights are assigned to the relationship chains to construct a knowledge graph that supports full-link inference.

[0023] Exemplarily, use OpenCV to crop (retain the main area), scale (unify the size to 224×224), and denoise (Gaussian filtering) the dish image to improve the quality of feature extraction. Extract global visual features such as ingredient distribution, color distribution, and overall shape based on the convolutional neural network ResNet50, and then combine local descriptors, such as using SIFT / SURF to extract ingredient textures, tableware details, etc., to jointly generate an image feature vector.

[0024] Use NLP tools such as Jieba word segmentation and spaCy named entity recognition to extract entity features from dish text data, and construct text semantic vectors using TF-IDF or BERT embeddings. Match the global features of the image with the text semantic vectors through cosine similarity to screen the initially matched texts; further, generate "image-text" matching pairs through the fine-grained association of local features with the text. Integrate the matched image features and text entities into multi-modal data entries.

[0025] Exemplarily, extract core entity types from multi-modal data: dish nodes, ingredient nodes, nutrition nodes, disease nodes, scenario nodes, etc. Attach attributes to each node. For example, dish nodes include "cuisine", "taste", "preparation time", and ingredient nodes include "origin", "season", "calorie".

[0026] Establish various basic relationship types such as "dish-ingredient" inclusion relationship, "ingredient-nutrition" attribute relationship, "disease-taboo" association relationship, etc. Count the co-occurrence times of "Kung Pao Chicken" and "chicken" to generate the weight of the "inclusion" relationship. Calculate the semantic distance between "high fat" and "hyperlipidemia" through Word2Vec to generate the weight of the "taboo" relationship.

[0027] Use the Neo4j graph database to store entities and relationships, construct a radial graph centered on dishes, and support multi-hop queries such as "dish→ingredient→nutrition→disease". Eliminate duplicate nodes through entity alignment algorithms such as TF-IDF-based synonym table matching to ensure the consistency of the graph. Ensure that there is at least one inference path between any two nodes. Dynamically adjust the weight of the relationship chain through user search logs to enhance the inference efficiency.

[0028] Through the above steps, a complete technical link covering "image-text matching→multi-modal data construction→knowledge graph empowerment" can be formed, providing a structured data basis for subsequent intelligent analysis.

[0029] S102: Cluster the dish multi-modal dataset to obtain multiple dish data subsets. Label the search priority of each dish data subset based on the data volume.

[0030] In this embodiment, the dish data subsets are obtained by dividing the multi-modal dataset into several subsets with similar features through a clustering algorithm. The dishes within each subset have high consistency in image vision, text semantics, or structured attributes. The search priority is the priority retrieval order given to them during the search according to the scale of the data subset or user high-frequency requirements. The subsets with large data volume or high frequency are preferentially matched to improve the query efficiency.

[0031] In this embodiment, the dish data is clustered through multimodal feature fusion to form subsets with semantic similarity; a priority function is designed based on the subset data volume, and by annotating priorities for each subset of dish data, it is ensured that subsets of high-frequency and large-scale data are preferentially retrieved during user queries, reducing the computational overhead.

[0032] Exemplarily, the K-means++ or DBSCAN algorithm is adopted to determine the optimal number of clusters K according to the elbow method. The cosine similarity is used to calculate the distance between multimodal vectors to ensure that visual and semantic similarity are measured simultaneously. The multimodal feature matrix is input into the clustering model to obtain the cluster labels of each data point. The data volume within each cluster is counted to generate subset metadata including the cluster center, entity label, and data volume.

[0033] After clustering, multiple subsets of dish data are obtained. The priority scores of each subset are calculated, and a target search sequence is generated in descending order. For example, high-priority subsets preferentially include "home-cooked dishes" and "popular dishes". The priority metadata is stored in the knowledge graph and associated with entity nodes. When the user inputs a query, the entity nodes within the high-priority subsets are preferentially retrieved. When the user does not specify the dish type, the search defaults to starting from the "home-cooked dishes" subset. Additionally, the subset data volume needs to be recalculated regularly. When the data volume of a certain subset increases by more than a preset ratio, its priority is automatically increased to ensure that the search strategy is dynamically optimized with the change of data distribution.

[0034] S103: In response to receiving the input information, search for target dish information from the multimodal knowledge graph based on the input information and the search priorities of each subset of dish data.

[0035] In this embodiment, the input information refers to the query content input by the user, which can include audio question information, text questions, or dish pictures, etc., for triggering the knowledge graph search. The input information can also include the user's historical relevant data obtained by the system. The target dish information refers to the dish details matching the input information obtained through the search, which can include multi-dimensional data such as ingredients, nutritional components, suitable populations, cooking steps, etc., and can also generate a visual complete reasoning process.

[0036] Such as Figure 2As shown, exemplarily, the user inputs: the user inputs a question and a picture, for example: "Is this dish suitable for people with three highs?" plus a picture of the dish. Entity recognition intent analysis and picture retrieval: Use a large model to recognize the key entities in the user input question, such as "people with three highs". Conduct intent analysis on the entities to match similar nodes in the knowledge graph, such as "hyperlipidemia", "hypertension", and "hyperglycemia". For the picture retrieval process, it includes: Picture preprocessing: Preprocess the picture uploaded by the user, including operations such as resizing the picture, denoising, and cropping, to ensure picture quality and consistency. Feature extraction: Use deep learning models such as convolutional neural networks to extract features from the picture. The convolutional neural network performs convolutional operations on the picture through multiple convolutional kernels to extract features at different levels, such as edges, textures, and shapes. Feature matching and similarity calculation: Match the extracted picture features with the features of the dish category pictures in the knowledge graph. Determine the most similar dish picture by calculating cosine similarity, Euclidean distance, etc. between the features.

[0037] Exemplarily, in this embodiment, the input information is parsed: For text information, large models such as BERT can be used to perform entity recognition on the input information, extract keywords, and determine the type of user needs to generate a query target. For image data, a convolutional neural network (CNN) can be used to extract global visual features and local features to generate feature vectors. Generate query conditions based on the obtained keywords, entity features, or feature vectors, etc., and determine a subset of dish data that meets the conditions to narrow the retrieval scope. Retrieve the multi-modal knowledge graph according to the priority of the dish data subset from high to low, find associated nodes through breadth-first search, and perform logical operations on all the retrieved results, such as merging the common taboo nodes ("high salt") of "diabetes" and "hypertension".

[0038] Generate an inference path, such as "user input → entity node → subset priority → matching dish", and label the knowledge source. Display the inference trajectory in the form of a mind map, such as "diabetes → taboo high sugar → matching dish 'cold spinach' → low GI value", and additionally supplement and label the dish attributes and associated scenarios. If the retrieval results are insufficient, prompt the user to supplement the input through a large model, such as "Do you need to query local special dishes?".

[0039] It can be concluded from the above that in this embodiment, by deeply integrating image and text data, a three-dimensional knowledge network containing visual features and semantic information is constructed, significantly improving the accuracy and efficiency of dish analysis.

[0040] On the one hand, the multi-modal data fusion of this embodiment effectively solves the problem of limited single-modal information. Through the cross-verification of the local features of the image and the text description of the ingredients, the accurate parsing of fine-grained attributes is realized. The full-link inference structure supports complex relationship reasoning, such as the association mining of "ingredients - cooking techniques - flavor characteristics", providing a decision-making basis for scenarios such as intelligent recipe generation and nutritional calculation.

[0041] On the other hand, the dynamic priority clustering mechanism of this embodiment enables the high-frequency dish data to respond quickly. Combining with the weight decay algorithm of the weighted relationship chain, while maintaining the analysis depth, the retrieval efficiency is improved. This disclosure supports cross-modal semantic alignment. In the fuzzy query scenario, through multi-round iterative reasoning, the missing information is automatically completed, promoting the evolution of dish analysis from simple classification to knowledge reasoning.

[0042] In an embodiment of the present disclosure, the dish image data and the dish text data are matched, and multi-modal dish data is constructed based on the matched data, including: Extract the global semantic features and local semantic features of the dish image data.

[0043] Match based on the global semantic features and the dish text data to obtain the first text description that matches the dish image data successfully.

[0044] Match based on the local semantic features and the dish text data to obtain the second text description that matches the dish image data successfully.

[0045] Construct multi-modal dish data based on the dish image data, the first text description, and the second text description.

[0046] In this embodiment, matching based on the global semantic features and the dish text data to obtain the first text description that matches the dish image data successfully includes: Calculate the first matching degree between the global semantic features and the dish text data.

[0047] If the first matching degree is greater than or equal to the first matching threshold, it is determined that the dish text data matches the global semantic features successfully, and the dish text data is used as the first text description.

[0048] In this embodiment, matching based on the local semantic features and the dish text data to obtain the second text description that matches the dish image data successfully includes: Calculate the third matching degree between the local semantic features and the dish text data.

[0049] If the third matching degree is greater than or equal to the third matching threshold, it is determined that the dish text data matches the local semantic features successfully, and the dish text data is used as the second text description.

[0050] In this embodiment, the global semantic features may include abstract features representing the overall visual semantics of the dish image, such as macroscopic visual information like color distribution, shape contour, composition structure, etc., which are used to describe the overall category or main attributes of the dish. The local semantic features may include the detailed visual features of key regions in the dish image, such as ingredient textures, specific ingredients, cooking states, etc., which are used to precisely match the specific descriptions in the text. The first matching degree refers to the similarity between the global semantic features and the text semantic vector, which is used to measure the matching degree between the overall visual of the image and the macroscopic description of the text. The third matching degree refers to the similarity between the local features and the text detail semantics, which is used to measure the similarity between the detailed visual of the image and the specific description of the text. The first matching threshold and the third matching threshold are preset similarity critical values, which are used to determine whether the global matching is successful.

[0051] In this embodiment, the overall features of the image can be extracted using a CNN, and the similarity with the global semantic vector of the text can be calculated to filter out the text that matches the macroscopic category of the image, such as macroscopic semantics like "Kung Pao Chicken", "Sichuan cuisine", "spicy", etc. The key regions such as ingredients and seasonings in the image are located through object detection, and after extracting the detailed features, they are matched with the specific descriptions in the text to supplement and refine the information. The original image data, the macroscopic descriptions obtained from the global matching, and the detailed descriptions obtained from the local matching are integrated to form structured multi-modal data containing "overall category + detailed attributes", providing hierarchical input for the knowledge graph.

[0052] Exemplarily, use OpenCV to crop the image to the main body region and preprocess the cropped image. Input the preprocessed image into the pre-trained ResNet50 model to obtain the 2048-dimensional global features output by the avg_pool layer. Reduce the dimension to 1024 dimensions through a fully connected layer, and perform L2 normalization to obtain the global feature vector v_global_image.

[0053] Use the YOLOv5s model to detect the ingredient / seasoning regions in the image and generate a list of bounding boxes: including category labels such as "chicken", "peanut", "chili pepper". Crop the sub-images within each bounding box as local feature extraction units. Extract Scale-Invariant Feature Transform (SIFT) features for each sub-image: construct a Gaussian pyramid, detect the extreme points of the Difference of Gaussians (DOG) to locate the key points; calculate the 8-direction gradient histogram centered on the key points to generate a 128-dimensional descriptor; sort by the response value and retain L = 500 strongest descriptors to form the local feature set D_local_image.

[0054] Use spaCy for named entity recognition to parse the dish text into three types of entities: global entities: dish name, cuisine, dish type; local entities: ingredients, seasonings, cooking status; attribute entities: taste, nutrition, preparation time.

[0055] Input the global entity list into the BERT-base model to obtain the 768-dimensional embedding vector v_global_text at the CLS position, which represents the overall semantics of the text. For each local entity, use the Word2Vec model to generate 300-dimensional word vectors; calculate the weighted average according to the entity occurrence frequency to obtain the set of local semantic vectors V_local_text.

[0056] In this embodiment, calculating the first matching degree between the global semantic feature and the dish text data includes: Calculate the first matching degree between the global semantic feature and the dish text data based on the global matching degree function.

[0057] The global matching degree function is:

[0058] Where Sim_global represents the first matching degree, v_global_image represents the global semantic feature, and v_global_text represents the dish text data.

[0059] In this embodiment, calculating the third matching degree between the local semantic feature and the dish text data includes: For each local semantic feature: find the nearest text vector in the dish text data.

[0060] Calculate the third matching degree between the local semantic feature and the nearest text vector based on the local matching degree function.

[0061] The local matching degree function is:

[0062]

[0063] Where Sim_local represents the third matching degree, Dist represents the Euclidean distance between the local semantic feature and the nearest text vector, represents the i-th local semantic feature, represents the i-th nearest text vector.

[0064] Through the hierarchical matching of global semantic features and local semantic features, this embodiment realizes cross-modal alignment of macro classification and micro refinement, avoids the ambiguity of single feature matching, and improves the matching accuracy. This embodiment combines the original image features, macro text descriptions, and detailed text descriptions to form a standardized data structure containing overall categories and detailed attributes, providing multi-dimensional input for the knowledge graph and supporting subsequent in-depth reasoning. This embodiment first filters out irrelevant texts through global matching, reducing the amount of local matching calculations; local matching achieves precise screening in the remaining text data, improving the matching efficiency.

[0065] In one embodiment of the present disclosure, dish image data and dish text data are matched, and multi-modal dish data is constructed based on the matched data, including: Extract the semantic features of the dish text data, and divide the dish text data into multiple dish units based on the semantic features. Each dish unit includes multiple entity data.

[0066] Match each dish unit with the dish image data, and construct multi-modal dish data based on the matched data.

[0067] In this embodiment, the multiple entity data includes ingredient data, recipe data, dish data, nutrition data, and scenario data.

[0068] Match each dish unit with the dish image data, and construct multi-modal dish data based on the matched data, including: For each dish unit: Calculate the second matching degree of the dish data, recipe data, and dish image data.

[0069] If the second matching degree is greater than or equal to the second matching threshold, the dish image data matches the dish unit successfully, and multi-modal dish data is constructed based on the dish image data and the dish unit.

[0070] In this embodiment, the dish unit is an independent information set divided from the dish text data according to semantic features, and each dish unit contains entity data such as ingredients and recipes. The second matching degree is used to measure the similarity of the dish data, recipe data, and dish image data in the dish unit. The second matching threshold is a preset critical value used to determine whether the matching is successful.

[0071] Exemplarily, semantic analysis is performed on the dish text data and divided into multiple dish units, and each unit represents the relevant information of a dish. For each dish unit, calculate the matching degree of the dish data, recipe data, and dish image data therein. If the matching degree reaches the threshold, it indicates that the dish unit matches the image data successfully, and finally the two are fused to construct multi-modal dish data.

[0072] In this embodiment, calculating the second matching degree of dish data, recipe data, and dish image data includes: Calculating the fourth matching degree between the dish data and the dish image data, and calculating the fifth matching degree between the recipe data and the dish image data.

[0073] Calculating the second matching degree based on the dish unit matching function, the fourth matching degree, and the fifth matching degree.

[0074] The dish unit matching function is:

[0075]

[0076]

[0077] Where α and β represent weight coefficients, α + β = 1, Sim_unit represents the second matching degree, Sim_dish represents the fourth matching degree, and Sim_recipe represents the fifth matching degree.

[0078] In this embodiment, the text is divided into independent dish units, and the matching degrees between the dish data and the recipe data are calculated respectively, avoiding the ambiguity of traditional single matching and improving the matching accuracy. In this embodiment, the matching priority between the dish and the recipe is dynamically adjusted through weight coefficients to meet diverse requirements. This embodiment constructs structured data containing "dish - recipe - image" to provide fine - grained input for the knowledge graph, supports cross - modal reasoning of "ingredient → cooking → nutrition", solves the problem of single dimension in the prior art, and improves the comprehensiveness of analysis. This embodiment combines hierarchical matching of image features and text semantics, and can still effectively match fuzzy input through a weighted strategy, with strong robustness.

[0079] This embodiment mainly uses text data to search for matching image data. The previous embodiment is for image data to search for matching text data. This two - way matching method can improve the integrity and accuracy of multi - modal dish data. Searching for images by text can accurately locate visual information, and searching for text by images can supplement detailed descriptions. The combination of the two can more comprehensively explore the relationships between data, provide richer and more three - dimensional dish information for users, and enhance the practicality of the knowledge graph and the effectiveness of intelligent recommendation.

[0080] In an embodiment of the present disclosure, the multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes.

[0081] Constructing a multi - modal knowledge graph based on the multi - modal dish data includes: Performing correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes in the multi - modal dish data to obtain the correlation coefficients between every two entity nodes.

[0082] Generate a weighted relationship chain for every two entity nodes based on the correlation coefficient.

[0083] In this embodiment, the food ingredient nodes may include attributes such as name, category, and edible taboos; the recipe nodes may include attributes such as cooking methods, steps, tools, and time consumption; the dish nodes may include attributes such as dish name, cuisine, flavor, and production difficulty; the nutrition nodes may include attributes such as nutrition component name, content, and health impact; the scenario nodes may include attributes such as applicable population labels, occasions, and seasons. The correlation coefficient is a numerical value that quantifies the association strength between two entity nodes, and the higher the value, the stronger the association. The weighted relationship chain refers to a directed edge connecting two entity nodes, and the edge weight is the correlation coefficient.

[0084] Exemplarily, for the food ingredient nodes: extract the name from the dish text ingredient list and the image food ingredient detection results, and supplement the nutrition attributes; For the dish nodes: extract the name from the image retrieval results and the text title, and associate attributes such as cuisine and flavor; Merge duplicate nodes through a custom synonym table and TF-IDF similarity to ensure entity uniqueness. For example, "potato" is unified as "sweet potato".

[0085] For the correlation analysis between food ingredients and dishes, and between dishes and recipes, the co-occurrence frequency method can be used to count the number of times a food ingredient appears in a dish, and calculate the normalized co-occurrence coefficient. The co-occurrence coefficient = the number of times the food ingredient appears in the dish / the total number of times the food ingredient appears.

[0086] For the correlation analysis between nutrition and scenarios, and between food ingredients and nutrition, BERT can be used to calculate the cosine similarity of entity vectors. For example, the similarity between "high fat" and "hyperlipidemia" is 0.85, which is used as the weight of the "nutrition-taboo-disease" relationship.

[0087] For the correlation analysis between diseases and scenarios, and between recipes and nutrition, strong association weights can be set according to relevant dietary guidelines. For example, the weight of "hypertension - recommended diet - low salt" = 1.0, and the weight of "high sugar - taboo - diabetes" = 0.95.

[0088] In this embodiment, a weighted relationship chain is constructed through co-occurrence frequency, semantic similarity, and domain rules to achieve cross-dimensional association of five types of entities: ingredients, recipes, dishes, nutrition, and scenarios, covering the entire "ingredient - dish - scenario" link, solving the problem of single analysis dimension in the prior art, and improving the inference accuracy. The weighted relationship chain in this embodiment supports weighted path search and can also dynamically adjust the inference path in combination with the user's historical information to make the recommendation results more in line with personalized needs. In this embodiment, a synonym table and TF-IDF are used to remove duplicates to ensure the uniqueness of entities and avoid data redundancy; the combination of domain rules and data-driven enhances the scientific basis of the knowledge graph and reduces inference deviation. The weighted relationship chain clearly marks the association strength, and the inference process is transparent, which can improve user trust.

[0089] In one embodiment of the present disclosure, the search priority of each dish data subset is marked based on the data volume, including: Determine the search priority of each dish data subset based on the search priority function and the data volume.

[0090] The search priority function is:

[0091] Wherein, represents the search priority of the dish data subset , represents the data volume of the data subset , N represents the total data volume of all dish data subsets, represents the historical search frequency of the data subset , F represents the total global search frequency, and γ represents the weight coefficient.

[0092] Mark all dish data subsets based on the search priority.

[0093] In this embodiment, the search priority function combines two factors: data volume and historical search frequency. The data volume ratio reflects the scale of the subset in the overall data, and the historical search frequency ratio reflects the degree of attention of the subset. By weighted summing the two with the weight coefficient, the search priority of each subset can be obtained. Mark and sort all subsets according to the priority. When performing dish search, search the dish data subsets in sequence according to the target search sequence, and give priority to processing subsets with high priority. To improve search efficiency and accuracy.

[0094] In this embodiment, through the weighted fusion of data volume and historical search frequency, the data scale and user preference are dynamically balanced, so that high-frequency subsets and large-scale subsets are retrieved first, improving the search efficiency.

[0095] Corresponding to the dish analysis method based on the multimodal knowledge graph in the above embodiment, Figure 3The block diagram of the dish analysis device based on the multimodal knowledge graph provided by an embodiment of the present disclosure. For ease of illustration, only the parts related to the embodiments of the present disclosure are shown. Refer to Figure 3 The dish analysis device 20 based on the multimodal knowledge graph includes: a data construction module 21, a priority division module 22, and a search and analysis module 23.

[0096] Among them, the data construction module 21 is used to match the dish image data and the dish text data, and construct multimodal dish data based on the matched data. A multimodal knowledge graph is constructed based on the multimodal dish data. The multimodal knowledge graph includes a full-link data inference structure in which multiple entity nodes communicate with each other pairwise, and each two entity nodes correspond to a weighted relationship chain.

[0097] The priority division module 22 is used to cluster the dish multimodal data set to obtain multiple dish data subsets. The search priority of each dish data subset is marked based on the data volume.

[0098] The search and analysis module 23 is used to respond to the received input information, and search for target dish information from the multimodal knowledge graph based on the input information and the search priority of each dish data subset.

[0099] In an embodiment of the present disclosure, the data construction module 21 is specifically used to extract the global semantic feature and the local semantic feature of the dish image data.

[0100] Match based on the global semantic feature and the dish text data to obtain the first text description that matches the dish image data successfully.

[0101] Match based on the local semantic feature and the dish text data to obtain the second text description that matches the dish image data successfully.

[0102] Construct multimodal dish data based on the dish image data, the first text description, and the second text description.

[0103] In an embodiment of the present disclosure, the data construction module 21 is specifically further used to calculate the first matching degree between the global semantic feature and the dish text data.

[0104] If the first matching degree is greater than or equal to the first matching threshold, it is determined that the dish text data matches the global semantic feature successfully, and the dish text data is used as the first text description.

[0105] In an embodiment of the present disclosure, the data construction module 21 is specifically further used to match the dish image data and the dish text data, and construct multimodal dish data based on the matched data, including: Extract the semantic features of the dish text data, and divide the dish text data into multiple dish units based on the semantic features. Each dish unit includes multiple entity data.

[0106] Match each dish unit with the dish image data, and construct multimodal dish data based on the matched data.

[0107] In an embodiment of the present disclosure, the multiple entity data include ingredient data, recipe data, dish data, nutrition data, and scenario data. The data construction module 21 is further specifically configured to, for each dish unit: Calculate the second matching degree of the dish data, the recipe data, and the dish image data.

[0108] If the second matching degree is greater than or equal to the second matching threshold, the dish image data matches the dish unit successfully, and multimodal dish data is constructed based on the dish image data and the dish unit.

[0109] In an embodiment of the present disclosure, the multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes. The data construction module 21 is further specifically configured to perform a correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scenario nodes in the multimodal dish data to obtain the correlation coefficients between every two entity nodes.

[0110] Generate a weighted relationship chain for every two entity nodes based on the correlation coefficients.

[0111] In an embodiment of the present disclosure, the priority division module 22 is specifically configured to determine the search priority of each dish data subset based on the search priority function and the data volume.

[0112] The search priority function is:

[0113] Wherein, represents the search priority of the dish data subset , represents the data volume of the data subset , N represents the total data volume of all dish data subsets, represents the historical search frequency of the data subset , F represents the global total search frequency, and α represents the weight coefficient.

[0114] Label all dish data subsets based on the search priority.

[0115] See Figure 4 , Figure 4 is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. As Figure 4The electronic device 300 in the present embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The above-mentioned processors 301, input devices 302, output devices 303, and memories 304 communicate with each other through a communication bus 305. The memory 304 is used to store a computer program, and the computer program includes program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. Among them, the processor 301 is configured to call the program instructions to execute the functions of each module in the above device embodiments, for example Figure 3 the functions of the data construction module 21, the priority division module 22, and the search and analysis module 23 shown.

[0116] It should be understood that in the embodiments of the present disclosure, the so-called processor 301 may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0117] The input device 302 may include a touchpad, a fingerprint acquisition sensor (for acquiring the fingerprint information and the direction information of the fingerprint of the user), a microphone, etc., and the output device 303 may include a display (such as an LCD), a speaker, etc.

[0118] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A part of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0119] In specific implementation, the processors 301, input devices 302, and output devices 303 described in the embodiments of the present disclosure may execute the implementation manners described in the first embodiment and the second embodiment of the dish analysis method based on a multimodal knowledge graph provided by the embodiments of the present disclosure, and may also execute the implementation manner of the electronic device 300 described in the embodiments of the present disclosure, which will not be elaborated herein.

[0120] In another embodiment of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, all or part of the processes in the methods of the above embodiments are implemented. It can also be completed by instructing relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0121] The computer-readable storage medium can be the internal storage unit of the electronic device in any of the foregoing embodiments, such as the hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store the data that has been output or will be output.

[0122] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0123] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described electronic devices and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0124] In several embodiments provided by this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection to each other can be an indirect coupling or communication connection through some interfaces or units, or can also be an electrical, mechanical or other form of connection.

[0125] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can also be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present disclosure.

[0126] In addition, each functional unit in various embodiments of the present disclosure can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0127] The above is only the specific implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present disclosure can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A dish analysis method based on multimodal knowledge graph, characterized in that: include: Matching dish image data and dish text data, and constructing multimodal dish data based on the matched data; Constructing a multimodal knowledge graph based on the multimodal dish data; The multimodal knowledge graph includes a full-link data reasoning structure in which multiple entity nodes are interconnected in pairs, and every two entity nodes correspond to a weighted relationship chain; Clustering the dish multimodal dataset to obtain multiple dish data subsets; Label the search priority of each dish data subset based on the data volume; In response to receiving the input information, target dish information is searched from the multimodal knowledge graph based on the input information and the search priority of each dish data subset.

2. The dish analysis method based on multimodal knowledge graph according to claim 1, characterized in that: The step of matching the dish image data and the dish text data and constructing multimodal dish data based on the matched data includes: Extract global semantic features and local semantic features of dish image data; Matching the dish text data based on the global semantic feature to obtain a first text description that successfully matches the dish image data; Matching the local semantic features with the dish text data to obtain a second text description that successfully matches the dish image data; Multimodal dish data is constructed based on the dish image data, the first text description and the second text description.

3. The dish analysis method based on multimodal knowledge graph according to claim 2, characterized in that: The matching based on the global semantic feature and the dish text data to obtain a first text description that successfully matches the dish image data includes: Calculating a first matching degree between the global semantic feature and the dish text data; If the first matching degree is greater than or equal to a first matching threshold, it is determined that the dish text data matches the global semantic feature successfully, and the dish text data is used as the first text description.

4. The dish analysis method based on multimodal knowledge graph according to claim 2, characterized in that: The step of matching the dish image data and the dish text data and constructing multimodal dish data based on the matched data includes: Extracting semantic features of dish text data, and dividing the dish text data into a plurality of dish units based on the semantic features; each dish unit includes a plurality of entity data; Each dish unit is matched with the dish image data, and multimodal dish data is constructed based on the matched data.

5. The dish analysis method based on multimodal knowledge graph according to claim 4, characterized in that: The plurality of entity data includes ingredient data, recipe data, dish data, nutritional data and scenario data; The step of matching each dish unit with the dish image data and constructing multimodal dish data based on the matched data includes: For each dish unit: Calculating a second matching degree of the dish data, the recipe data, and the dish image data; If the second matching degree is greater than or equal to a second matching threshold, the dish image data and the dish unit are successfully matched, and multimodal dish data is constructed based on the dish image data and the dish unit.

6. The dish analysis method based on multimodal knowledge graph according to claim 1, characterized in that: The multiple entity nodes include ingredient nodes, recipe nodes, dish nodes, nutrition nodes and scene nodes; The constructing a multimodal knowledge graph based on the multimodal dish data includes: Performing correlation analysis on the ingredient nodes, recipe nodes, dish nodes, nutrition nodes, and scene nodes in the multimodal dish data to obtain a correlation coefficient between every two entity nodes; A weighted relationship chain is generated for every two entity nodes based on the correlation coefficient.

7. The dish analysis method based on multimodal knowledge graph according to claim 1, characterized in that: The step of marking the search priority of each dish data subset based on the data volume includes: Determine the search priority of each dish data subset based on the search priority function and the data volume; The search priority function is: in, Represents a subset of dish data The search priority of Representing a subset of data N represents the total data volume of all dish data subsets. Representing a subset of data The historical search frequency, F represents the total global search frequency, and α represents the weight coefficient; All dish data subsets are labeled based on the search priority.

8. A dish analysis device based on a multimodal knowledge graph, characterized in that: include: A data construction module, used to match dish image data and dish text data, and construct multimodal dish data based on the matched data; Constructing a multimodal knowledge graph based on the multimodal dish data; the multimodal knowledge graph includes a full-link data reasoning structure in which multiple entity nodes are interconnected in pairs, and each two entity nodes correspond to a weighted relationship chain; A priority division module, used for clustering the dish multimodal data set to obtain multiple dish data subsets; Label the search priority of each dish data subset based on the data volume; A search analysis module is used to search for target dish information from a multimodal knowledge graph in response to receiving input information based on the input information and the search priority of each dish data subset.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images