Multi-source food data fusion system based on monge-kantorovich distance

Abstract

The application relates to a multi-source food data fusion system based on a Monge-Kantorovich distance, relates to the technical field of multi-source food data fusion, and comprises a data acquisition module, a feature extraction module, a data fusion module and a fusion evaluation module. The data acquisition module collects and integrates various food data, including food image data and text data of food component analysis. The feature extraction module maps different forms of food data to a hidden space through a neural network for unified digital description. The data fusion module adopts a Monge-Kantorovich optimal transport distance to fuse different features in the hidden space. The fusion evaluation module uses the data set formed after data fusion as a training set to train a food image recognition classification network, uses a Food2k data set as a test, and evaluates the data fusion capability according to the test result. The unified mechanism is used to express the multi-source food data, and the problem of how to effectively fuse and represent the correlation and complementarity of different types of data description of the same food is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-source food data fusion, and particularly relates to a multi-source food data fusion system and method based on Monge-Kantorovich distance and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, more and more enterprises begin to pay attention to and utilize food multi-source data. Food multi-source data includes images, texts and other formats, and different data sources contain rich information.

[0003] Through data fusion of different data sources, more rich information and more accurate results can be obtained. For example, image data can be used to identify the type, color, shape, size and other characteristics of food, while text data can be used to obtain the name, ingredients, nutritional value and other information of food. By fusing image and text data, the characteristics and attributes of food can be more comprehensively understood, and better decisions and judgments can be made.

[0004] In the food industry, data fusion also has a wide range of applications. For example, image data and text data can be used to realize food safety detection and quality control, and by analyzing and comparing the appearance, ingredients, nutritional value and other information of food, problems existing in food can be quickly found, and then corresponding measures can be taken. In addition, data fusion technology can also be used to predict and analyze food sales trends, helping enterprises to develop more reasonable sales strategies and marketing plans.

[0005] In the prior art, most of the modeling is for single type food data (such as text or picture), and then respective cognitive description is performed. At present, there is still a lack of unified mechanism for multi-type food big data fusion representation. How to effectively fuse and represent, and solve the problem of complementary description of different types of data of the same food, is still a big difficulty. In view of this, data fusion technology combines data from multiple information sources, and uses data estimation, modeling, management and other means to integrate related data according to time or spatial information based on certain criteria, to produce more accurate and consistent description or explanation of the observed object.

[0006] Through multi-source data fusion of food, the value and utilization efficiency of data can be greatly improved, and the classification / prediction performance in the later stage can be greatly improved. However, most of the current research work on data fusion is mainly for single type data, and it is difficult to fuse multi-type food big data. Therefore, it is urgent to construct a multi-type food big data fusion model, which has important significance for the development of the food industry. SUMMARY

[0007] The application aims to provide a multi-source food data fusion system, method and storage medium based on Monge-Kantorovich distance, aiming to solve the above problems. The application realizes that multi-source food data is expressed by a unified mechanism, and solves the problem of how to effectively fuse and represent the correlation and complementarity of different types of data describing the same food.

[0008] The application is mainly realized by the following technical solutions:

[0009] On the one hand, a multi-source food data fusion system based on Monge-Kantorovich distance is provided, which includes a data acquisition module, a feature extraction module, a data fusion module and a fusion evaluation module.

[0010] The data acquisition module is used to collect and integrate various food data, including food image data and text data of food ingredient analysis.

[0011] The feature extraction module is used to map the image data and text data collected by the data acquisition module to the latent space through a neural network for unified digital description, to obtain image features and text features.

[0012] The data fusion module is used to fuse the two different types of features, i.e. image features and text features, obtained by the feature extraction module, through a Monge-Kantorovich optimal transport algorithm, to form a data set after fusion.

[0013] The fusion evaluation module is used to train a food image recognition classification network based on the data set formed after data fusion by the data fusion module, and uses Food2k data set as a test, and evaluates the data fusion capability according to the test result.

[0014] On the other hand, a multi-source food data fusion method based on Monge-Kantorovich distance is provided, which is applied to the multi-source food data fusion system based on Monge-Kantorovich distance as claimed in any one of claims 1-8, and the method comprises:

[0015] Collect and integrate various food data, including image data and text data of food.

[0016] The layered attention network is used to obtain image features of image data through the ResNet50 network, and the layered attention network is used to obtain text features of text data through the TextCNN network, then the two different types of features are mapped to the same hidden space representation based on dynamic weight, the self-attention module and the component attention module are used to transform multiple feature maps of different sizes, and the cross-attention module is used to fuse and map the feature maps of different levels layer by layer, and map them to the same hidden space representation.

[0017] The Monge-Kantorovich optimal transport algorithm is used to fuse two different types of features in the same hidden space, the Monge loss is defined to calculate the minimum distance between the two feature distributions, the Euclidean function is used to calculate the distance, the Monge loss is used to update the weight of the feature extraction network, and the optimal matching between the two different types of features is calculated by minimizing the Monge loss, and the data set after fusion is formed;

[0018] The data set formed after data fusion is used as a training set to train a food image recognition classification network, and the Food2k data set is used as a test, and the data fusion capability is evaluated according to the test result.

[0019] On the other hand, a computer readable storage medium is provided, and the readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to realize the above method.

[0020] The technical scheme provided by the application has at least the following beneficial effects:

[0021] (1) A multi-source food data fusion system based on Monge-Kantorovich distance is proposed, which can effectively solve the problem of unified digital description of different forms of food data and integration of food data from different sources, that is, the data from multiple information sources are combined through data fusion technology, thereby improving the classification / prediction performance in the later stage of the task, which has important significance for food safety and food evaluation.

[0022] (2) For different types of image features and text features obtained by the feature extraction module, the Monge-Kantorovich optimal transport algorithm adopted by the data fusion module is used to fuse two different types of features in the same hidden space. The biggest difference between the method of the present application and the prior art is that after obtaining the point pair with the shortest distance between the text features and the image features, the two point pairs are hidden in the hidden space, avoiding the conversion of all points in one set to one point in another set, ensuring the matching of each text feature and image feature data point, preventing the model from paying too much attention to single-class specific samples, so as to match only the same instances and reduce the model collapse. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 The structure schematic diagram of the multi-source food data fusion system based on Monge-Kantorovich distance provided by an exemplary embodiment of the present application is shown.

[0025] Figure 2 The structure schematic diagram of the hierarchical attention network adopted by the feature extraction module provided by an exemplary embodiment of the present application is shown.

[0026] Figure 3 The structure schematic diagram of the component attention module adopted by the hierarchical attention network for text data provided by an exemplary embodiment of the present application is shown.

[0027] Figure 4 The flowchart of the calculation of Monge-Kantorovich distance provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0029] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail with reference to the drawings.

[0030] Figure 1 A structural diagram of a multi-source food data fusion system based on Monge-Kantorovich distance provided by an example embodiment of the present application is shown. In the embodiment of the present application, a multi-source food data fusion system based on Monge-Kantorovich distance includes a data acquisition module, a feature extraction module, a data fusion module, and a fusion evaluation module.

[0031] The data acquisition module is used to integrate various food data, including food image data and text data of food ingredient analysis. In the embodiment, data is obtained through different channels, and data cleaning and preprocessing tools are used to obtain accurate and complete data sets.

[0032] The feature extraction module is used to map different forms of food data to a hidden space through a neural network to unify digital description. The data fusion module uses the Monge-Kantorovich optimal transport distance to fuse data features and text features in the hidden space, calculates the optimal matching between two different types of features by minimizing the distance loss, and uses this distance loss to update the weight of the extraction network.

[0033] The fusion evaluation module uses the data set formed after the data fusion of the data fusion module as a training set to train a food image recognition classification network, and uses the Food2k data set for testing to evaluate the data fusion capability.

[0034] In summary, the multi-source food data fusion system based on Monge-Kantorovich distance provided by the embodiment of the present application can effectively solve the problem of unified digital description of different forms of food data and integration of food data from different sources, that is, combining data from multiple information sources through data fusion technology, thereby improving the classification / prediction performance in the later stage of the task, which has important significance for food safety and food evaluation.

[0035] Figure 2 A structural diagram of a hierarchical attention network used by the feature extraction module provided by an example embodiment of the present application is shown. As shown in Figure 2 In the embodiment of the present application, the feature extraction module uses a hierarchical attention network to extract and splice features of different types of image data and text data, and then models the modalities between image data features and text data features through a cross-attention module, and maps the two different types of features, image data features and text data features, to the same hidden space.

[0036] The cross-attention module is composed of a Transformer unit, which is used for synchronous modeling of intra-modal and inter-modal relationships. The image features are passed through an average pooling layer, and the text features are passed through a one-dimensional convolution layer to obtain the final representations of the image and the text.

[0037] The hierarchical attention network uses a ResNet50 network and a channel attention module for feature extraction on image data.

[0038] The RestNet50 network includes convolution layers, batch normalization layers, Relu activation function layers, maximum pooling layers, residual blocks, and average pooling layers arranged in sequence.

[0039] The channel attention module includes maximum pooling layers, average pooling layers, fully connected layers, weight addition layers, and Sigmoid activation function layers arranged in sequence.

[0040] The hierarchical attention network uses a word embedding module, a TextCNN network, and a component attention module for feature extraction on text data.

[0041] The word embedding module is used to map each word in the text sequence to a vector through a Word2Vec network, thereby obtaining a text vector matrix.

[0042] The TextCNN network is used for text feature extraction, and includes convolution layers, pooling layers, and fully connected layers arranged in sequence.

[0043] The component attention module receives image vectors and text vectors to obtain better text feature representations. The component attention module is composed of two convolution layers, as shown in Figure 3 .

[0044] In the embodiments of the present application, for image data, the ResNet50 network is first input, and the spatial and channel information in each local receptive field in the image feature is fused through a series of convolution operations to construct information features. Then, the channel attention mechanism is used to compress the global spatial information and give each channel a weight, allowing different channels to assign different weights to the result.

[0045] In the embodiments of the present application, for text features, the TextCNN network is used for feature extraction. First, the text is divided into a sequence of words through the word embedding module, and the words are mapped to vectors through the pre-trained Word2Vec network, thereby obtaining a text vector matrix. Compared with one-hot encoding, Word2Vec can obtain low-dimensional dense word vectors, thereby solving the problem of high-dimensional sparse word vectors and reducing the storage and computing overhead.

[0046] Then the word vectors are input into the TextCNN. Since each row vector of the input represents a word, the width of the convolution kernel is kept consistent with the dimension of the word vector, so that the TextCNN not only considers the word meaning when performing convolution on the input sentence, but also considers the relevance of the word order and its context. Compared with the Transformer feature extractor, the TextCNN network structure is simple, the number of parameters is small, the amount of calculation is small, and the training speed is fast.

[0047] In the embodiments of the present application, considering that the literal description of different foods is different and contains different numbers of literal ingredients, a pooling operation is used to embed and aggregate multiple literal ingredients into a single vector to alleviate the change in length. First, average pooling is used to uniformly fuse ingredient embeddings. The average pooling expression is as follows:

[0048]

[0049] where |X i | represents the length of the literal ingredient of the i-th image. However, the average pooling lacks flexibility in dynamically adjusting the ingredient weight. Each food has main ingredients and ingredients, and this flexibility is particularly important. For example, there are ingredients such as sugar and salt in most foods, and these ingredients cannot be used to distinguish different foods. Therefore, we believe that the embedding of the ingredient should be combined with the adaptive weight to capture the relationship between the literal ingredient and the food image. The adaptive fusion expression is as follows:

[0050]

[0051] Since the attention mechanism allows different parts to make different contributions when they are compressed into a single representation, the present embodiment proposes an ingredient-level attention module to model the adaptive weight Specifically, as Figure 3 shown, we use a two-layer network to calculate where the literal feature and the image feature m i are input, and the calculation method is as follows:

[0052]

[0053]

[0054] where i represents the number of images, m i represents the feature vector of the i-th image, W1, W2 and b1 represent the trained weights. The ingredient attention module calculates the adaptive weight by introducing the image feature to calculate the adaptive weight with the text feature, thereby guiding the aggregation of the text feature vector.

[0055] By modeling different types of data respectively, different types of features are obtained. Considering that these different types of features do not have a unified representation, the embodiment will splice the image features and text features extracted respectively, and then input them into a cross-attention module. The cross-attention module is composed of a Transformer unit, which can model the intra-modal and inter-modal relationships synchronously. Then the image features pass through an average pooling layer, and the text features pass through a one-dimensional convolution layer to obtain the final representation of the image and the text. At this point, the feature extraction module extracts different types of features and maps them to the same hidden space.

[0056] In the embodiment of the application, for different types of image features and text features obtained by the feature extraction module, the Monge-Kantorovich optimal transport algorithm adopted by the data fusion module is used to fuse the two different types of features in the same hidden space.

[0057] In the prior art, such as WGAN, LSGAN, MMDGAN, and RaGAN, multi-modal data is modeled and mapped to a hidden space, and the distance between different data features is calculated. Currently, the distance formula includes the Wasserstein distance, the least square distance, the maximum mean difference distance, and the relativistic average distance. However, these distance calculation methods can cause problems such as gradient explosion, gradient zero, and slow model training speed, which can cause the model to collapse. Therefore, the embodiment of the application proposes to use the Monge-Kantorovich distance to fuse multi-source food data, which is expressed as the Monge problem of optimal mapping. First, we express two discrete measures as:

[0058]

[0059]

[0060] where δ xi represents the Dirac measure at x, and a represents the weight. The Monge problem needs to find a mapping T to push a to β, that is, T: X = [x 1 ,…,x n ]→Y = [y 1 ,…,y m ], which can be expressed as T # a = β, where T # represents the measure transformation, and the mapping must be surjective. The Monge problem shows that the mapping should minimize the transportation cost between the two measures. Since in the embodiment, the data points in the X and Y domains are the same. In this way, we have n = m, and the weights a i or b j are consistent. In this way, the Monge problem is equivalent to the optimal mapping problem.

[0061] Therefore, in the embodiment, the shortest distance between the hidden space text feature X = [x 1 ,…,x k ] and the image feature Y = [y 1 ,…,y k ] is measured by using the optimal mapping, and the target is to minimize the transportation cost. Specifically, the optimal mapping will visit each mapped data point and consider the shortest distance between x i and y j as the optimal transportation cost. Unlike the Wasserstein distance, in the embodiment of the application, firstly, T(x i ) = y j is reserved; then, the pair of points x i and y j in the hidden space is hidden, and the shortest distance between the remaining points in X and Y is calculated, as shown in Figure 4 , which avoids the concentration of data points in X on one point in Y, which guarantees the matching of each text feature and image feature data point, prevents the model from paying too much attention to a specific sample of a single class, and thus only matches the same instance, reducing the model collapse. The distance between two points is calculated one by one until all points are processed. Finally, the total shortest distance is considered as the optimal transportation loss, defined as the Monge loss, and the expression is as follows:

[0062]

[0063] Wherein, i, j ∈ [1, k], T # represents the mapping between the two distributions X and Y, and c(T(x i ), x i ) represents the optimal transportation loss when T(y) = x.

[0064] The Monge loss is used to update the weight of the feature extraction network, and the Monge loss is used as the minimum value of the distance between the image feature and the text feature distribution, and the Euclidean function is used to calculate the distance.

[0065] In summary, the hierarchical attention network is designed to map different features to the same hidden space representation based on dynamic weights. Through the self-attention module and the component attention module, multiple feature maps of different sizes are transformed, and through the cross-attention module, the feature maps of different levels are fused and mapped layer by layer to map them to the same hidden space representation. The attention mechanism is used to dynamically select and adjust the weights of different feature maps, and their information is fused in different proportions.

[0066] For different types of image features and text features obtained by the feature extraction module, the Monge-Kantorovich optimal transport algorithm adopted by the data fusion module is used to fuse two different types of features in the same hidden space. The biggest difference between the method of the present application and the prior art is that after obtaining the point pair with the shortest distance between the text features and the image features, the two point pairs are hidden in the hidden space, avoiding the conversion of all points in one set to one point in another set, ensuring the matching of each text feature and image feature data point, preventing the model from paying too much attention to single-class specific samples, so as to match only the same instances and reduce the model collapse.

[0067] The embodiment of the present application provides a multi-source food data fusion method flowchart based on Monge-Kantorovich distance. The method comprises the following steps:

[0068] Step S100: Collect and integrate various food data, including image data and text data of food, and use data cleaning and preprocessing tools to obtain accurate and complete data sets.

[0069] Step S200: Adopt a hierarchical attention network to obtain image features through a ResNet50 network for image data, and adopt a hierarchical attention network to obtain text features through a TextCNN network for text data, and then map the two different types of features, image features and text features, to the same hidden space representation based on dynamic weights, transform multiple different size feature maps through a self-attention module and a component attention module, and fuse and map different levels of feature maps layer by layer through a cross-attention module, and map them to the same hidden space representation. The attention mechanism is used to dynamically select and adjust the weights of different feature maps, and their information is fused in different proportions.

[0070] Step S300: Adopt the Monge-Kantorovich optimal transport algorithm to fuse two different types of features in the same hidden space, define the Monge loss for calculating the minimum value of the distance between the two feature distributions, use the Euclidean function to calculate the distance, use the Monge loss to update the weight values of the feature extraction network, and calculate the optimal matching between the two different types of features by minimizing the Monge loss, and form a data set after fusion.

[0071] Step S400: The data set formed after data fusion is used as a training set to train a food image recognition classification network, and the Food2k data set is used as a test, and the data fusion capability is evaluated according to the test result.

[0072] Specifically, the embodiments of the present application collect different food data for training and verification, mainly including: 1) recipe data and food flavor data set of eight major Chinese cuisines, including 30 flavor characteristics; 2) cuisine image (picture size 3x512x512) of foreign food image database RECIPE1M, food raw material data (picture size 3x128x128); 3) compound text information (containing 88,136 food material and compound relationships) of flavor database FOODKGD. These data basically cover different types of data such as pictures and texts.

[0073] Tables 1 and 2 show the experimental results, which are demonstrated by the following specific experiments:

[0074] The multi-source food data fusion system based on Monge-Kantorovich distance provided by the present application is used for classification test, and the test data set is RecipeNet data set. RecipeNet data set includes two different tasks (i.e. recipe category classification and recipe region prediction). For all tasks, we select 70% for training, 15% for verification, and 15% for testing. The data set contains 77,333 recipes (containing corresponding number of image and cooking introduction text data), 9,271 food raw material digital data and 1,683 flavor compound digital data. Among them, the recipe category contains 9 categories, including appetizer (Appetizer), beverage (Beverage), bread (Bread), soup (Soups), salad (Salads), dessert (Desserts), vegetable (Vegetables), main dish (Main-dish) and other (Others). The label of the region category is collected from the website Recipe1M, and the label contains 4 categories, including American (American), European (European), Asian (Asian) and Mexican (Mexican). In this experiment, in order to improve the classification accuracy, we use the fusion method based on Monge-Kantorovich optimal transportation to aggregate different morphological data belonging to the same dish, and then use the classifier to classify the aggregated data. In order to verify the effectiveness and innovation of the method, we prepare 10 benchmark models and compare them with the method proposed in this patent, and the classification results are shown in the following table.

[0075] Table 1. Recipe category classification

[0076]

[0077] Table 2. Recipe region classification

[0078]

[0079] The experimental results show that, by using the method of the present application, the accuracy of appetizers is 77.4, the accuracy of beverages is 88.1, the accuracy of breads is 78.1, the accuracy of soups is 69.5, the accuracy of salads is 72.3, the accuracy of desserts is 84.5, the accuracy of vegetables is 68.4, the accuracy of main-dishes is 69.1, and the accuracy of others is 67.5. By using the method of the present application, the accuracy of American is 72.9, the accuracy of European is 67.2, the accuracy of Asian is 54.3, and the accuracy of Mexican is 63.9.

[0080] In summary, the method of the present application achieves the best classification results in all models, showing the innovation of the method.

[0081] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-source food data fusion system based on Monge-Kantorovich distance, characterized in that, It includes a data acquisition module, a feature extraction module, a data fusion module, and a fusion evaluation module; The data acquisition module is used to collect and integrate various food data, including food image data and text data of food composition analysis. The feature extraction module is used to map the image data and text data collected by the data acquisition module into the latent space through a neural network for unified digital description, thereby obtaining image features and text features. The data fusion module is used to fuse two different types of features in the same latent space based on image features and text features obtained from the feature extraction module, using the Monge-Kantorovich optimal transport algorithm to form a dataset. The fusion evaluation module is used to train a food image recognition and classification network based on the dataset formed by data fusion in the data fusion module, and uses the Food2k dataset as a test set to evaluate the data fusion capability based on the test results. The feature extraction module employs a hierarchical attention network to extract and concatenate features from different types of image and text data. Then, a cross-attention module models the modalities of image and text features, mapping these two different types of features to the same latent space. The cross-attention module, composed of Transformer units, synchronously models intramodal and intermodal relationships. Image features are processed through an average pooling layer, and text features through a one-dimensional convolutional layer to obtain the final representations of the image and text. The hierarchical attention network uses a ResNet50 network and a channel attention module for feature extraction from image data. The ResNet50 network includes, in sequence, convolutional layers, batch normalization layers, ReLU activation function layers, max pooling layers, residual blocks, and average pooling layers. The channel attention module includes, in sequence, max pooling layers, average pooling layers, fully connected layers, weighted summation layers, and sigmoid activation function layers. The data fusion module measures text features in the latent space through optimal mapping during training. and image features The shortest distance between them will and The shortest distance between them is considered the optimal transportation cost. According to the Monge-Kantorovich optimal transportation algorithm, the following is obtained: and After finding the shortest distance between the two points, first, retain... Then, hide from the hidden space. and For these two point pairs, calculate the shortest distance between the remaining points in X and Y. Finally, consider the total shortest distance as the optimal transportation loss, and define the Monge loss as follows: Where i, j∈[1, k], This represents the mapping between two distributions, X and Y. Indicates when The optimal transportation loss at that time.

2. The multi-source food data fusion system based on Monge-Kantorovich distance according to claim 1, characterized in that, The hierarchical attention network uses a word embedding module, a TextCNN network, and a component attention module to extract features from text data. The word embedding module is used to map each word in the text sequence into a vector through the Word2Vec network, thereby obtaining a text vector matrix; The TextCNN network is used for text feature extraction. The TextCNN network includes convolutional layers, pooling layers and fully connected layers arranged in sequence. The component attention module receives image vectors and text vectors to obtain better text feature representations. The component attention module consists of two convolutional layers.

3. The multi-source food data fusion system based on Monge-Kantorovich distance according to claim 2, characterized in that, The component attention module introduces image features to calculate adaptive weights with text features, thereby guiding the aggregation of text feature vectors, where the text features are represented as follows: k represents the number of components in the text, and d represents the dimension of each component vector after mapping by TextCNN; the aggregation operation is represented as follows: ， ， in, Indicates the image number. Indicates the first Feature vectors of an image , and This represents the weights used in training.

4. The multi-source food data fusion system based on Monge-Kantorovich distance according to claim 1, characterized in that, Monge loss is used to update the weights of the feature extraction network. Monge loss is used to calculate the minimum distance between the distributions of image features and text features, and the distance is calculated using the Euclidean function.

5. A method for fusing multi-source food data based on Monge–Kantorovich distance, wherein the method is applied to the multi-source food data fusion system based on Monge–Kantorovich distance as described in any one of claims 1-4, characterized in that, The method includes: Collect and integrate various food data, including food image data and text data; A hierarchical attention network is used to obtain image features from image data through a ResNet50 network, and a hierarchical attention network is used to obtain text features from text data through a TextCNN network. Then, these two different types of features, image features and text features, are mapped to the same latent space representation based on dynamic weights. Multiple feature maps of different sizes are transformed through self-attention modules and component attention modules, and feature maps of different levels are fused and mapped layer by layer through cross-attention modules, mapping them to the same latent space representation. The Monge–Kantorovich optimal transport algorithm is used to fuse two different types of features in the same latent space. The Monge loss is defined to calculate the minimum distance between two feature distributions. The Euclidean function is used to calculate the distance. The Monge loss is used to update the weights of the feature extraction network. The optimal match between the two different types of features is calculated by minimizing the Monge loss. The fused data forms the dataset. The dataset formed by data fusion is used as the training set to train a food image recognition and classification network, and the Food2k dataset is used as the test set to evaluate the data fusion capability based on the test results.

6. A computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, characterized in that, The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the method as described in claim 5.

Images