A food safety monitoring method and system based on multimodal large model

Abstract

The present invention proposes a food safety monitoring method and system based on a multimodal large model, wherein the method comprises: collecting food information of multiple data modalities, and for food information of each data modality, using a corresponding neural network to extract features; using a self-attention mechanism to fuse the extracted features to obtain fused features; using a multi-layer perceptron to calculate a safety score based on the fused features, and obtaining the predicted future risk of the food according to the historical safety score and the fused features of the current time step; constructing a probabilistic graphical model for traceability analysis, and calculating a comprehensive risk score of the food in combination with safety assessment, predicted future risk of the food and traceability analysis results. Through this method and the corresponding system, multimodal data and multi-dimensional analysis can be comprehensively considered to help decision makers make more informed decisions.

Description

Technical Field

[0001] The invention proposes a food safety monitoring method and system based on a multimodal large model, and relates to the technical field of food safety management. Background Art

[0002] With the development of technology, traditional food monitoring methods are gradually unable to meet current needs. A single data source or a single technical method is no longer sufficient to deal with the complex problems encountered in the modern food supply chain. Therefore, it is urgent to develop a comprehensive solution that can integrate multiple data sources and technologies. Summary of the invention

[0003] The present invention provides a food safety monitoring method and system based on a multimodal large model to solve the above-mentioned problems:

[0004] The present invention proposes a food safety monitoring method based on a multimodal large model, the method comprising:

[0005] Collect food information of multiple data modalities, and use the corresponding neural network to extract features for each data modality of food information;

[0006] Use the self-attention mechanism to fuse the extracted features to obtain fused features:

[0007] The safety score is calculated using a multi-layer perceptron based on the fusion features, and the predicted future risk of the food is obtained based on the fusion features of the historical safety score and the current time step;

[0008] A probabilistic graphical model is constructed to conduct traceability analysis, and the comprehensive risk score of the food is calculated by combining the safety assessment, the predicted future risk of the food, and the traceability analysis results.

[0009] Furthermore, food information of multiple data modes is collected, and for each data mode of food information, the corresponding neural network is used to extract features, including:

[0010] Collecting food information in multiple data modalities, wherein the food information in multiple data modalities includes visual data, text data, sensor data, and supply chain data, wherein the visual data includes food images and packaging images, the text data includes ingredient lists, production dates, and batch numbers, the sensor data includes temperature, humidity, and gas concentration, and the supply chain data includes transportation routes and storage conditions;

[0011] For visual data, convolutional neural networks are used to extract features. For sensor data, long short-term memory networks are used to extract features. For text data, long short-term memory networks are used to extract features. For supply chain data, fully connected networks are used to extract features:

[0012]

[0013] Among them, X m is the original input, F m is the extracted feature.

[0014] Furthermore, the self-attention mechanism is used to fuse the extracted features to obtain fused features, including:

[0015] Define three weight matrices for each mode m , and , the weight matrix is ​​used to extract the features Convert to query vector , key vector Sum value vector :

[0016]

[0017] Using Query Vectors and key vector The attention score is calculated by taking the dot product of , and then the scale of the score is adjusted by scaling the dot product. After adjustment, the softmax function is applied to the result to obtain the final attention weight :

[0018]

[0019] Among them, d k is the dimension of the attention head;

[0020] The attention weight Apply to a vector of values By aggregating different feature representations through weighted summation, we can obtain the fusion feature :

[0021]

[0022] Furthermore, a multi-layer perceptron is used to calculate the safety score based on the fusion features, and the predicted future risk of the food is obtained based on the fusion features of the historical safety score and the current time step, including:

[0023] The fused features Through the linear transformation of the first layer, a nonlinear activation function, namely the ReLU function, is applied to the output of the linear transformation of the first layer:

[0024] Z1 = torch.matmul( , W1) + b1, A1 = torch.nn.ReLU()(Z1);

[0025] The ReLU activated data A1 is transformed linearly in the second layer, and the Sigmoid function is applied to obtain the final food safety score:

[0026] Z2 = torch.matmul(A1, W2) + b2, Safety = torch.sigmoid(Z2);

[0027] Fusion features of historical safety scores and current time step Combined, use the LSTM network to process the combined data obtained in the previous step, extract key time features, and output :

[0028]

[0029] in, is the safety score for the past k time steps, is the fusion feature of the current time step;

[0030] The output of LSTM The input is sent to a multi-layer perceptron, which is further processed to obtain the predicted future risk Rt+1: .

[0031] Furthermore, a probabilistic graphical model is constructed for traceability analysis, and the comprehensive risk score of food is calculated by combining safety assessment, predicted future risk of food and traceability analysis results, including:

[0032] Construct a supply chain probability graph model G, including nodes V, edges E, node features Fnode and edge features Fedge;

[0033] For each node we , calculate its relationship with its neighbors N ( i ) of the potential function φ;

[0034] Using the value calculated by the potential function, calculate each node we Conditional probability P(vi|G) under a given supply chain probabilistic graphical model G:

[0035] ;

[0036] Analyze the probabilities of all nodes and find the node with the maximum probability value;

[0037] Combining the results of real-time security assessment, predictive modeling and traceability analysis, a comprehensive risk score is calculated based on predefined weights:

[0038] .

[0039] The present invention proposes a food safety monitoring system based on a multimodal large model, the system comprising:

[0040] A feature extraction module is used to collect food information of multiple data modes, and for each data mode of food information, a corresponding neural network is used to extract features;

[0041] The fusion feature module is used to use the self-attention mechanism to fuse the extracted features to obtain fusion features:

[0042] The risk prediction module is used to calculate the safety score based on the fusion features using a multi-layer perceptron, and obtain the predicted future risk of the food according to the historical safety score and the fusion features of the current time step;

[0043] The module for calculating the comprehensive risk score is used to build a probabilistic graphical model for traceability analysis, and calculate the comprehensive risk score of food by combining safety assessment, predicted future risks of food and traceability analysis results.

[0044] Furthermore, the feature extraction module includes:

[0045] A multimodal data collection module is used to collect food information in multiple data modalities, wherein the food information in multiple data modalities includes visual data, text data, sensor data and supply chain data, wherein the visual data includes food images and packaging images, the text data includes ingredient list, production date and batch number, the sensor data includes temperature, humidity and gas concentration, and the supply chain data includes transportation route and storage conditions;

[0046] Use neural network to extract features. For visual data, use convolutional neural network to extract features. For sensor data, use long short-term memory network to extract features. For text data, use long short-term memory network to extract features. For supply chain data, use fully connected network to extract features:

[0047]

[0048] Among them, X m is the original input, F m is the extracted feature.

[0049] Furthermore, the self-attention mechanism is used to fuse the extracted features to obtain fused features, including:

[0050] Transformation vector module, used to define three weight matrices for each mode m , and , the weight matrix is ​​used to extract the features Convert to query vector , key vector Sum value vector :

[0051]

[0052] Get the attention weight module for using the query vector and key vector The attention score is calculated by taking the dot product of , and then the scale of the score is adjusted by scaling the dot product. After adjustment, the softmax function is applied to the result to obtain the final attention weight :

[0053]

[0054] Among them, d k is the dimension of the attention head;

[0055] Get the fusion feature module to convert the attention weight Apply to a vector of values By aggregating different feature representations through weighted summation, we can obtain the fusion feature :

[0056]

[0057] Furthermore, a multi-layer perceptron is used to calculate the safety score based on the fusion features, and the predicted future risk of the food is obtained based on the fusion features of the historical safety score and the current time step, including:

[0058] The first layer of linear transformation module is used to transform the fused features Through the linear transformation of the first layer, a nonlinear activation function, namely the ReLU function, is applied to the output of the linear transformation of the first layer:

[0059] Z1 = torch.matmul( ,W1) + b1,A1 = torch.nn.ReLU()(Z1);

[0060] The second-layer linear transformation module is used to transform the ReLU activated data A1 through the second-layer linear transformation and apply the Sigmoid function to obtain the final food safety score:

[0061] Z2 = torch.matmul(A1, W2) + b2, Safety = torch.sigmoid(Z2);

[0062] Fusion features of historical safety scores and current time step Combined, use the LSTM network to process the combined data obtained in the previous step, extract key time features, and output :

[0063]

[0064] in, is the safety score for the past k time steps, is the fusion feature of the current time step;

[0065] Get the future risk prediction module to convert the output of LSTM The input is sent to a multi-layer perceptron, which is further processed to obtain the predicted future risk Rt+1: .

[0066] Furthermore, the module for calculating the comprehensive risk score includes:

[0067] Build a supply chain probability graph model module, which is used to build a supply chain probability graph model G, including nodes V, edges E, and node features F node and edge feature F edge ;

[0068] Calculate the relationship potential function module for each node we , calculate its relationship with its neighbors N ( i ) of the potential function φ;

[0069] The conditional probability calculation module is used to calculate the value of each node using the potential function. we Conditional probability P(vi|G) under a given supply chain probabilistic graphical model G:

[0070] ;

[0071] Get the maximum probability value node module, which is used to analyze the probabilities of all nodes and find the node with the maximum probability value;

[0072] The module for calculating the comprehensive risk score is used to combine the results of real-time security assessment, prediction model and traceability analysis to calculate the comprehensive risk score according to predefined weights:

[0073] .

[0074] Beneficial effects of the present invention: Enhanced decision support. By comprehensively considering multimodal data and multidimensional analysis (safety assessment, risk prediction, traceability analysis), the technical solution provides a comprehensive food safety evaluation framework that can help decision makers make more informed decisions; Improve traceability efficiency and accuracy. Using probabilistic graphical models for traceability analysis can more quickly and accurately identify the source of the problem and its transmission path, providing a scientific basis for emergency response and prevention of food safety accidents;

[0075] Strong adaptability and scalability. This technical solution is not only applicable to the current data modality, but also easy to expand to new data types and modalities because its core mechanisms (such as self-attention and probabilistic graph models) have good versatility and adaptability;

[0076] Dynamic risk assessment capability: By combining real-time data analysis and historical data trend prediction, the solution can dynamically adjust risk assessment results to better adapt to environmental changes and data updates; improve the level of food safety assurance, and overall improve the ability to control food safety risks, reduce the occurrence of food safety accidents, and protect public health. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a schematic diagram of a food safety monitoring method based on a multimodal large model described in the present invention. DETAILED DESCRIPTION

[0078] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0079] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0081] One embodiment of the present invention is a food safety monitoring method based on a multimodal large model, the method comprising:

[0082] Collect food information of multiple data modalities, and use the corresponding neural network to extract features for each data modality of food information;

[0083] Use the self-attention mechanism to fuse the extracted features to obtain fused features;

[0084] The safety score is calculated using a multi-layer perceptron based on the fusion features, and the predicted future risk of the food is obtained based on the fusion features of the historical safety score and the current time step;

[0085] A probabilistic graphical model is constructed to conduct traceability analysis, and the comprehensive risk score of the food is calculated by combining the safety assessment, the predicted future risk of the food, and the traceability analysis results.

[0086] The working principle and effect of the above technical solution are as follows: Feature extraction, by using neural network models designed for different data modalities to extract features, the specific information carried by each modality can be effectively captured, for example, convolutional neural networks (CNN) are used to process image data, while recurrent neural networks (RNN) or Transformer models are used to process text data; Feature fusion, the fusion of features of multiple modalities is usually difficult to process directly because they may be very different in form and semantics. The self-attention mechanism can focus on the dependencies between different modalities, and by calculating the weight distribution between different features, a fusion feature that comprehensively represents the important information of each modality is generated; Multi-layer perceptron (MLP) is used to predict the safety score based on the fusion features. This score reflects the safety level of the current food information and is a comprehensive learning result based on real-time data and historical data. Risk prediction combines fusion features and historical safety scores within a certain time window, and uses time series analysis methods (such as LSTM) to predict future food safety risks; Traceability analysis analyzes the mutual influence of each node in the food supply chain and its contribution to the overall risk by constructing a probabilistic graph model, thereby calibrating the possible starting point or critical path of the problem; The results of safety assessment, future risk prediction and traceability analysis are combined to calculate the total risk score. This score reflects the contribution of each aspect to the overall risk through weight distribution. Enhanced decision support: By comprehensively considering multimodal data and multi-dimensional analysis (safety assessment, risk prediction, traceability analysis), this technical solution provides a comprehensive food safety evaluation framework that can help decision makers make more informed decisions; improve traceability efficiency and accuracy, and use probabilistic graphical models for traceability analysis to more quickly and accurately identify the source of the problem and its transmission path, providing a scientific basis for emergency response and prevention of food safety accidents;

[0087] Strong adaptability and scalability. This technical solution is not only applicable to the current data modality, but also easy to expand to new data types and modalities because its core mechanisms (such as self-attention and probabilistic graph models) have good versatility and adaptability;

[0088] Dynamic risk assessment capability: By combining real-time data analysis and historical data trend prediction, the solution can dynamically adjust risk assessment results to better adapt to environmental changes and data updates; improve the level of food safety assurance, and overall improve the ability to control food safety risks, reduce the occurrence of food safety accidents, and protect public health.

[0089] In one embodiment of the present invention, food information of multiple data modes is collected, and for each data mode of food information, features are extracted using a corresponding neural network, including:

[0090] Collecting food information in multiple data modalities, wherein the food information in multiple data modalities includes visual data, text data, sensor data, and supply chain data, wherein the visual data includes food images and packaging images, the text data includes ingredient lists, production dates, and batch numbers, the sensor data includes temperature, humidity, and gas concentration, and the supply chain data includes transportation routes and storage conditions;

[0091] For visual data, convolutional neural networks are used to extract features. For sensor data, long short-term memory networks are used to extract features. For text data, long short-term memory networks are used to extract features. For supply chain data, fully connected networks are used to extract features:

[0092]

[0093] Among them, X m is the original input, F m is the extracted feature.

[0094] The working principle and effect of the above technical solution are as follows: data collection, collecting multiple modal data related to food safety monitoring, including: visual data, such as food images and packaging images, recording the appearance and packaging conditions of food, text data including ingredient list, production date and batch number, etc., providing detailed ingredients and production traceability information of food, sensor data recording key parameters such as temperature, humidity and gas concentration during food storage and transportation; supply chain data includes transportation routes and storage conditions, giving logistics information of food from production to consumption, visual data processing uses convolutional neural network (CNN) to process image data of food and packaging, which can effectively capture the spatial hierarchy and detail features in the image; sensor and text data processing: using long short-term memory network (LSTM) to process sensor data and text data, LSTM can process time series information in data, which is particularly effective for sensor data with time dependence or serialized text description;

[0095] Supply chain data processing uses a fully connected network (FCN) to process supply chain data, thereby extracting and learning complex features related to the food supply chain. Efficient and accurate feature extraction uses a specialized network structure to extract key features for different types of data, effectively improving the accuracy and robustness of data processing. CNN can identify visual defects that may affect food safety (such as mold, packaging damage, etc.) when processing visual data; the use of LSTM enables the system to understand and predict time-based patterns, such as predicting food spoilage that may occur due to poor storage conditions; through comprehensive analysis of multimodal data, the system can provide a more comprehensive food safety assessment and increase the ability to identify potential food safety issues; real-time monitoring of sensor data can instantly warn of abnormal storage or transportation conditions and prevent possible food safety issues; analyzing supply chain data helps trace the source and flow of food, and improves the transparency and traceability of the entire supply chain; supply chain data analysis is also important for optimizing logistics, reducing waste and improving overall supply chain efficiency; this technical solution can effectively improve the monitoring efficiency and accuracy of food safety by integrating multimodal data processing technology, while optimizing supply chain management and providing strong technical support for food safety and quality control.

[0096] An embodiment of the present invention is a food safety monitoring method based on a multimodal large model, which uses a self-attention mechanism to fuse extracted features to obtain fused features, including:

[0097] Define three weight matrices for each mode m , and , the weight matrix is ​​used to extract the features Convert to query vector , key vector Sum value vector :

[0098]

[0099] Using Query Vectors and key vector The attention score is calculated by taking the dot product of , and then the scale of the score is adjusted by scaling the dot product. After adjustment, the softmax function is applied to the result to obtain the final attention weight :

[0100]

[0101] Among them, d k is the dimension of the attention head;

[0102] The attention weight Apply to a vector of values By aggregating different feature representations through weighted summation, we can obtain the fusion feature :

[0103]

[0104] The working principle and effect of the above technical solution are as follows: The above technical solution utilizes the attention mechanism, and the specific steps are as follows:

[0105] Feature transformation: For each data modality m, a pre-trained feature extraction network (such as CNN, LSTM, or fully connected network) produces a feature vector F m , through three specific weight matrices, the features F of each mode are m Convert to query vector (Query) Q m 、Key vector (Key) K m , and value vector (Value) V m , the attention score is calculated using the query vector Q m and the key vector K m The original attention score is calculated by the dot product of To adjust the score scale, apply the softmax function to normalize the adjusted scores to obtain the final attention weights, feature fusion, apply the attention weights to the value vector, and aggregate feature representations from different modalities through weighted summation. This weighted summation method allows the model to automatically adjust the contribution ratio of each modality feature according to the needs of the current task, thereby realizing dynamic fusion of features. Dynamic adjustment of feature representation. The attention mechanism allows the model to automatically adjust the weights of its feature representation based on the relevance and contribution of each modality, which means that the model can focus on more informative modalities while ignoring or reducing reliance on irrelevant or noisy modalities. Enhanced model interpretability. Attention weights provide an intuitive way to understand the importance of different modal data in specific tasks. By analyzing these weights, researchers and practitioners can better understand the decision-making process of the model. Improve the generalization ability of the model. By fusing the rich features of multimodal data, the model can capture more complex data patterns and improve the adaptability, robustness, scalability and flexibility of the model when facing diverse data. This solution embeds the attention mechanism into the feature fusion process, providing a framework for processing more types of modalities or more complex data relationships while maintaining a high degree of customizability and scalability.

[0106] One embodiment of the present invention is a food safety monitoring method based on a multimodal large model, which uses a multi-layer perceptron to calculate a safety score based on fusion features, and obtains a predicted future risk of food based on the fusion features of the historical safety score and the current time step, including:

[0107] The fused features Through the linear transformation of the first layer, a nonlinear activation function, namely the ReLU function, is applied to the output of the linear transformation of the first layer:

[0108] Z1 = torch.matmul( , W1) + b1, A1 = torch.nn.ReLU()(Z1);

[0109] The ReLU activated data A1 is transformed linearly in the second layer, and the Sigmoid function is applied to obtain the final food safety score:

[0110] Z2 = torch.matmul(A1, W2) + b2, Safety = torch.sigmoid(Z2);

[0111] Fusion features of historical safety scores and current time step Combined, use the LSTM network to process the combined data obtained in the previous step, extract key time features, and output :

[0112]

[0113] in, is the safety score for the past k time steps, is the fusion feature of the current time step;

[0114] The output of LSTM The input is sent to a multi-layer perceptron, which is further processed to obtain the predicted future risk Rt+1: .

[0115] The working principle and effect of the above technical solution are as follows: through a series of carefully designed calculation steps to predict and evaluate food safety risks, feature fusion processing, first of all, the feature F obtained by fusing multiple types of data through the model fused , which ensures that the model can integrate information from different sources; F fused After linear transformation (through weight matrix W1 and bias b1) and ReLU activation function processing, A1 is generated. ReLU activation function introduces nonlinearity to help capture complex data relationships. After ReLU activation, data A1 undergoes linear transformation again (W2 and b2 before Sigmoid function), and finally the food safety score S is obtained through Sigmoid function processing. safety, which can be regarded as a direct assessment of the current food safety status; the historical safety scores are combined with the current fusion features and processed through the LSTM network. LSTM can capture the time dependency in the data and is ideal for processing time series data. The output it generates is h t Contains the temporal dynamic information of the current and historical safety scores; finally, the output of LSTM h t It is fed into a multi-layer perceptron, which further refines the risk assessment based on the dynamic features of the input and outputs the future risk R t+1 . By integrating different data sources, this technical solution can provide a comprehensive perspective to evaluate food safety by fusing information from multiple data sources, enhancing the accuracy and robustness of the model; dynamic risk assessment, using LSTM to analyze time series data of food safety, so that the model can consider historical trends and patterns, thereby providing dynamic and forward-looking risk assessment; nonlinear representation ability, through ReLU and Sigmoid activation functions, the model can capture complex nonlinear relationships, which is crucial for understanding and predicting food safety levels, and accurate prediction, multi-layer perceptron further processes the output of LSTM, which can effectively improve the accuracy of future risk predictions and the quality of related decisions; this technical solution provides a systematic solution framework for the prediction and evaluation of food safety. By comprehensively considering multimodal data and time series information, it can effectively evaluate and predict food safety risks. This method has great potential in practical applications, especially in industries that require timely and accurate assessment of food safety.

[0116] One embodiment of the present invention is a food safety monitoring method based on a multimodal large model, which constructs a probabilistic graphical model for traceability analysis, combines safety assessment, predicted future risk of food and traceability analysis results, and calculates a comprehensive risk score of food, including:

[0117] Construct a supply chain probability graph model G, including nodes V, edges E, node features Fnode and edge features Fedge;

[0118] For each node we , calculate its relationship with its neighbors N ( i ) of the potential function φ;

[0119] Using the value calculated by the potential function, calculate each node we Conditional probability P(vi|G) under a given supply chain probabilistic graphical model G:

[0120] ;

[0121] Analyze the probabilities of all nodes and find the node with the maximum probability value;

[0122] Combining the results of real-time security assessment, predictive modeling and traceability analysis, a comprehensive risk score is calculated based on predefined weights:

[0123] .

[0124] The working principle and effect of the above technical solution are as follows: The above technical solution mainly involves building a probabilistic graph model of the supply chain, and using the model to calculate and evaluate the impact or risk of each node in the supply chain. The specific working principle is as follows: build a probabilistic graph model, by defining node V, edge E, node feature F node and edge feature F edge , create a probabilistic graphical model G representing the supply chain, for each node v i , considering its relationship with its neighbors N(i), and measuring the interaction between nodes through the potential function, which takes into account the characteristics of the node itself, the characteristics of the adjacent nodes and the characteristics of the edges between them; using the potential function to calculate the conditional probability P(vi|G) of the node under the given supply chain probability graph model G, which is achieved by summing up the exponential relationship potential functions and normalizing them (through the partition function Z); analyzing the conditional probabilities of all nodes, finding the node with the maximum probability value, combining the real-time security assessment, prediction model and traceability analysis results, using predefined weights α, β, and γ, to calculate the comprehensive risk score R total . Comprehensive risk assessment. The solution provides a comprehensive risk assessment method by comprehensively utilizing the output of graph models, real-time security assessments, and predictive models. This helps to more accurately understand and predict potential risks throughout the supply chain; quantitative analysis capabilities. The impact and role of supply chain nodes are quantified in a probabilistic form, which can more systematically and intuitively identify and quantitatively analyze key impact points or risk sources in the supply chain; dynamic monitoring and early warning. The model allows dynamic monitoring of the supply chain system and updates risk assessments based on real-time data. This not only improves the ability to respond to emergencies, but also avoids or mitigates potential problems through early warning mechanisms; enhanced decision support. By focusing on nodes with higher probability of impact, decision makers can more effectively allocate resources and formulate intervention measures, thereby optimizing the management of the entire supply chain; improved transparency and accountability tracking. Through detailed probabilistic analysis of supply chain models and related nodes, the transparency of operations is increased, which helps to track the responsibility and source of problems and ensure the traceability and clear responsibility of the supply chain; This technical solution provides an advanced method that combines probabilistic graph models and multiple data analysis methods to assess and manage risks in the supply chain. It is suitable for various industries that need to accurately monitor and manage supply chain risks, such as manufacturing, food safety, and pharmaceutical industries.

[0125] One embodiment of the present invention is a food safety monitoring system based on a multimodal large model, the system comprising:

[0126] A feature extraction module is used to collect food information of multiple data modes, and for each data mode of food information, a corresponding neural network is used to extract features;

[0127] The fusion feature module is used to fuse the extracted features using the self-attention mechanism to obtain fusion features;

[0128] The risk prediction module is used to calculate the safety score based on the fusion features using a multi-layer perceptron, and obtain the predicted future risk of the food according to the historical safety score and the fusion features of the current time step;

[0129] The module for calculating the comprehensive risk score is used to build a probabilistic graphical model for traceability analysis, and calculate the comprehensive risk score of food by combining safety assessment, predicted future risks of food and traceability analysis results.

[0130] In one embodiment of the present invention, the feature extraction module includes:

[0131] A multimodal data collection module is used to collect food information in multiple data modalities, wherein the food information in multiple data modalities includes visual data, text data, sensor data and supply chain data, wherein the visual data includes food images and packaging images, the text data includes ingredient list, production date and batch number, the sensor data includes temperature, humidity and gas concentration, and the supply chain data includes transportation route and storage conditions;

[0132] Use neural network to extract features. For visual data, use convolutional neural network to extract features. For sensor data, use long short-term memory network to extract features. For text data, use long short-term memory network to extract features. For supply chain data, use fully connected network to extract features:

[0133]

[0134] Among them, X m is the original input, F m is the extracted feature.

[0135] In one embodiment of the present invention, the fusion feature module includes:

[0136] Transformation vector module, used to define three weight matrices for each mode m , and , the weight matrix is ​​used to extract the features Convert to query vector , key vector Sum value vector :

[0137]

[0138] Get the attention weight module for using the query vector and key vector The attention score is calculated by taking the dot product of , and then the scale of the score is adjusted by scaling the dot product. After adjustment, the softmax function is applied to the result to obtain the final attention weight :

[0139]

[0140] Among them, d k is the dimension of the attention head;

[0141] Get the fusion feature module to convert the attention weight Apply to a vector of values By aggregating different feature representations through weighted summation, we can obtain the fusion feature :

[0142]

[0143] In one embodiment of the present invention, the risk prediction module includes:

[0144] The first layer of linear transformation module is used to transform the fused features Through the linear transformation of the first layer, a nonlinear activation function, namely the ReLU function, is applied to the output of the linear transformation of the first layer:

[0145] Z1 = torch.matmul( , W1) + b1, A1 = torch.nn.ReLU()(Z1);

[0146] The second-layer linear transformation module is used to transform the ReLU activated data A1 through the second-layer linear transformation and apply the Sigmoid function to obtain the final food safety score:

[0147] Z2 = torch.matmul(A1, W2) + b2, Safety = torch.sigmoid(Z2);

[0148] Extract time feature module to fuse historical safety scores with the current time step Combined, use the LSTM network to process the combined data obtained in the previous step, extract key time features, and output :

[0149]

[0150] in, is the safety score for the past k time steps, is the fusion feature of the current time step;

[0151] The output of LSTM The input is sent to a multi-layer perceptron, which is further processed to obtain the predicted future risk Rt+1: .

[0152] In one embodiment of the present invention, the module for calculating the comprehensive risk score includes:

[0153] Build a supply chain probability graph model module, which is used to build a supply chain probability graph model G, including nodes V, edges E, and node features F node and edge feature F edge ;

[0154] Calculate the relationship potential function module for each node we , calculate its relationship with its neighbors N ( i ) of the potential function φ;

[0155] The conditional probability calculation module is used to calculate the value of each node using the potential function. we Conditional probability P(vi|G) under a given supply chain probabilistic graphical model G:

[0156] ;

[0157] Get the maximum probability value node module, which is used to analyze the probabilities of all nodes and find the node with the maximum probability value;

[0158] The module for calculating the comprehensive risk score is used to combine the results of real-time security assessment, prediction model and traceability analysis to calculate the comprehensive risk score according to predefined weights:

[0159] .

[0160] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A food safety monitoring method based on a multimodal large model, characterized in that: The method comprises: Collect food information of multiple data modalities, and use the corresponding neural network to extract features for each data modality of food information; Use the self-attention mechanism to fuse the extracted features to obtain fused features; The safety score is calculated using a multi-layer perceptron based on the fusion features. The predicted future risk of food is obtained based on the historical safety score and the fusion features of the current time step, including: The fused features Through the linear transformation of the first layer, a nonlinear activation function, namely the ReLU function, is applied to the output of the linear transformation of the first layer: Z1 = torch.matmul( , W1) + b1,A1 = torch.nn.ReLU()(Z1) Among them, W1 represents the weight matrix of the first layer, and bl represents the bias term of the first layer; The ReLU activated data A1 is transformed linearly in the second layer, and the Sigmoid function is applied to obtain the final food safety score: Z2 = torch.matmul(A1, W2) + b2, Safety = torch.sigmoid(Z2) Among them, W2 represents the weight matrix of the second layer, and b2 represents the bias term of the second layer; Fusion features of historical safety scores and current time step Combined, use the LSTM network to process the combined data obtained in the previous step, extract key time features, and output : in, is the safety score for the past k time steps, is the fusion feature of the current time step; The output of LSTM The input is sent to a multi-layer perceptron, which is further processed to obtain the predicted future risk Rt+1: ; Construct a probabilistic graphical model for traceability analysis, combine safety assessment, predicted future risk of food and traceability analysis results to calculate the comprehensive risk score of food, including: Construct a supply chain probability graph model G, including nodes V, edges E, node features Fnode and edge features Fedge; For each node vi , calculate its relationship with its neighbors N ( i ) of the potential function φ; Using the value calculated by the potential function, calculate each node vi Conditional probability P(vi|G) under a given supply chain probabilistic graphical model G: Where Z is the partition function, Representation and Node Adjacent nodes, Respectively represent nodes and neighbor nodes The characteristic vector of Represents a connection node and The characteristics of the edge; Analyze the probabilities of all nodes and find the node with the maximum probability value; Combining the results of real-time security assessment, predictive modeling and traceability analysis, a comprehensive risk score is calculated based on predefined weights: .

2. A food safety monitoring method based on a multimodal large model according to claim 1, characterized in that: Collect food information of multiple data modalities. For each data modality, use the corresponding neural network to extract features, including: Collecting food information in multiple data modalities, wherein the food information in multiple data modalities includes visual data, text data, sensor data, and supply chain data, wherein the visual data includes food images and packaging images, the text data includes ingredient lists, production dates, and batch numbers, the sensor data includes temperature, humidity, and gas concentration, and the supply chain data includes transportation routes and storage conditions; For visual data, convolutional neural networks are used to extract features. For sensor data, long short-term memory networks are used to extract features. For text data, long short-term memory networks are used to extract features. For supply chain data, fully connected networks are used to extract features: Among them, X m is the original input, F m is the extracted feature, Represents a specific neural network model designed based on the input data type.

3. The food safety monitoring method based on a multimodal large model according to claim 1 is characterized in that: Use the self-attention mechanism to fuse the extracted features to obtain fused features, including: Define three weight matrices for each mode m , and , the weight matrix is ​​used to extract the features Convert to query vector , key vector Sum value vector : ； Using Query Vectors and The attention score is calculated by taking the dot product of , and then the scale of the score is adjusted by scaling the dot product. After adjustment, the softmax function is applied to the result to obtain the final attention weight : Among them, d k is the dimension of the attention head; The attention weight Apply to a vector of values By aggregating different feature representations through weighted summation, we can obtain the fusion feature : A vector of values ​​representing different positions.

4. A food safety monitoring system based on a multimodal large model, characterized in that: The system comprises: A feature extraction module is used to collect food information of multiple data modes, and for each food information of each data mode, a corresponding neural network is used to extract features; The fusion feature module is used to fuse the extracted features using the self-attention mechanism to obtain fusion features; The risk prediction module is used to calculate the safety score based on the fusion features using a multi-layer perceptron, and obtain the predicted future risk of food based on the historical safety score and the fusion features of the current time step, including: The first layer of linear transformation module is used to transform the fused features Through the linear transformation of the first layer, a nonlinear activation function, namely the ReLU function, is applied to the output of the linear transformation of the first layer: Z1 = torch.matmul( , W1) + b1,A1 = torch.nn.ReLU()(Z1) Among them, W1 represents the weight matrix of the first layer, and bl represents the bias term of the first layer; The second-layer linear transformation module is used to transform the ReLU activated data A1 through the second-layer linear transformation and apply the Sigmoid function to obtain the final food safety score: Z2 = torch.matmul(A1, W2) + b2, Safety = torch.sigmoid(Z2) Among them, W2 represents the weight matrix of the second layer, and b2 represents the bias term of the second layer; Extract time feature module to fuse historical safety scores with the current time step Combined, use the LSTM network to process the combined data obtained in the previous step, extract key time features, and output : in, is the safety score for the past k time steps, is the fusion feature of the current time step; Get the future risk prediction module to convert the output of LSTM The input is sent to a multi-layer perceptron, which is further processed to obtain the predicted future risk Rt+1: ; The module for calculating the comprehensive risk score is used to build a probabilistic graph model for traceability analysis. It combines safety assessment, predicted future risks of food and traceability analysis results to calculate the comprehensive risk score of food, including: Build a supply chain probability graph model module, which is used to build a supply chain probability graph model G, including nodes V, edges E, and node features F node and edge feature F edge ; Calculate the relationship potential function module for each node vi , calculate its relationship with its neighbors N ( i )’s potential function φ; The conditional probability calculation module is used to calculate the value of each node using the potential function. vi Conditional probability P(vi|G) under a given supply chain probabilistic graphical model G: Where Z is the partition function, Representation and Node Adjacent nodes, Respectively represent nodes and neighbor nodes The characteristic vector of Represents a connection node and The characteristics of the edge; Get the maximum probability value node module, which is used to analyze the probabilities of all nodes and find the node with the maximum probability value; The module for calculating the comprehensive risk score is used to combine the results of real-time security assessment, prediction model and traceability analysis to calculate the comprehensive risk score according to predefined weights: 。 5. A food safety monitoring system based on a multimodal large model according to claim 4, characterized in that: The feature extraction module comprises: A multimodal data collection module is used to collect food information in multiple data modalities, wherein the food information in multiple data modalities includes visual data, text data, sensor data and supply chain data, wherein the visual data includes food images and packaging images, the text data includes ingredient list, production date and batch number, the sensor data includes temperature, humidity and gas concentration, and the supply chain data includes transportation route and storage conditions; Use neural network to extract features. For visual data, use convolutional neural network to extract features. For sensor data, use long short-term memory network to extract features. For text data, use long short-term memory network to extract features. For supply chain data, use fully connected network to extract features: Among them, X m is the original input, F m is the extracted feature, Represents a specific neural network model designed based on the input data type.

6. A food safety monitoring system based on a multimodal large model according to claim 4, characterized in that: The fusion feature module includes: Transformation vector module, used to define three weight matrices for each mode m , and , the weight matrix is ​​used to extract the features Convert to query vector , key vector Sum value vector : ； Using Query Vectors and The attention score is calculated by taking the dot product of , and then the scale of the score is adjusted by scaling the dot product. After adjustment, the softmax function is applied to the result to obtain the final attention weight : Among them, d k is the dimension of the attention head; The attention weight Apply to a vector of values By aggregating different feature representations through weighted summation, we can obtain the fusion feature : A vector of values ​​representing different positions.

Images