Food safety data analysis system and method based on multi-modal retrieval

Through a multimodal search system combining the Internet of Things and video surveillance data, food safety data analysis is carried out, which solves the coverage and timeliness of traditional regulatory methods, realizes real-time monitoring and personalized recommendations of food production and operation processes, and improves food The effect of safety supervision.

CN118916498AInactive Publication Date: 2024-11-08上海市大数据中心
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410970105.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional food safety supervision methods rely on manual inspection and sampling inspection, which have insufficient coverage, low detection frequency and poor timeliness, and are unable to effectively use multi-source data for comprehensive analysis, resulting in limited regulatory effects.

Method used

Using a food safety data analysis system based on multimodal retrieval, data preprocessing and feature extraction are carried out through real-time acquisition of IoT devices, video surveillance data and food data, and machine learning algorithms such as convolutional neural networks, random forests and deep nerves. The network performs abnormal detection and personalized recommendations.

Benefits of technology

Real-time monitoring and abnormal detection of food production and operation processes are realized, timely detection of situations that do not comply with safety standards, and personalized food safety information and recommendations are provided, which improves the accuracy and intelligence level of data analysis and information retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916498A_ABST
    Figure CN118916498A_ABST
Patent Text Reader

Abstract

The invention discloses a food safety data analysis system and method based on multi-modal retrieval, and belongs to the technical field of artificial intelligence. According to the invention, food safety related data is collected in real time from a plurality of data sources, data preprocessing is carried out, and the correctness and integrity of the data are verified; collecting user information, performing feature extraction, and establishing a user portrait; establishing a food safety standard grading model by using a decision tree algorithm; for video monitoring data, using a convolutional neural network to carry out image classification and anomaly detection; performing anomaly detection on the data of the Internet of Things equipment, and recording data which do not accord with a security threshold and a timestamp of the data; according to the timestamp, food related to food production and operation sites in the abnormal time period is marked as non-conforming to food safety; constructing a multi-modal retrieval model based on deep learning; carrying out semantic understanding on the user instruction by utilizing BERT, and carrying out semantic matching; and according to the user portrait and the retrieval result, generating corresponding food data conforming to food safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a food safety data analysis system and method based on multimodal retrieval. Background Art

[0002] With the improvement of people's living standards and the enhancement of food safety awareness, food safety issues have received more and more attention. Traditional food safety supervision methods mainly rely on manual inspection and sampling testing, which is not only inefficient, but also has problems of insufficient coverage and poor timeliness. In recent years, the development of technologies such as the Internet of Things, big data, and artificial intelligence has provided new means for food safety supervision, which can achieve real-time monitoring and management of the entire process of food production, processing, circulation, and consumption.

[0003] Traditional food safety supervision mainly relies on manual inspection and sampling testing, which has problems such as insufficient coverage, low detection frequency, and poor timeliness, making it difficult to fully and timely discover and deal with food safety hazards. The existing food safety supervision system has a single data source and mainly relies on self-reported data from enterprises and sampling test data. It lacks comprehensive analysis of data from multiple aspects such as production processes and environmental monitoring, and cannot fully reflect the food safety situation. The existing system lacks the ability to process and analyze large amounts of diverse data, and cannot effectively use multi-source data such as IoT devices, video surveillance, and historical records for comprehensive analysis, resulting in limited supervision results. Summary of the invention

[0004] The purpose of the present invention is to provide a food safety data analysis system and method based on multimodal retrieval to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A food safety data analysis method based on multimodal retrieval, the method comprising the following steps:

[0007] S100, collecting food safety related data in real time from several data sources, the data sources including IoT device data, video surveillance data and food data; performing data preprocessing, verifying the correctness and integrity of the data through rule verification, and archiving and backing up the data;

[0008] S200, collecting basic information and preference data of users, performing feature extraction, classifying users using K-means clustering algorithm, and establishing user portraits; establishing a food safety standard grading model using a decision tree algorithm, with the model input being user feature vectors and food safety related data, and the output being whether the food complies with the food safety standard of the user;

[0009] S300. For video surveillance data in food safety related data, use convolutional neural networks to perform image classification and anomaly detection, record frame images and timestamps that do not meet operating specifications or sanitary conditions, and report them; use random forest algorithms to perform anomaly detection on IoT device data, record data that does not meet safety thresholds and their timestamps; mark food related to food production and business premises within the abnormal time period as not meeting food safety standards based on timestamps;

[0010] S400. Use deep neural networks to build a multimodal retrieval model based on deep learning. Use convolutional neural networks to extract features from image data. Use recurrent neural networks to extract features from text data and perform model training. Use BERT to understand user instructions semantically and perform semantic matching. Generate corresponding food data that complies with food safety based on user portraits and retrieval results.

[0011] According to step S100, the IoT device data includes temperature data, humidity data, air pressure data, light data, air quality data and equipment vibration data; wherein, the temperature data comes from a temperature sensor for monitoring the temperature of a food production and storage environment, the humidity data comes from a humidity sensor for monitoring the humidity of a food production and storage environment, the air pressure data comes from an air pressure sensor for monitoring the air pressure of a food production and storage environment, the light data comes from a light sensor for monitoring the light conditions of a production and storage environment, the air quality data comes from an air quality sensor for monitoring harmful substances in the air, and the equipment vibration data comes from a vibration sensor for monitoring the operating status and abnormal conditions of the equipment; the video surveillance data comes from a surveillance camera for real-time monitoring of the operating specifications and sanitary conditions of food production and business premises; the food data includes food category data, food ingredient data, food nutrition data, food production date, food shelf life, food supply chain data and food testing data.

[0012] The data preprocessing includes missing value processing, noise filtering, duplicate data deduplication, timestamp alignment, data type conversion and Z-score normalization; through range checking, verify whether the data is within a reasonable range, through format checking, verify whether the data format is correct, through consistency checking, verify whether the data between different data sources are consistent, classify different types of data and store them in a distributed database, perform version control on the data, and back up the data regularly.

[0013] According to step S200, the user's basic information and preference data are converted into a feature vector u i =[u i1 ,u i2 ,...,u id ], where d is the total number of feature dimensions, u irepresents the feature vector of the i-th user, u i1 to u id It is the 1st dimension of the feature vector of the i-th user to the d-th dimension of the feature vector of the i-th user;

[0014] Basic information includes age, gender, occupation, health status, etc., preference data: taste preference, brand preference, eating habits, etc.

[0015] Use K-means clustering algorithm to classify users. The clustering goal is to divide users into k clusters. The optimization goal is:

[0016]

[0017] Among them, c j represents the center of the jth cluster, ||u i -c j || represents u i and c j The Euclidean distance of

[0018] According to the clustering results, a user portrait is generated for each cluster to describe the common characteristics and preferences of the users in the cluster. According to the user portrait, users are divided into different groups, including: Health-conscious group: characterized by older users or users with health problems, who pay attention to food ingredients, nutritional value and shelf life, and tend to buy organic food, low-sugar and low-fat food, such as diabetics, hypertensive patients, and the elderly; Taste preference group: characterized by users with specific taste preferences, who have a higher preference for certain specific flavors, such as spicy, sweet, sour, etc., and tend to buy foods with these flavors; Convenience and fast group: characterized by users with a fast pace of life, who tend to buy convenience foods, ready-to-eat foods and pre-packaged foods, and pay attention to the convenience of food; Special needs group: characterized by users with special health needs, who need special formula foods, such as lactose-free foods, gluten-free foods, and low-allergen foods, such as lactose-intolerant patients, users with allergies, and patients; Infant group: characterized by infants and their guardians, who pay attention to the safety and nutritional content of food, and tend to buy special foods such as infant formula milk powder and baby food.

[0019] According to user portraits, users are divided into different groups; a food safety standard grading model is established using a decision tree algorithm;

[0020] The goal of the decision tree is to classify the input features and output whether the food meets the user's food safety standards. The node splitting basis of the decision tree is information gain, and the formula is as follows:

[0021]

[0022] Where T is food safety related data, a is the input feature, H(T) is the entropy of data set T, Values(a) is all possible values ​​of a, T v is a data subset with feature a value v; H(T) is defined as follows:

[0023]

[0024] Among them, p i' is the probability that the i'th food category does not meet the standard, n is the total number of food categories;

[0025] Calculate the Gini coefficient G(T,a) using the following formula:

[0026]

[0027] Among them, p k is the probability that the food belongs to the standard judgment category k, indicating the probability that the food meets the standard, and m is the number of standard judgment categories, which include two categories: meeting the standard and not meeting the standard;

[0028] The feature vector u of the i-th user i and the feature vector f of the jth food j As input, where f j Represents food safety related data and uses the decision tree algorithm for training. The goal is to output whether the food meets the user's food safety standards. The accuracy and robustness of the model are evaluated using the cross-validation method, and the accuracy, recall, and F1-score of the model are calculated.

[0029] According to step S300, frame images are extracted from the video surveillance data periodically and at specific events, and a number of images of normal and abnormal operations are collected and annotated to create a training data set; a convolutional neural network model for image classification and anomaly detection is designed and implemented, including a number of convolutional layers, pooling layers, and fully connected layers; the convolutional neural network model is trained using the prepared data set so that the model can distinguish between normal and abnormal operations; frame images are continuously extracted from the video surveillance stream, and these frames are input into the trained CNN model for real-time analysis; the convolutional neural network model outputs a classification result to determine whether each frame image belongs to a normal operation or abnormal operation category;

[0030] Normal operation images are images taken during normal production and operation and are marked as normal operation. In addition, various images that do not meet operating specifications or sanitary conditions are collected and marked, such as employees not wearing masks, not wearing work clothes, and unclean sanitary environments.

[0031] For frame images classified as abnormal operations, the images, time and location of occurrence are recorded and reported; based on the time and location of the abnormal record, the food produced in that time period and location is marked as food that does not meet food safety standards.

[0032] According to step S300, historical sensor data within a period of time is collected, normal data and abnormal data are marked, and features are extracted from the historical data, including the reading of each sensor and the relevant timestamp; the random forest algorithm is selected as the anomaly detection model, and the random forest model is trained using the marked historical data; for sensor data determined to be abnormal, its specific data value and corresponding timestamp are recorded and reported; based on the recorded abnormal data and timestamp, all food production and storage conditions within the abnormal time period are determined, and the food produced and stored within the time period is marked as food that does not meet food safety standards, the safety status of the food is updated, and relevant personnel are notified to handle and rectify.

[0033] According to step S400, image data and text data are collected from food safety related data, and data annotation is performed, wherein the annotation content includes food type, operation specification and test result; feature extraction is performed on the image data using a convolutional neural network, and feature extraction is performed on the text data using a recurrent neural network, and the image feature vector extracted from the convolutional neural network model and the text feature vector extracted from the recurrent neural network model are fused through feature concatenation, and the fused feature vector represents the comprehensive features of the multimodal data;

[0034] A multimodal retrieval model is constructed using a deep neural network, with the input being the fused feature vector and the output being the relevance score of the retrieval results. The retrieval model is trained using a multimodal dataset, and the model parameters are optimized to enable it to evaluate the relevance between different inputs. Contrastive learning is used during the training process. The performance of the retrieval model is evaluated using a validation dataset, and the retrieval accuracy, recall rate, and F1-score indicators are calculated. The model parameters and structure are adjusted based on the evaluation results. The trained multimodal retrieval model is integrated into the system to provide real-time retrieval services.

[0035] According to step S400, the user inputs a query instruction through text, and the query instruction includes querying the safety information, production process and nutritional ingredients of a specific food; the input query instruction is preprocessed, and the preprocessing includes removing special characters, word segmentation and lowercase conversion; a pre-trained BERT model is loaded to perform in-depth semantic understanding of the text; the pre-processed text is input into the BERT model to extract the feature vector of the text; the BERT model can capture the contextual information and semantic relationship of the text and generate a text representation; the feature vector output by the BERT model represents the semantic information of the user instruction;

[0036] Build a food safety-related knowledge base and use the BERT model to convert text data in the knowledge base into feature vectors. Use cosine similarity to calculate the similarity between the feature vectors of user instructions and the feature vectors of each record in the knowledge base. Find several records that are most similar to user instructions as search results to match the most relevant food safety data. Filter the search results based on user portraits, and generate recommended content by combining user portraits and semantic matching search results.

[0037] Food safety data analysis system based on multimodal retrieval, including:

[0038] Data collection and preprocessing module: including: data collection unit, data preprocessing unit and data archiving and backup unit; wherein the data collection unit is responsible for real-time collection of food safety related data from several data sources, including IoT device data, video surveillance data and food data; the data preprocessing unit is responsible for data preprocessing and verifying the correctness and integrity of the data through rule verification; the data archiving and backup unit is responsible for archiving and backing up the data;

[0039] User portrait and food safety standard grading module: including: user data collection unit, user classification unit and food safety standard grading unit; among them, the user data collection unit is responsible for collecting the user's basic information and preference data, the user classification unit is responsible for feature extraction, using the K-means clustering algorithm to classify users and establish user portraits, and the food safety standard grading unit is responsible for establishing a food safety standard grading model using a decision tree algorithm. The model input is the user feature vector and food safety related data, and the output is whether the food meets the user's food safety standards;

[0040] Anomaly detection and marking module: including: video data processing unit, Internet of Things data processing unit and anomaly marking unit; wherein the video data processing unit uses a convolutional neural network to perform image classification and anomaly detection on the video surveillance data in the food safety related data, records and reports the frame images and timestamps that do not meet the operating specifications or sanitary conditions, and the Internet of Things data processing unit uses a random forest algorithm to perform anomaly detection on the Internet of Things device data, records the data that does not meet the safety threshold and its timestamp, and the anomaly marking unit marks the food related to the food production and business premises within the abnormal time period as not meeting the food safety requirements based on the timestamp;

[0041] Multimodal retrieval module: includes: feature extraction unit, multimodal retrieval unit, user instruction processing unit and personalized recommendation unit; wherein, the feature extraction unit uses a deep neural network to build a multimodal retrieval model based on deep learning, uses a convolutional neural network to extract features from image data, and uses a recurrent neural network to extract features from text data. The multimodal retrieval unit performs model training, the user instruction processing unit uses BERT to semantically understand user instructions and perform semantic matching, and the personalized recommendation unit generates corresponding food data that complies with food safety based on user portraits and retrieval results.

[0042] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0043] The present invention utilizes Internet of Things devices and video surveillance systems to monitor the food production and operation processes in real time, performs anomaly detection through machine learning algorithms, promptly discovers and records situations that do not meet safety regulations, and implements real-time early warning and processing.

[0044] The present invention collects users' basic information and preference data, establishes user portraits, uses machine learning algorithms to perform personalized food safety standard grading and recommendations, and provides food safety information that meets user needs.

[0045] The present invention uses a deep learning algorithm to extract multimodal data features and utilizes the BERT model to perform semantic understanding and matching of user instructions, thereby improving the accuracy and intelligence level of data analysis and information retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0047] Figure 1 It is a schematic diagram of the steps of the food safety data analysis method based on multimodal retrieval of the present invention;

[0048] Figure 2 It is a system structure diagram of the food safety data analysis system based on multimodal retrieval of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0051] According to one embodiment of the present invention, Figure 1 As shown in the schematic diagram of the steps of the food safety data analysis method based on multimodal retrieval, the food safety data analysis method based on multimodal retrieval comprises the following steps:

[0052] S100, collecting food safety related data in real time from several data sources, the data sources including IoT device data, video surveillance data and food data; performing data preprocessing, verifying the correctness and integrity of the data through rule verification, and archiving and backing up the data;

[0053] S200, collecting basic information and preference data of users, performing feature extraction, classifying users using K-means clustering algorithm, and establishing user portraits; establishing a food safety standard grading model using a decision tree algorithm, with the model input being user feature vectors and food safety related data, and the output being whether the food complies with the food safety standard of the user;

[0054] S300. For video surveillance data in food safety related data, use convolutional neural networks to perform image classification and anomaly detection, record frame images and timestamps that do not meet operating specifications or sanitary conditions, and report them; use random forest algorithms to perform anomaly detection on IoT device data, record data that does not meet safety thresholds and their timestamps; mark food related to food production and business premises within the abnormal time period as not meeting food safety standards based on timestamps;

[0055] S400. Use deep neural networks to build a multimodal retrieval model based on deep learning. Use convolutional neural networks to extract features from image data. Use recurrent neural networks to extract features from text data and perform model training. Use BERT to understand user instructions semantically and perform semantic matching. Generate corresponding food data that complies with food safety based on user portraits and retrieval results.

[0056] According to step S100, the IoT device data includes temperature data, humidity data, air pressure data, light data, air quality data and equipment vibration data; wherein, the temperature data comes from a temperature sensor for monitoring the temperature of a food production and storage environment, the humidity data comes from a humidity sensor for monitoring the humidity of a food production and storage environment, the air pressure data comes from an air pressure sensor for monitoring the air pressure of a food production and storage environment, the light data comes from a light sensor for monitoring the light conditions of a production and storage environment, the air quality data comes from an air quality sensor for monitoring harmful substances in the air, and the equipment vibration data comes from a vibration sensor for monitoring the operating status and abnormal conditions of the equipment; the video surveillance data comes from a surveillance camera for real-time monitoring of the operating specifications and sanitary conditions of food production and business premises; the food data includes food category data, food ingredient data, food nutrition data, food production date, food shelf life, food supply chain data and food testing data.

[0057] The data preprocessing includes missing value processing, noise filtering, duplicate data deduplication, timestamp alignment, data type conversion and Z-score normalization; through range checking, verify whether the data is within a reasonable range, through format checking, verify whether the data format is correct, through consistency checking, verify whether the data between different data sources are consistent, classify different types of data and store them in a distributed database, perform version control on the data, and back up the data regularly.

[0058] According to step S200, the user's basic information and preference data are converted into a feature vector u i =[u i1 ,u i2 ,...,u id ], where d is the total number of feature dimensions, u i represents the feature vector of the i-th user, u i1 to u id It is the 1st dimension of the feature vector of the i-th user to the d-th dimension of the feature vector of the i-th user;

[0059] Basic information includes age, gender, occupation, health status, etc., preference data: taste preference, brand preference, eating habits, etc.

[0060] In this embodiment, as shown in Table 1, the user basic information and preference data table contains partial data of the following 10 users:

[0061] Table 1 User basic information and preference data table

[0062]

[0063] Use K-means clustering algorithm to classify users. The clustering goal is to divide users into 3 clusters. The optimization goal is:

[0064]

[0065] Among them, c j represents the center of the jth cluster, ||u i -c j || represents u i and c j The Euclidean distance of

[0066] Through the K-means algorithm, we get the following clustering results: Cluster 1: U1, U4, U6, U8; Cluster 2: U2, U5, U9, U10; Cluster 3: U3, U7;

[0067] Based on the clustering results, a user profile for each cluster is generated: Cluster 1 (health-conscious group): users are older or have health problems, pay attention to the safety and healthy ingredients of food, and tend to buy low-sugar, low-salt, low-lactose, and low-allergy foods. Cluster 2 (brand-sensitive group): users pay attention to food brands and taste preferences, and tend to buy branded foods and foods with specific flavors. Cluster 3 (convenient and fast group): users are young and have a fast pace of life, and tend to buy convenience foods and fast food.

[0068] According to user portraits, users are divided into different groups; a food safety standard grading model is established using a decision tree algorithm;

[0069] The goal of the decision tree is to classify the input features and output whether the food meets the user's food safety standards. The node splitting basis of the decision tree is information gain, and the formula is as follows:

[0070]

[0071] Where T is food safety related data, a is the input feature, H(T) is the entropy of data set T, Values(a) is all possible values ​​of a, T v is a data subset with feature a value v; H(T) is defined as follows:

[0072]

[0073] Among them, p i' is the probability that the i'th food category does not meet the standard, n is the total number of food categories;

[0074] Calculate the Gini coefficient G(T,a) using the following formula:

[0075]

[0076] Among them, pk is the probability that the food belongs to the standard judgment category k, indicating the probability that the food meets the standard, and m is the number of standard judgment categories, which include two categories: meeting the standard and not meeting the standard;

[0077] The feature vector u of the i-th user i and the feature vector f of the jth food j As input, where f j Represents food safety related data and uses the decision tree algorithm for training. The goal is to output whether the food meets the user's food safety standards. The accuracy and robustness of the model are evaluated using the cross-validation method, and the accuracy, recall, and F1-score of the model are calculated.

[0078] According to Table 2, the food data table, in this embodiment, there are the following food data:

[0079] Table 2 Food data table

[0080]

[0081] These data were input into the decision tree model, and after training and evaluation, the following results were obtained: model accuracy: 0.85, recall: 0.80, F1-score: 0.82; these results show that the model has high accuracy and robustness in distinguishing whether food meets the user's food safety standards.

[0082] Based on the model’s predictions, the following foods were marked as not meeting food safety standards for some groups: F2: Not meeting standards for health-conscious groups and special needs groups due to high salt content and unsatisfactory test results.

[0083] According to step S300, frame images are extracted from the video surveillance data periodically and at specific events, and a number of images of normal and abnormal operations are collected and annotated to create a training data set; a convolutional neural network model for image classification and anomaly detection is designed and implemented, including a number of convolutional layers, pooling layers, and fully connected layers; the convolutional neural network model is trained using the prepared data set so that the model can distinguish between normal and abnormal operations; frame images are continuously extracted from the video surveillance stream, and these frames are input into the trained CNN model for real-time analysis; the convolutional neural network model outputs a classification result to determine whether each frame image belongs to a normal operation or abnormal operation category;

[0084] Normal operation images are images taken during normal production and operation and are marked as normal operation. In addition, various images that do not meet operating specifications or sanitary conditions are collected and marked, such as employees not wearing masks, not wearing work clothes, and unclean sanitary environments.

[0085] For frame images classified as abnormal operations, the images, time and location of occurrence are recorded and reported; based on the time and location of the abnormal record, the food produced in that time period and location is marked as food that does not meet food safety standards.

[0086] According to step S300, historical sensor data within a period of time is collected, normal data and abnormal data are marked, and features are extracted from the historical data, including the reading of each sensor and the relevant timestamp; the random forest algorithm is selected as the anomaly detection model, and the random forest model is trained using the marked historical data; for sensor data determined to be abnormal, its specific data value and corresponding timestamp are recorded and reported; based on the recorded abnormal data and timestamp, all food production and storage conditions within the abnormal time period are determined, and the food produced and stored within the time period is marked as food that does not meet food safety standards, the safety status of the food is updated, and relevant personnel are notified to handle and rectify.

[0087] According to step S400, image data and text data are collected from food safety related data, and data annotation is performed, wherein the annotation content includes food type, operation specification and test result; feature extraction is performed on the image data using a convolutional neural network, and feature extraction is performed on the text data using a recurrent neural network, and the image feature vector extracted from the convolutional neural network model and the text feature vector extracted from the recurrent neural network model are fused through feature concatenation, and the fused feature vector represents the comprehensive features of the multimodal data;

[0088] A multimodal retrieval model is constructed using a deep neural network, with the input being the fused feature vector and the output being the relevance score of the retrieval results. The retrieval model is trained using a multimodal dataset, and the model parameters are optimized to enable it to evaluate the relevance between different inputs. Contrastive learning is used during the training process. The performance of the retrieval model is evaluated using a validation dataset, and the retrieval accuracy, recall rate, and F1-score indicators are calculated. The model parameters and structure are adjusted based on the evaluation results. The trained multimodal retrieval model is integrated into the system to provide real-time retrieval services.

[0089] According to step S400, the user inputs a query instruction through text, and the query instruction includes querying the safety information, production process and nutritional ingredients of a specific food; the input query instruction is preprocessed, and the preprocessing includes removing special characters, word segmentation and lowercase conversion; a pre-trained BERT model is loaded to perform in-depth semantic understanding of the text; the pre-processed text is input into the BERT model to extract the feature vector of the text; the BERT model can capture the contextual information and semantic relationship of the text and generate a text representation; the feature vector output by the BERT model represents the semantic information of the user instruction;

[0090] Build a food safety-related knowledge base and use the BERT model to convert text data in the knowledge base into feature vectors. Use cosine similarity to calculate the similarity between the feature vectors of user instructions and the feature vectors of each record in the knowledge base. Find several records that are most similar to user instructions as search results to match the most relevant food safety data. Filter the search results based on user portraits, and generate recommended content by combining user portraits and semantic matching search results.

[0091] According to another embodiment of the present invention, Figure 2 As shown in the system structure diagram of the food safety data analysis system based on multimodal retrieval, the food safety data analysis system based on multimodal retrieval includes:

[0092] Data collection and preprocessing module: including: data collection unit, data preprocessing unit and data archiving and backup unit; wherein the data collection unit is responsible for real-time collection of food safety related data from several data sources, including IoT device data, video surveillance data and food data; the data preprocessing unit is responsible for data preprocessing and verifying the correctness and integrity of the data through rule verification; the data archiving and backup unit is responsible for archiving and backing up the data;

[0093] User portrait and food safety standard grading module: including: user data collection unit, user classification unit and food safety standard grading unit; among them, the user data collection unit is responsible for collecting the user's basic information and preference data, the user classification unit is responsible for feature extraction, using the K-means clustering algorithm to classify users and establish user portraits, and the food safety standard grading unit is responsible for establishing a food safety standard grading model using a decision tree algorithm. The model input is the user feature vector and food safety related data, and the output is whether the food meets the user's food safety standards;

[0094] Anomaly detection and marking module: including: video data processing unit, Internet of Things data processing unit and anomaly marking unit; wherein the video data processing unit uses a convolutional neural network to perform image classification and anomaly detection on the video surveillance data in the food safety related data, records and reports the frame images and timestamps that do not meet the operating specifications or sanitary conditions, and the Internet of Things data processing unit uses a random forest algorithm to perform anomaly detection on the Internet of Things device data, records the data that does not meet the safety threshold and its timestamp, and the anomaly marking unit marks the food related to the food production and business premises within the abnormal time period as not meeting the food safety requirements based on the timestamp;

[0095] Multimodal retrieval module: includes: feature extraction unit, multimodal retrieval unit, user instruction processing unit and personalized recommendation unit; wherein, the feature extraction unit uses a deep neural network to build a multimodal retrieval model based on deep learning, uses a convolutional neural network to extract features from image data, and uses a recurrent neural network to extract features from text data. The multimodal retrieval unit performs model training, the user instruction processing unit uses BERT to semantically understand user instructions and perform semantic matching, and the personalized recommendation unit generates corresponding food data that complies with food safety based on user portraits and retrieval results.

[0096] In this embodiment, we have the following sample data: image data: 1,000 surveillance images of food production and business sites, text data: 500 food inspection reports and user comments.

[0097] The pre-trained ResNet-50 model is used to extract image features. Each image passes through the convolution layer, pooling layer, and fully connected layer to generate a feature vector. The LSTM model is used to extract text features. Each text passes through the embedding layer, LSTM layer, and fully connected layer to generate a feature vector.

[0098] For multimodal model training, the feature vectors of images and texts are input into the joint embedding space, and the multimodal contrast loss function is used for training to make similar image and text features closer in the embedding space; we use the following data for training: training set: 800 images and 400 texts, validation set: 200 images and 100 texts. The Adam optimizer is used with a learning rate of 0.001, and 50 epochs are trained. The model performance is verified after each epoch.

[0099] BERT is used to perform semantic understanding and semantic matching of user instructions. The pre-trained BERT model is used to perform semantic understanding of user instructions, convert user instructions into feature vectors, and match them with the feature vectors in the multimodal retrieval model. We have the following user instructions: "Find low-sugar foods suitable for diabetics" and "Recommend organic and lactose-free foods."

[0100] According to the user portrait and search results, the corresponding food data that meets food safety standards is generated. We have the following user portraits and search results: User portrait: Group 1 (diabetics): focus on low-sugar foods, Group 2 (lactose intolerant): focus on lactose-free foods. Search results: Image data: images that meet hygiene standards, text data: test reports that meet safety standards.

[0101] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0102] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A food safety data analysis method based on multimodal retrieval, characterized in that: The method comprises the following steps: S100, collecting food safety related data in real time from several data sources, the data sources including IoT device data, video surveillance data and food data; performing data preprocessing, verifying the correctness and integrity of the data through rule verification, and archiving and backing up the data; S200, collecting basic information and preference data of users, performing feature extraction, classifying users using K-means clustering algorithm, and establishing user portraits; establishing a food safety standard grading model using a decision tree algorithm, with the model input being user feature vectors and food safety related data, and the output being whether the food complies with the food safety standard of the user; S300: For the video surveillance data in the food safety related data, use the convolutional neural network to perform image classification and anomaly detection, record the frame images and timestamps that do not meet the operating specifications or sanitary conditions, and report them; Use the random forest algorithm to detect anomalies in IoT device data and record data that does not meet security thresholds and their timestamps; Based on the timestamp, food related to food production and business premises during the abnormal time period is marked as non-compliant with food safety; S400. Use deep neural networks to build a multimodal retrieval model based on deep learning. Use convolutional neural networks to extract features from image data. Use recurrent neural networks to extract features from text data and perform model training. Use BERT to understand user instructions semantically and perform semantic matching. Generate corresponding food data that complies with food safety based on user portraits and retrieval results.

2. The food safety data analysis method based on multimodal retrieval according to claim 1, characterized in that: According to step S100, the IoT device data includes temperature data, humidity data, air pressure data, light data, air quality data and equipment vibration data; wherein, the temperature data comes from a temperature sensor for monitoring the temperature of a food production and storage environment, the humidity data comes from a humidity sensor for monitoring the humidity of a food production and storage environment, the air pressure data comes from an air pressure sensor for monitoring the air pressure of a food production and storage environment, the light data comes from a light sensor for monitoring the light conditions of a production and storage environment, the air quality data comes from an air quality sensor for monitoring harmful substances in the air, and the equipment vibration data comes from a vibration sensor for monitoring the operating status and abnormal conditions of the equipment; the video surveillance data comes from a surveillance camera for real-time monitoring of the operating specifications and sanitary conditions of food production and business premises; the food data includes food category data, food ingredient data, food nutrition data, food production date, food shelf life, food supply chain data and food testing data.

3. The food safety data analysis method based on multimodal retrieval according to claim 2 is characterized in that: The data preprocessing includes missing value processing, noise filtering, duplicate data deduplication, timestamp alignment, data type conversion and Z-score normalization; through range checking, verify whether the data is within a reasonable range, through format checking, verify whether the data format is correct, through consistency checking, verify whether the data between different data sources are consistent, classify different types of data and store them in a distributed database, perform version control on the data, and back up the data regularly.

4. The food safety data analysis method based on multimodal retrieval according to claim 1, characterized in that: According to step S200, the user's basic information and preference data are converted into a feature vector u i =[u i1 ,u i2 ,...,u id ], where d is the total number of feature dimensions, u i represents the feature vector of the i-th user, u i1 to u id It is the 1st dimension of the feature vector of the i-th user to the d-th dimension of the feature vector of the i-th user; Use K-means clustering algorithm to classify users. The clustering goal is to divide users into k clusters. The optimization goal is: Among them, c j represents the center of the jth cluster, ||u i -c j || represents u i and c j The Euclidean distance of Based on the clustering results, a user portrait is generated for each cluster, describing the common characteristics and preferences of the users in the cluster.

5. The food safety data analysis method based on multimodal retrieval according to claim 4 is characterized in that: According to user portraits, users are divided into different groups; a food safety standard grading model is established using a decision tree algorithm; The goal of the decision tree is to classify the input features and output whether the food meets the user's food safety standards. The node splitting basis of the decision tree is information gain, and the formula is as follows: Where T is food safety related data, a is the input feature, H(T) is the entropy of data set T, Values(a) is all possible values ​​of a, T v is a data subset with feature a value v; H(T) is defined as follows: Among them, p i' is the probability that the i'th food category does not meet the standard, n is the total number of food categories; Calculate the Gini coefficient G(T,a) using the following formula: Among them, p k is the probability that the food belongs to the standard judgment category k, indicating the probability that the food meets the standard, and m is the number of standard judgment categories, which include two categories: meeting the standard and not meeting the standard; The feature vector u of the i-th user i and the feature vector f of the jth food j As input, where f j Represents food safety related data and uses the decision tree algorithm for training. The goal is to output whether the food meets the user's food safety standards. The accuracy and robustness of the model are evaluated using the cross-validation method, and the accuracy, recall, and F1-score of the model are calculated.

6. The food safety data analysis method based on multimodal retrieval according to claim 1, characterized in that: According to step S300, frame images are extracted from the video surveillance data regularly and at specific events, and a number of images of normal and abnormal operations are collected and annotated to create a training data set; a convolutional neural network model for image classification and anomaly detection is designed and implemented, including a number of convolutional layers, pooling layers, and fully connected layers; the convolutional neural network model is trained using the prepared data set so that the model can distinguish between normal and abnormal operations; frame images are continuously extracted from the video surveillance stream, and these frames are input into the trained CNN model for real-time analysis; the convolutional neural network model outputs a classification result to determine whether each frame image belongs to a normal operation or abnormal operation category; For frame images classified as abnormal operations, the images, time and location of occurrence are recorded and reported; based on the time and location of the abnormal record, the food produced in that time period and location is marked as food that does not meet food safety standards.

7. The food safety data analysis method based on multimodal retrieval according to claim 6 is characterized in that: According to step S300, historical sensor data within a period of time is collected, normal data and abnormal data are marked, and features are extracted from the historical data, including the reading of each sensor and the relevant timestamp; the random forest algorithm is selected as the anomaly detection model, and the random forest model is trained using the marked historical data; for sensor data determined to be abnormal, its specific data value and corresponding timestamp are recorded and reported; based on the recorded abnormal data and timestamp, all food production and storage conditions within the abnormal time period are determined, and the food produced and stored within the time period is marked as food that does not meet food safety standards, the safety status of the food is updated, and relevant personnel are notified to handle and rectify.

8. The food safety data analysis method based on multimodal retrieval according to claim 1, characterized in that: According to step S400, image data and text data are collected from food safety related data, and data annotation is performed, wherein the annotation content includes food type, operation specification and test result; feature extraction is performed on the image data using a convolutional neural network, and feature extraction is performed on the text data using a recurrent neural network, and the image feature vector extracted from the convolutional neural network model and the text feature vector extracted from the recurrent neural network model are fused through feature concatenation, and the fused feature vector represents the comprehensive features of the multimodal data; A multimodal retrieval model is constructed using a deep neural network, with the input being the fused feature vector and the output being the relevance score of the retrieval results. The retrieval model is trained using a multimodal dataset, and the model parameters are optimized to enable it to evaluate the relevance between different inputs.During the training process, contrastive learning is used for training; the performance of the retrieval model is evaluated using a validation dataset, the retrieval accuracy, recall rate, and F1-score indicators are calculated, and the model parameters and structure are adjusted according to the evaluation results; the trained multimodal retrieval model is integrated into the system to provide real-time retrieval services.

9. The food safety data analysis method based on multimodal retrieval according to claim 8, characterized in that: According to step S400, the user inputs a query instruction through text, and the query instruction includes querying the safety information, production process and nutritional ingredients of a specific food; the input query instruction is preprocessed, and the preprocessing includes removing special characters, word segmentation and lowercase conversion; a pre-trained BERT model is loaded to perform in-depth semantic understanding of the text; the pre-processed text is input into the BERT model to extract the feature vector of the text; the BERT model can capture the contextual information and semantic relationship of the text and generate a text representation; the feature vector output by the BERT model represents the semantic information of the user instruction; Build a knowledge base related to food safety, and use the BERT model to convert text data in the knowledge base into feature vectors. Use cosine similarity to calculate the similarity between the feature vectors of user instructions and the feature vectors of each record in the knowledge base. Find several records that are most similar to the user instructions as search results, and match the most relevant food safety data. Filter the search results according to the user portrait, and generate recommended content by combining the user portrait and semantic matching search results.

10. A food safety data analysis system based on multimodal retrieval, using the food safety data analysis method based on multimodal retrieval according to any one of claims 1 to 9, characterized in that: include: Data collection and preprocessing module: including: data collection unit, data preprocessing unit and data archiving and backup unit; wherein the data collection unit is responsible for real-time collection of food safety related data from several data sources, including IoT device data, video surveillance data and food data; the data preprocessing unit is responsible for data preprocessing and verifying the correctness and integrity of the data through rule verification; the data archiving and backup unit is responsible for archiving and backing up the data; User portrait and food safety standard grading module: including: user data collection unit, user classification unit and food safety standard grading unit; among them, the user data collection unit is responsible for collecting the user's basic information and preference data, the user classification unit is responsible for feature extraction, using the K-means clustering algorithm to classify users and establish user portraits, and the food safety standard grading unit is responsible for establishing a food safety standard grading model using a decision tree algorithm. The model input is the user feature vector and food safety related data, and the output is whether the food meets the user's food safety standards; Anomaly detection and marking module: including: video data processing unit, Internet of Things data processing unit and anomaly marking unit; wherein the video data processing unit uses a convolutional neural network to perform image classification and anomaly detection on the video surveillance data in the food safety related data, records and reports the frame images and timestamps that do not meet the operating specifications or sanitary conditions, and the Internet of Things data processing unit uses a random forest algorithm to perform anomaly detection on the Internet of Things device data, records the data that does not meet the safety threshold and its timestamp, and the anomaly marking unit marks the food related to the food production and business premises within the abnormal time period as not meeting the food safety requirements based on the timestamp; Multimodal retrieval module: includes: feature extraction unit, multimodal retrieval unit, user instruction processing unit and personalized recommendation unit; wherein, the feature extraction unit uses a deep neural network to build a multimodal retrieval model based on deep learning, uses a convolutional neural network to extract features from image data, and uses a recurrent neural network to extract features from text data. The multimodal retrieval unit performs model training, the user instruction processing unit uses BERT to semantically understand user instructions and perform semantic matching, and the personalized recommendation unit generates corresponding food data that complies with food safety based on user portraits and retrieval results.

Citation Information

Cited By

  • Platform reporting method and system based on AIGC robot and medium

    CN119378569A

  • User identification method and device for canteen intelligent settlement

    CN119442205A

  • User identification method and device for canteen intelligent settlement

    CN119442205B

  • Food processing safety supervision method, device and equipment and storage medium

    CN120014552A