Food safety traceability method and system based on deep learning

Abstract

The application discloses a food safety traceability method and system based on deep learning, and relates to the technical field of food safety traceability. The method comprises the following steps: acquiring full-life-cycle multi-modal data; dividing the full life cycle into multiple nodes, establishing a mapping relationship between the multi-modal data and the nodes, and identifying known nodes with complete data and unknown nodes with missing data; according to the monitoring data of the known nodes and the space-time relationship between the known nodes and the unknown nodes, performing safety state evaluation on the known nodes through a deep learning model, and performing state prediction and data completion on the unknown nodes, thereby constructing a product life cycle data chain; and performing food safety state visual conversion on each node according to the product life cycle data chain, and generating a visual traceability report. The technical problem that the food safety traceability data is incomplete in the prior art, resulting in insufficient traceability result credibility, is solved, and the technical effect of improving traceability integrity and credibility by constructing a product life cycle data chain is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of food safety traceability, and particularly relates to a food safety traceability method and system based on deep learning. BACKGROUND

[0002] With the continuous extension of the food industry chain, a large amount of information will be generated in each link from production, processing, circulation to consumption of food. Food safety traceability, as an important means to protect consumer rights and improve regulatory level, has been widely used in the industry. However, the existing traceability methods mostly rely on manual recording or scattered information systems, resulting in inconsistent data sources and incomplete information chain, and often difficult to cover the whole life cycle of food. Once a food safety incident occurs, the traceability result is easy to have breakpoints and missing, so that the credibility of the traceability result is insufficient. SUMMARY

[0003] The present application provides a food safety traceability method and system based on deep learning, which solves the technical problem of insufficient traceability result credibility caused by incomplete food safety traceability data in the prior art.

[0004] In a first aspect, the present application provides a food safety traceability method based on deep learning, which comprises:

[0005] In response to a traceability request of a target food, full life cycle multi-modal data associated with a food unique identifier is obtained, the multi-modal data including time series sensor data, image video data and structured text data; based on a predefined multi-level node template, the full life cycle of the target food is divided into multiple nodes, and a mapping relationship between the multi-modal data and each node is established to identify known nodes with complete data and unknown nodes with missing data; according to the monitoring data of the known nodes and their spatio-temporal relationship with the unknown nodes, a deep learning model is used to evaluate the safety state of the known nodes and predict and complete the data of the unknown nodes, to construct a continuous and complete product life cycle data chain; and the food safety state of each node is visualized and converted according to the product life cycle data chain, to generate a visual traceability report.

[0006] In a second aspect, the present application provides a food safety traceability system based on deep learning, which comprises:

[0007] The data acquisition unit acquires full life cycle multi-modal data associated with the food unique identifier in response to a traceability request of the target food, the multi-modal data including time series sensor data, image video data and structured text data; the relationship establishment unit divides the full life cycle of the target food into multiple nodes based on a predefined multi-level node template, and establishes a mapping relationship between the multi-modal data and each node to identify known nodes with complete data and unknown nodes with missing data; the evaluation unit evaluates the safety state of the known nodes through a deep learning model according to the monitoring data of the known nodes and the space-time relationship with the unknown nodes, and predicts the state and completes the data of the unknown nodes to construct a continuous and complete product life cycle data chain; and the visualization unit visualizes the food safety state of each node according to the product life cycle data chain to generate a visual traceability report.

[0008] The one or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0009] Firstly, in response to a traceability request of the target food, full life cycle multi-modal data associated with the food unique identifier is acquired, and the multi-modal data includes time series sensor data, image video data and structured text data. Then, based on a predefined multi-level node template, the full life cycle of the target food is divided into multiple nodes, and a mapping relationship between the multi-modal data and each node is established to identify known nodes with complete data and unknown nodes with missing data. Then, according to the monitoring data of the known nodes and the space-time relationship with the unknown nodes, the safety state of the known nodes is evaluated through a deep learning model, and the state of the unknown nodes is predicted and the data is completed to construct a continuous and complete product life cycle data chain. Finally, the food safety state of each node is visualized according to the product life cycle data chain to generate a visual traceability report. The technical problem of incomplete food safety traceability data in the prior art, which leads to insufficient traceability result credibility, is solved, and the technical effect of improving traceability integrity and credibility by constructing a product life cycle data chain is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 A flowchart of a food safety traceability method based on deep learning provided by the embodiments of the present application is shown in the figure.

[0012] Figure 2A food safety traceability system structure based on deep learning is provided in the embodiments of the present application.

[0013] Reference signs: data acquisition unit 11, relationship establishment unit 12, evaluation unit 13, and visualization unit 14. DETAILED DESCRIPTION

[0014] The present application provides a food safety traceability method and system based on deep learning, which solves the technical problem of incomplete food safety traceability data in the prior art, resulting in insufficient traceability result reliability.

[0015] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0016] It should be noted that the terms "comprise" and "have" are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server comprising a series of steps or units does not have to be limited to those clearly listed steps or units, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] Embodiment one, as shown in the present application provides a food safety traceability method based on deep learning, wherein the method comprises: Figure 1

[0018] In response to a traceability request of a target food, full-life-cycle multi-modal data associated with a food unique identifier is acquired, and the multi-modal data includes time series sensor data, image / video data and structured text data.

[0019] ​In the embodiments of the present application, when the system receives a user's traceability request for a target food, the food traceability database is called to search for the unique identifier of the target food. The unique identifier can be a bar code, a two-dimensional code, a radio frequency identification tag (RFID), or a unique traceability code generated based on a blockchain, which is used to maintain uniqueness and non-tamperability throughout the food supply chain. The system searches for the multi-modal data collected at each link of the target food in its production, processing, storage, transportation, and sales life cycle according to the unique identifier, including: time series sensor data collected by temperature, humidity, gas composition sensors, pressure sensors, etc., which reflect the dynamic changes of the food under the environmental conditions at each link; image and video data obtained by production workshop cameras, transportation monitoring videos, or quality inspection shooting equipment, which reflect the appearance state, packaging integrity, and operation behavior compliance of the food; and structured text data generated by enterprise management systems, quality inspection reports, logistics documents, etc., which record key text information such as production batch, detection index, transportation time, and supplier information.

[0020] Based on a predefined multi-level node template, the life cycle of the target food is divided into multiple nodes, and a mapping relationship between the multi-modal data and each node is established to identify known nodes with complete data and unknown nodes with missing data.

[0021] In the embodiments of the present application, the system pre-constructs a multi-level node template for structurally dividing the life cycle of the food. The multi-level node template is trained from general food life cycle data, and its hierarchical structure covers food categories, food types, and specific production or circulation links from high to low, each node being attached with information such as time, location, operation type, and food attribute labels. After receiving the multi-modal data of the target food, the system divides the life cycle process of the food according to the multi-level node template, for example, including raw material procurement, primary processing, deep processing, packaging, storage, transportation, distribution, and retail. Subsequently, the system matches the expected time, space, and operation type of each node in the multi-level node template based on the time stamp, geographic location, operation type, and other key fields in the multi-modal data, thereby establishing a mapping relationship between the multi-modal data and the template nodes. When a node can match a complete data set, the node is identified as a known node with complete data; when a node cannot match sufficient data or is missing, it is identified as an unknown node.

[0022] Further, based on the predefined multi-level node template, the life cycle of the target food is divided into multiple nodes, which includes:

[0023] Obtaining a target food traceability dataset, the traceability dataset containing full life cycle division nodes, cycle attributes and state data of a plurality of foods; performing cluster analysis on the life cycle division nodes in the traceability dataset to obtain a general node sequence representing different food categories; based on the effective coverage rate of the clustering result, constructing a hierarchical tree structure for the general node sequence, wherein the high-level nodes cover a wide range of food categories, and the low-level nodes correspond to subdivided food categories; inserting the abnormal nodes that cannot be clustered into the general node sequence as special leaf nodes into the bottom layer of the tree structure to construct the multi-level node template, wherein each node in the template is attached with a set of food attribute labels that can be matched.

[0024] First, a target food traceability dataset is obtained, which is composed of historical traceability information from different food enterprises, regulatory platforms and circulation links, and contains division nodes, cycle attributes (such as duration, frequency, and time sequence relationship) of each node, and state data corresponding to the nodes of a plurality of foods in the production, processing, transportation, storage, and sales of the full life cycle. Then, the life cycle division nodes in the traceability dataset of the target food are subjected to cluster analysis, and nodes with similar attributes and functions are divided into a group by a clustering algorithm, thereby obtaining a general node sequence that can represent the characteristics of different food categories. Next, based on the effective coverage rate of the clustering result, a hierarchical tree structure is constructed for the general node sequence: the high-level nodes of the tree structure represent a wide range of food categories (such as fresh food, dairy products, and beverages), while the low-level nodes correspond to more subdivided food categories or specific processing steps (such as pasteurization, vacuum packaging, and cold chain transportation). For abnormal nodes that cannot be clustered into the general node sequence, such as specific food-specific processing procedures or special detection steps, they are inserted as special leaf nodes into the bottom layer of the tree structure. Finally, the completed multi-level node template not only presents a hierarchical tree structure, but also has an attribute label set attached to each node, which includes time constraints, location constraints, operation types, detection indicators, etc.

[0025] Further, a mapping relationship between the multi-modal data and each node is established to identify known nodes with complete data and unknown nodes with missing data, including:

[0026] Based on the timestamp, location coordinate and operation type fields in the multi-modal data, the expected time and operation type of a specific node in the multi-level node template are matched; according to the matching relationship, the data matched successfully for the specific node is marked as known monitoring data of the corresponding node, and the mapping relationship between the multi-modal data and each node is established; based on the mapping relationship, the matched nodes are marked as known nodes, and the unmatched nodes are marked as unknown nodes.

[0027] Specifically, key field information is extracted from the multi-modal data, including timestamps, location coordinates, and operation types, etc. The system compares these fields with the expected time range, geographical location constraints, and operation type descriptions set in each node of the pre-defined multi-level node template one by one. For example, when the timestamp in the multi-modal data is within the expected time period of a node, the location coordinates match the geographical location of the node, and the operation type is consistent with the pre-set operation behavior of the node, it is determined that the data matches the node successfully. Subsequently, the matched data is marked as known monitoring data of the corresponding node, thereby gradually establishing a one-to-one mapping relationship between the multi-modal data and the nodes. On this basis, the system marks the nodes that successfully match the monitoring data as known nodes, and marks the nodes that fail to match or lack data as unknown nodes.

[0028] According to the monitoring data of the known nodes and their spatio-temporal relationship with the unknown nodes, the safety state of the known nodes is evaluated by a deep learning model, and the state of the unknown nodes is predicted and the missing data is completed, thereby constructing a continuous and complete product life cycle data chain.

[0029] In the embodiments of the present application, after identifying the known nodes and unknown nodes, the system uses a deep learning model to evaluate the safety state and complete the missing data. Specifically, the monitoring data of the known nodes is input and processed: time series sensor data is used to reflect the dynamic changes of environmental conditions (such as temperature, humidity, gas composition), image and video data are used to identify food appearance, packaging integrity, and processing behavior standardization, and structured text data is used to analyze quality inspection reports, logistics records, and supplier reputation information. The deep learning model fuses the above multi-modal features and outputs the comprehensive safety state evaluation result of the node.

[0030] According to the time sequence relationship and spatial relationship between the known nodes and the unknown nodes, the system extracts the multi-modal features and static attribute features of the adjacent nodes, constructs a joint representation vector with context semantics, and inputs it into a pre-trained prediction and completion model. The prediction and completion model based on the encoder-decoder architecture can generate the environmental data sequence, safety state score, and prediction confidence of the unknown node, thereby effectively completing the missing data link. Through the evaluation of the known nodes and the prediction of the unknown nodes, a continuous and complete product life cycle data chain is finally constructed, avoiding the problem of interruption of the traditional traceability information chain due to data missing, and providing data support for subsequent visualization and credibility evaluation.

[0031] Further, the safety state of the known nodes is evaluated by a deep learning model, including:

[0032] For time-series sensor data, the data is input into a rule learning module based on thresholds and durations to calculate temperature and humidity exceedance events and their cumulative impact, outputting quantitative evaluation indicators. For image and video data, the data is input into a pre-trained computer vision model to perform foreign object detection, lesion identification, packaging integrity analysis, or behavioral compliance analysis, outputting a safety score based on visual evidence. For structured text data, keyword extraction, anomaly record marking, and correlation analysis with supplier historical performance are performed to output text-level risk indicators. The quantitative evaluation indicators, the safety score based on visual evidence, and the text-level risk indicators are fused to generate a comprehensive safety score for the corresponding known nodes.

[0033] For time-series sensor data, environmental parameters such as temperature, humidity, and gas concentration are extracted and input into a rule learning module based on thresholds and duration. The rule learning module identifies events such as excessive temperature, excessive humidity, or abnormal gas concentration by combining preset threshold judgment with duration calculation. For example, when the temperature data continuously exceeds the 8°C threshold for cold chain storage and transportation for more than 30 minutes, it is determined as a high-risk event, and its duration and cumulative impact are calculated. The occurrence frequency, duration, and cumulative impact (the magnitude of the exceedance multiplied by the corresponding time interval) of all abnormal events are obtained. These three indicators are standardized to the same dimension and weighted according to preset weights to obtain a quantitative evaluation index.

[0034] For image and video data, the data is input into a pre-trained computer vision model based on a convolutional neural network. This model, through transfer learning on publicly available food testing datasets and enterprise-built quality testing sample sets, can identify common safety risks, including: foreign objects on food surfaces, mold or spoilage, damaged or leaking packaging, and operators not wearing protective equipment as required. The computer vision model outputs the location of the detection box, category label, and confidence score. The system calculates the visual safety score corresponding to each node based on the severity, coverage, and confidence score of the detection results.

[0035] For structured text data, key fields are extracted, including descriptions of detection results, records of abnormal events, and supplier identification information. The extracted text is then fed into a pre-trained natural language processing model based on BERT. This model first performs word segmentation and vectorization, then combines contextual semantics to identify anomalous expressions such as "unqualified," "exceeding standards," "contamination," and "return." For each detected anomalous term, the system assigns a score according to a preset risk level. Simultaneously, it queries the supplier's historical performance database to calculate its historical pass rate and applies a performance correction factor (1 - historical pass rate) to that node. The system multiplies the anomalous label score by the performance correction factor and sums them to output a text risk index.

[0036] The quantified evaluation indexes of the system time series sensor data, the security scores of the image and video data, and the risk indicators of the text data are fused to generate a comprehensive security score of the known node as the security state evaluation result of the node.

[0037] Further, the state prediction and data completion of the unknown node include:

[0038] Based on the unknown node, the multi-modal data features and static attribute features of the corresponding associated known nodes before and after the unknown node are extracted. According to the time interval and node association of the associated known nodes before and after the unknown node, the multi-modal data features and static attribute features are associated and fused to construct a joint representation vector, which is input into a pre-trained deep learning completion model to predict the environmental data sequence, security score, and prediction confidence of the missing node.

[0039] For the life cycle link marked as an unknown node, the system automatically retrieves the adjacent known nodes before and after the unknown node, extracts the corresponding multi-modal data features such as the time series trend features of the sensors, the environmental and operation features recognized in the images and videos, and the semantic information in the text data, and extracts the static attribute features of the node such as the food category, geographical location attribute, and expected process type. Then, the system calculates the association degree of the unknown node and the known nodes according to the time interval and spatial order relationship between the associated known nodes before and after the unknown node, and applies the association weight to the multi-modal features and static attribute features for cross-modal fusion processing to construct a joint representation vector with context semantics. The joint representation vector is input into a pre-trained deep learning completion model, which is trained based on an encoder-decoder architecture and can generate the environmental data sequence of the missing node while maintaining the continuity of the time series, and output the corresponding security state prediction score and prediction confidence. In this way, the system can effectively complete the data breakpoints caused by missing collection or transmission abnormalities, keep the product life cycle data chain continuous and complete, and improve the integrity and reliability of the food safety traceability result.

[0040] Further, the deep learning completion model is an encoder-decoder architecture, the encoder is used for feature encoding and fusion of the known context multi-modal data, and the decoder is used for generating the prediction data sequence of the unknown node; wherein the training target of the deep learning completion model includes the regression loss between the prediction data sequence and the real data sequence, and the classification loss between the prediction node security state and the real node security state.

[0041] The deep learning completion model adopts an encoder-decoder architecture; the encoder part is used to receive and process multi-modal data features and static attribute features from the known nodes associated before and after, including dynamic change features of time series sensors, visual semantic features of image videos, and semantic label features of structured text; the encoder extracts high-dimensional features and fuses cross-modalities through multi-layer neural networks, thereby forming a latent vector representation that can represent the context relationship; the decoder part gradually generates the predicted output of the unknown node based on the latent representation, including the predicted environmental data sequence (such as temperature and humidity change curve, detection index value), the predicted comprehensive safety state score, and the confidence index corresponding to the prediction result.

[0042] During the training process, the optimization target of the deep learning completion model not only considers the accuracy of numerical prediction, but also combines the classification accuracy of the safety state. Specifically, the loss function of the deep learning completion model consists of two parts: on the one hand, for the difference between the predicted data sequence and the real collected data sequence, the regression loss is calculated using mean square error or mean absolute error to ensure the continuity and approximation of environmental data prediction; on the other hand, for the difference between the predicted node safety state (such as safe, at risk, unqualified) and the real labeled state, the cross-entropy loss or weighted classification loss is used for constraint to improve the reliability of safety state prediction.

[0043] Further, the multi-modal data features and static attribute features are associated and fused to construct a joint representation vector, including:

[0044] According to the association time interval of the known nodes associated before and after, the attention mechanism is introduced based on the time interval between the predecessor node, the posterior node and the unknown node, and the association weight is dynamically calculated, wherein the node data with small time interval is given high attention weight; based on the association weight, the extracted multi-modal features are aligned and cross-modal fused to generate a unified joint representation vector with context semantics.

[0045] After the system identifies the unknown node, it locates the known nodes associated before and after, and calculates the time interval between these known nodes and the target unknown node. When the time interval between a known node and the unknown node is small, it means that its state has a stronger influence on the unknown node, and the system will allocate higher attention weight to the node data; on the contrary, the known node data with larger time interval is allocated lower weight.

[0046] The system utilizes the correlation weight calculated dynamically to weight the multi-modal features of the known nodes before and after, and aligns the features in the cross-modal fusion module. The final generated joint representation vector not only contains the comprehensive expression of multi-modal features, but also carries the temporal dependence and semantic association of the context, thereby providing a more accurate input representation for the deep learning completion model and improving the reliability of unknown node state prediction and data completion.

[0047] According to the product life cycle data chain, the food safety status of each node is visualized and converted, and a visual traceability report is generated.

[0048] In the embodiments of the present application, after completing the safety state evaluation of the known nodes and the data prediction and completion of the unknown nodes, the food safety status of each node is visualized and converted based on the generated product life cycle data chain, and a visual traceability report is output. Specifically, the system arranges the nodes in order on the time axis to intuitively present the whole life cycle process of food from raw material production, processing, packaging, storage, transportation to sales. For known nodes with complete data, solid color blocks are used for representation, in which the hue of the color is used to distinguish different safety levels (for example, green represents safety, yellow represents a slight risk, and red represents unqualified), and the brightness of the color is used to represent the data richness of the node, and the more complete the data is, the brighter it is displayed. For unknown nodes with missing data and completed by the model, hollow color blocks are used for representation, and transparency is introduced in the color system, and the higher the transparency is, the lower the prediction confidence is, so that the user can intuitively distinguish the credibility of the predicted data.

[0049] Further, according to the product life cycle data chain, the food safety status of each node is visualized and converted, and a visual traceability report is generated, including:

[0050] Solid color blocks and a first color system are used to represent known nodes on the time axis, the hue represents the safety level, and the brightness represents the data richness; hollow color blocks and a second color system are used to represent unknown nodes, the hue represents the predicted safety level, and the visual transparency represents the prediction confidence, the lower the confidence is, the more transparent it is, the corresponding data of the nodes in the product life cycle data chain is visualized and converted, and the visual traceability report is generated; wherein, in response to the user's click operation on any node, the down-drilling is performed and the parallel display of all multi-modal original data and module analysis results of the corresponding node is performed; the feedback behavior of the user in each traceability request is recorded, including the confirmation or questioning operation of the prediction result of the unknown node, and the feedback monitoring data, for the user's questioning operation combined with the feedback monitoring data, the corresponding prediction node and data are located, the feedback monitoring data is added to the retraining data set of the model as incremental data, and the model is iteratively optimized.

[0051] Specifically, the system sequentially displays each node of the food life cycle on a time axis. For known nodes, solid color blocks are used to identify the nodes with a first color system, in which the hue represents the food safety level of the node (e.g., green represents safety, yellow represents a certain risk, and red represents unqualified), and the lightness is used to reflect the completeness and richness of the monitoring data of the node. For unknown nodes, hollow color blocks are used to identify the nodes with a second color system, in which the hue represents the predicted safety level, and the visual transparency is used to represent the prediction confidence. The lower the confidence, the higher the transparency, so that the user can intuitively distinguish the credibility of the prediction results.

[0052] In addition, the visualized traceability report also supports user interaction operations. When the user clicks on any node, the system can perform a drill-down operation and display the multi-modal original data (including sensor records, image and video files, and structured text information) corresponding to the node and the processing results of each analysis module side by side, helping the user to freely switch between macro time axis display and micro data level.

[0053] The system records the feedback behavior of the user in each traceability request process, including the user's confirmation operation or questioning operation on the prediction result of the unknown node, and the uploaded supplementary monitoring data. When the system detects that the user has questioned a certain predicted node, it will automatically locate the corresponding predicted node in combination with the supplementary monitoring data fed back by the user, and add the data as an incremental sample to the retraining data set of the model, thereby realizing the continuous iterative optimization of the deep learning model.

[0054] Further, the visualized traceability report visualizes the food safety status of each node in the form of a time axis, and distinguishes the visual forms of known nodes and unknown nodes, and visualizes the traceability confidence calculated based on the data of each node.

[0055] The visualization traceability report adopts a time axis display form, sequentially arranges and dynamically displays each node in the whole life cycle of the target food, and enables the user to intuitively master the whole process of the food from raw materials, processing, packaging, transportation to sales. The system distinguishes and displays the forms of known nodes and unknown nodes when visualizing: the known nodes are displayed by solid color blocks, the color represents the food safety level, and the lightness reflects the completeness of the monitoring data; the unknown nodes are displayed by hollow color blocks, the color represents the predicted safety level, and the transparency corresponds to the confidence of the prediction result, the higher the transparency, the lower the prediction reliability. In addition, the system not only displays the safety state of each node on the time axis, but also further visualizes and presents the traceability reliability calculated based on the node data. The traceability reliability comprehensively considers factors such as data completeness, model prediction confidence and cross-node consistency, and is marked on the time axis in a numerical or graphical way, so that the user can quickly evaluate the reliability of the whole life cycle traceability chain in the overall level, and identify the links with higher risk or lower reliability in the local level, to realize more scientific and transparent food safety traceability.

[0056] In summary, the embodiments of the present application have at least the following technical effects:

[0057] First, in response to the traceability request of the target food, the whole life cycle multi-modal data associated with the food unique identifier is obtained, and the multi-modal data includes time series sensor data, image video data and structured text data. Next, based on the pre-defined multi-level node template, the whole life cycle of the target food is divided into multiple nodes, and the mapping relationship between the multi-modal data and each node is established to identify the known nodes with complete data and the unknown nodes with missing data. Then, according to the monitoring data of the known nodes and the spatio-temporal relationship with the unknown nodes, the safety state of the known nodes is evaluated by a deep learning model, and the state of the unknown nodes is predicted and the data is completed to build a continuous and complete product life cycle data chain. Finally, the food safety state of each node is visualized and converted according to the product life cycle data chain to generate a visualization traceability report. The technical problem of insufficient traceability result reliability caused by incomplete food safety traceability data in the prior art is solved, and the technical effect of improving traceability completeness and reliability by building a product life cycle data chain is achieved.

[0058] Embodiment two, based on the same invention concept as the food safety traceability method based on deep learning in the foregoing embodiments, as Figure 2 shown, the present application provides a food safety traceability system based on deep learning, wherein the system comprises:

[0059] The data acquisition unit 11 acquires full-life-cycle multi-modal data associated with a food unique identifier in response to a traceability request of a target food, the multi-modal data including time series sensor data, image / video data and structured text data; the relationship establishment unit 12 divides the full-life-cycle of the target food into multiple nodes based on a predefined multi-level node template, and establishes a mapping relationship between the multi-modal data and each node to identify known nodes with complete data and unknown nodes with missing data; the evaluation unit 13 evaluates the safety state of the known nodes through a deep learning model based on the monitoring data of the known nodes and their spatio-temporal relationship with the unknown nodes, and predicts the state and completes the data of the unknown nodes to construct a continuous and complete product life-cycle data chain; the visualization unit 14 visualizes the food safety state of each node based on the product life-cycle data chain and generates a visualized traceability report.

[0060] Further, the visualization unit 14 is configured to perform the following method:

[0061] The visualized traceability report visualizes the food safety state of each node in the form of a time axis, and distinguishes and displays the visual forms of the known nodes and the unknown nodes, and visualizes the traceability reliability calculated based on the data of each node.

[0062] Further, the relationship establishment unit 12 is configured to perform the following method:

[0063] The target food traceability dataset is acquired, and the traceability dataset includes full-life-cycle division nodes, cycle attributes and state data of multiple foods; the life-cycle division nodes in the traceability dataset are subjected to cluster analysis to obtain a general node sequence representing different food categories; based on the effective coverage rate of the cluster results, a hierarchical tree structure is constructed for the general node sequence, wherein a high-level node covers a wide food category, and a low-level node corresponds to a subdivided food category; an abnormal node that cannot be clustered into the general node sequence is inserted into the bottom layer of the tree structure as a special leaf node, and the multi-level node template is constructed, wherein each node in the template is attached with a food attribute label set that can be matched.

[0064] Further, the relationship establishment unit 12 is configured to perform the following method:

[0065] Based on the timestamp, location coordinate and operation type fields in the multi-modal data, the expected time and operation type of a specific node in the multi-level node template are matched; based on the matching relationship, the data matched successfully for the specific node is marked as known monitoring data of the corresponding node, and a mapping relationship between the multi-modal data and each node is established; based on the mapping relationship, the matched node is marked as a known node, and the unmatched node is marked as an unknown node.

[0066] Further, the evaluation unit 13 is configured to perform the following method:

[0067] For time series sensor data, input into the threshold and duration-based rule learning module, calculate temperature, humidity exceeding event and its cumulative impact, output quantitative evaluation index; for image video data, input into the pre-trained computer vision model, perform foreign matter detection, lesion identification, packaging integrity analysis or behavior compliance analysis, output safety score based on visual evidence; for structured text data, perform keyword extraction, abnormal record marking and supplier historical performance correlation analysis, output risk indicators at the text level; fuse the quantitative evaluation index, the safety score based on visual evidence, the risk indicators at the text level, generate a comprehensive safety score corresponding to the known node.

[0068] Further, the evaluation unit 13 is configured to perform the following method:

[0069] Based on the unknown node, the multi-modal data features and static attribute features of the corresponding associated known nodes before and after are extracted; according to the association time interval of the associated known nodes before and after and the node association, the multi-modal data features and static attribute features are associated and fused to construct a joint representation vector, which is input into a pre-trained deep learning completion model to predict the environment data sequence, safety score and prediction confidence of the missing node.

[0070] Further, the evaluation unit 13 is configured to perform the following method:

[0071] The deep learning completion model is an encoder-decoder architecture, the encoder is configured to encode and fuse the multi-modal data of the known context, and the decoder is configured to generate the predicted data sequence of the unknown node; wherein the training target of the deep learning completion model includes the regression loss between the predicted data sequence and the real data sequence, and the classification loss between the predicted node safety state and the real node safety state.

[0072] Further, the evaluation unit 13 is configured to perform the following method:

[0073] According to the association time interval of the associated known nodes before and after, the attention mechanism is introduced based on the time interval of the predecessor node, the posterior node and the unknown node, and the association weight is dynamically calculated, wherein the node data with small time interval is given high attention weight; based on the association weight, the extracted multi-modal features are aligned and cross-modal fused to generate a unified joint representation vector with context semantics.

[0074] Further, the visualization unit 14 is configured to perform the following method:

[0075] The known nodes are characterized on a time axis using solid color blocks and a first color system, in which hue represents a security level and lightness represents data richness; unknown nodes are characterized using hollow color blocks and a second color system, in which hue represents a predicted security level and visual transparency represents a prediction confidence, and the lower the confidence, the more transparent; the nodes in the product life cycle data chain are subjected to corresponding data visualization conversion, and the visualization traceability report is generated; wherein, further comprising: in response to a user's click operation on any node, performing downward drilling and displaying all multi-modal original data and module analysis results of the corresponding node side by side; recording the feedback behavior of the user in each traceability request, including confirmation or questioning operation on the prediction result of the unknown node, and feedback monitoring data; for the user's questioning operation, combining the feedback monitoring data, positioning the corresponding prediction node and data, and adding the feedback monitoring data as incremental data to the retraining data set of the model, and iteratively optimizing the model.

[0076] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. The above describes a specific embodiment of the present application. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0077] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0078] The present specification and drawings are only exemplary descriptions of the present application, and are considered to cover any and all modifications, changes, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the scope of the present application. Thus, if these modifications and changes of the present application belong to the scope of the present application and its equivalents, the present application intends to include these modifications and changes.

Claims

1. A food safety traceability method based on deep learning, characterized in that, The method includes: In response to the traceability request of the target food, acquire full life cycle multimodal data associated with the food's unique identifier, including time-series sensor data, image and video data, and structured text data; Based on a predefined multi-level node template, the entire life cycle of the target food is divided into multiple nodes, and a mapping relationship between the multimodal data and each node is established to identify known nodes with complete data and unknown nodes with missing data. Based on the monitoring data of known nodes and their spatiotemporal relationship with unknown nodes, a deep learning model is used to evaluate the safety status of known nodes and to predict the status and complete the data of unknown nodes, thus constructing a continuous and complete product lifecycle data chain. Based on the product lifecycle data chain, the food safety status of each node is visualized and transformed to generate a visual traceability report. Based on a predefined multi-level node template, the entire lifecycle of the target food is divided into multiple nodes, including: Obtain the target food traceability dataset, which contains the full life cycle division nodes, periodic attributes, and status data of various foods; Cluster analysis was performed on the life cycle segmentation nodes in the traceability dataset to obtain a general node sequence representing different food categories; Based on the effective coverage of the clustering results, a hierarchical tree structure is constructed for the general node sequence, in which high-level nodes cover a wide range of food categories, and low-level nodes correspond to specific food categories. Abnormal nodes that cannot be clustered into the general node sequence are inserted as special leaf nodes at the bottom level of the tree structure to construct the multi-level node template, wherein each node in the template is accompanied by a set of food attribute tags that can be used for matching. Security status evaluation of known nodes is performed using deep learning models, including: For time-series sensor data, the data is input into a rule learning module based on thresholds and durations to calculate temperature and humidity exceedance events and their cumulative impact, and output quantitative evaluation indicators. For image and video data, the data is input into a pre-trained computer vision model to perform foreign object detection, lesion identification, packaging integrity analysis, or behavioral compliance analysis, and output a safety score based on visual evidence. For structured text data, keyword extraction, anomaly record marking, and correlation analysis with supplier historical performance are performed to output text-level risk indicators; By integrating the aforementioned quantitative evaluation indicators, the security score of visual evidence, and the risk indicators at the text level, a comprehensive security score for the corresponding known node is generated. Perform state prediction and data completion for unknown nodes, including: Extract multimodal data features and static attribute features of known nodes that are related to unknown nodes; Based on the association time interval and node correlation of known nodes, the multimodal data features and static attribute features are associated and fused to construct a joint representation vector, which is then input into a pre-trained deep learning completion model to predict the environmental data sequence, security score, and prediction confidence of missing nodes.

2. The food safety traceability method based on deep learning according to claim 1, characterized in that, The visual traceability report displays the food safety status of each node in a timeline format, distinguishes between the visualization of known and unknown nodes, and visually presents the traceability credibility calculated based on the data of each node.

3. The food safety traceability method based on deep learning according to claim 1, characterized in that, Establishing a mapping relationship between the multimodal data and each node to identify known nodes with complete data and unknown nodes with missing data includes: Based on the timestamp, location coordinates, and operation type fields in the multimodal data, the expected time and operation type of a specific node in the multi-level node template are matched. Based on the matching relationship, data that successfully matches a specific node is marked as the known monitoring data of the corresponding node, and a mapping relationship between the multimodal data and each node is established; Based on the mapping relationship, nodes that match successfully are marked as known nodes, and nodes that do not match successfully are marked as unknown nodes.

4. The food safety traceability method based on deep learning according to claim 1, characterized in that, The deep learning completion model is an encoder-decoder architecture, where the encoder is used to encode and fuse features of multimodal data with known context, and the decoder is used to generate predicted data sequences for unknown nodes. The training objectives of the deep learning completion model include both the regression loss between the predicted data sequence and the real data sequence, and the classification loss between the predicted node safety state and the real node safety state.

5. The food safety traceability method based on deep learning according to claim 1, characterized in that, The multimodal data features and static attribute features are correlated and fused to construct a joint representation vector, including: Based on the association time interval of known nodes, an attention mechanism is introduced to dynamically calculate the association weight based on the time interval between the predecessor node, successor node and unknown node, where node data with shorter time intervals are given higher attention weights. Based on the association weights, the extracted multimodal features are aligned and fused across modalities to generate a unified joint representation vector with contextual semantics.

6. The food safety traceability method based on deep learning according to claim 1, characterized in that, Based on the product lifecycle data chain, the food safety status of each node is visualized and transformed to generate a visualized traceability report, including: Solid color blocks and a first color system are used to represent known nodes on the timeline, with hue representing the security level and brightness representing the data richness; hollow color blocks and a second color system are used to represent unknown nodes, with hue representing the predicted security level and visual transparency representing the prediction confidence level. The lower the confidence level, the more transparent the data. Corresponding data visualization transformations are performed on the nodes in the product lifecycle data chain to generate the visualization traceability report. This also includes: responding to a user's click on any node, performing drill-down and displaying all multimodal raw data and module analysis results of the corresponding node side-by-side; Record user feedback behavior in each tracing request, including confirmation or questioning of prediction results for unknown nodes, as well as feedback monitoring data. For user questioning, combine the feedback monitoring data to locate the corresponding prediction node and data, and add the feedback monitoring data as incremental data to the model's retraining dataset for feedback iterative optimization.

7. A food safety traceability system based on deep learning, characterized in that, The system is used to implement the deep learning-based food safety traceability method according to any one of claims 1-6, the system comprising: Data acquisition unit: In response to the traceability request of the target food, acquire full life cycle multimodal data associated with the unique identifier of the food, the multimodal data including time series sensor data, image and video data and structured text data; Relationship Establishment Unit: Based on a predefined multi-level node template, the entire life cycle of the target food is divided into multiple nodes, and the mapping relationship between the multimodal data and each node is established to identify known nodes with complete data and unknown nodes with missing data. Evaluation Unit: Based on the monitoring data of known nodes and their spatiotemporal relationship with unknown nodes, a deep learning model is used to evaluate the safety status of known nodes and to predict the status and complete the data of unknown nodes, thus constructing a continuous and complete product lifecycle data chain. Visualization Unit: Based on the product lifecycle data chain, the food safety status of each node is visualized and transformed to generate a visual traceability report.

Images