Food processing quality analysis and evaluation method and system based on multi-modal data

Abstract

This application discloses a method and system for food processing quality analysis and evaluation based on multimodal data. The method includes: establishing a process segment data table according to the production line process flow, dividing the operating time into time slices, and collecting equipment operating parameters, batch and formula information, online detection results, and image features to form a multimodal data record; generating a material transit time allocation matrix based on equipment operating status, start / stop logs, speed changes, and batch boundaries, and applying corresponding constraints; aggregating upstream data corresponding to the exit inspection samples based on matrix correlation strength, constructing training samples and inputting them into a quality assessment model, outputting instantaneous quality scores, defect category probabilities, and confidence levels; simultaneously constructing sample packages based on delayed quality inspection results, calculating sample contribution probabilities, generating composite labels and label confidence levels to update the quality assessment model and the material transit time allocation matrix. This solution can improve the accuracy of food processing quality evaluation.

Description

Technical Field

[0001] This application relates to the field of food processing quality testing and intelligent evaluation technology, specifically to a food processing quality analysis and evaluation method and system based on multimodal data. Background Technology

[0002] Food processing lines, whether continuous or semi-continuous, typically include multiple stages such as feeding, heating, heat preservation, cooling, forming, testing, and packaging. Existing production sites are capable of collecting data such as equipment operating parameters, batch formula information, product images, online testing results, and delayed quality inspection results through PLCs, SCADA systems, MES, industrial cameras, online detection devices, and quality management systems. This data is then used for product quality monitoring and anomaly traceability.

[0003] However, in actual production, the products detected at the export point do not correspond to the upstream process data at a fixed moment, but are formed after continuous processing through multiple previous processes. When the production line slows down, stops, resumes operation, or switches batches, the actual time it takes for the material to travel from the upstream process to the export inspection point will change. If the export inspection results are still associated with the upstream data using a fixed timestamp or a fixed delay value, it is easy to cause deviations in the sample correspondence and affect the accuracy of the quality assessment results.

[0004] Furthermore, delayed quality inspection results such as microbial testing, physicochemical testing, and sample retesting usually only correspond to batches, production time ranges, or packaging labels, making it difficult to directly pinpoint individual export test samples. Existing technologies directly backfill delayed abnormal results to all samples in the same batch or within the same time period, which can easily lead to actual qualified samples being incorrectly labeled, reducing the credibility of training labels. Summary of the Invention

[0005] This application provides a method and system for food processing quality analysis and evaluation based on multimodal data, in order to at least solve some of the technical problems existing in the related technologies described above.

[0006] According to a first aspect of the embodiments of this application, a method for food processing quality analysis and evaluation based on multimodal data is provided, including: Based on the process flow configuration information of the production line, a process segment data table is established, and the running time of each process segment is divided into time slices according to a fixed sampling interval. Equipment operating parameters, batch and formula information, online detection results and image features are collected for each time slice to form a multimodal data record. Based on equipment operation data, equipment start-up and shutdown logs, speed change records, and batch boundaries, a material passage time allocation matrix is ​​generated and time-series constraints, batch boundary constraints, no-material-passage constraints, and normalization constraints are applied. Based on the correlation strength of the material passing through the time allocation matrix, multimodal data records of the corresponding upstream time slices are aggregated for the exit inspection samples to construct multimodal training samples; the multimodal training samples are input into the quality assessment model to output instantaneous quality scores, defect category probabilities, and confidence levels; A sample package is constructed based on the delayed quality inspection results. The contribution probability of each outbound inspection sample in the sample package is calculated, and a composite label and label credibility are generated based on the contribution probability. The quality assessment model and the material time allocation matrix are updated with the composite label and label credibility.

[0007] As an optional approach, generating the material transit time allocation matrix includes: using the exit inspection sample as the matrix rows and the upstream time slice as the matrix columns; calculating the baseline transit time of the conveyor belt conveying process segment based on the conveying distance and conveyor belt speed; statistically analyzing the dwell time range of equipment with material retention characteristics based on historical stable production data; determining the upstream time slice subset corresponding to each exit inspection sample by combining shutdown events, speed reduction events, resumption of operation events, and batch switching events; assigning initial correlation strength and normalizing it to the time slices within the upstream time slice subset, and assigning a value of zero to the time slices outside the upstream time slice subset.

[0008] As an optional approach, applying timing constraints, batch boundary constraints, no-material-passage constraints, and normalization constraints includes: establishing a mask matrix with the same dimension as the material passage time allocation matrix; marking upstream time slices whose time is later than the detection time of the exit detection sample, upstream time slices whose batch number is different from the exit detection sample, and upstream time slices in a state of equipment shutdown and no material passage as restricted time slices; in each training iteration, multiplying the material passage time allocation matrix element by element by the mask matrix, and performing normalization processing on the matrix rows.

[0009] As an optional approach, constructing the multimodal training samples includes: reading the matrix rows in the material passage time allocation matrix corresponding to the exit detection samples, and extracting upstream time slices with non-zero correlation strength; arranging the multimodal data records of the upstream time slices in chronological order to form a time slice sequence, and binding the correlation strength to the corresponding time slice in the time slice sequence; and merging the time slice sequence, exit detection data, and traceability information into the multimodal training samples.

[0010] As an optional approach, the quality assessment model includes an input projection module, a location encoding module, a weighted attention encoding module, an exit feature fusion module, and an output head module connected in sequence. The input projection module concatenates and maps the numerical device parameter vector, image feature vector, and discrete information embedding vector of each time slice into a sequence element representation. The weighted attention encoding module incorporates the correlation strength into the attention score normalization process. The exit feature fusion module performs weighted pooling on the encoded sequence representation according to the correlation strength and concatenates it with the exit detection data and the traceability information to generate a fusion vector. The output head module outputs the instantaneous quality score, the defect category probability, and the confidence level based on the fusion vector.

[0011] As an optional approach, updating the quality assessment model and the material transit time allocation matrix includes: calculating the immediate label loss based on the online detection labels and the delayed label loss based on the delayed quality inspection labels; weighting and summing the immediate label loss and the delayed label loss according to preset weight coefficients to obtain the total loss; jointly updating the model parameters of the quality assessment model, the material transit time allocation matrix, and the contribution probability based on the total loss; and after updating the material transit time allocation matrix, re-performing the element-wise multiplication of the mask matrix and the normalization of the matrix rows.

[0012] As an optional approach, constructing the sample package based on the delayed quality inspection results includes: reading the inspection item name, inspection result value, judgment conclusion, corresponding batch number, corresponding production time range, and packaging identification from the delayed quality inspection results; filtering export inspection samples based on the corresponding batch number and the corresponding production time range to obtain a candidate sample set; retaining the export inspection samples that match the packaging identification when the packaging identification exists; reading the matrix row of each export inspection sample in the candidate sample set in the material transit time allocation matrix and extracting the multimodal data record of its associated upstream time slice; and combining the retained export inspection samples, the corresponding multimodal data record, and the judgment conclusion to form the sample package.

[0013] As an optional approach, calculating the contribution probability includes: inputting the quality assessment model into each outgoing detection sample within the sample packet to obtain the predicted output and fusion vector of that outgoing detection sample; inputting the fusion vector into independent fully connected branches to obtain the anomaly score of that outgoing detection sample; taking the exponent value of all anomaly scores within the sample packet, and dividing the exponent value of each anomaly score by the sum of the exponent values ​​of all anomaly scores to obtain the contribution probability of each outgoing detection sample; weighting and aggregating the predicted outputs of each outgoing detection sample according to the contribution probability to obtain the packet-level prediction result; and calculating the delay label loss based on the packet-level prediction result and the judgment conclusion.

[0014] As an optional approach, generating composite labels and label credibility includes: writing the immediate quality status, delayed quality inspection status, label source, and label credibility into the composite label; setting the label credibility to a preset benchmark value when online detection is qualified and no delayed quality inspection results cover it; adjusting the label credibility based on the comparison result of the contribution probability and a preset threshold when online detection is qualified and delayed quality inspection is abnormal; reducing the weight of defect labels and retaining the corresponding exit detection samples when online detection is abnormal and delayed review is qualified; and multiplying the loss value of each exit detection sample by its label credibility and including it in the total loss during training.

[0015] According to a second aspect of the embodiments of this application, a food processing quality analysis and evaluation system based on multimodal data is also provided, comprising: The data recording module is configured to establish a process segment data table based on the process flow configuration information of the production line, and divide the running time of each process segment into time slices according to a fixed sampling interval, and collect equipment operating parameters, batch and formula information, online detection results and image features for each time slice to form multimodal data records; The matrix generation module is configured to generate a material passage time allocation matrix and apply timing constraints, batch boundary constraints, no material passage constraints, and normalization constraints based on equipment operation data, equipment start-stop logs, speed change records, and batch boundaries. The training sample construction and quality assessment module is configured to construct multimodal training samples by aggregating multimodal data records of the upstream time slices corresponding to the material passing through the time allocation matrix based on the correlation strength of the material passage time. The multimodal training samples are then input into the quality assessment model, which outputs an instantaneous quality score, defect category probability, and confidence level. The delayed quality inspection processing and updating module is configured to construct a sample package based on the delayed quality inspection results, calculate the contribution probability of each outbound inspection sample in the sample package, and generate a composite label and label credibility based on the contribution probability; and update the quality assessment model and the material time allocation matrix with the composite label and label credibility.

[0016] This application establishes a material transit time allocation matrix, associating the exit inspection samples with the time slices of each upstream process segment according to the actual material flow relationship. During the training process, it continuously applies time sequence, batch boundary, shutdown without material passage, and normalization constraints, enabling the sample construction to adapt to actual working conditions such as speed changes, shutdowns, resumption of operation, and batch switching, reducing mismatches between exit quality results and upstream process data. Based on the above, equipment parameters, batch formulas, online inspection results, and image features are uniformly used for quality assessment, and upstream process data is weighted in conjunction with correlation strength, improving the pertinence of real-time quality scoring and defect judgment. For delayed quality inspection results such as physicochemical and microbiological testing, this application calculates the contribution probability of each exit inspection sample through sample packages, generating composite labels and label credibility, avoiding the coarse backfilling of batch-level abnormal results to all samples, which would cause label contamination. After the quality assessment model and the material transit time allocation matrix are jointly updated, the traceability relationship and model parameters can be corrected using new quality inspection feedback during continuous production, thereby improving the accuracy and stability of food processing quality assessment.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Furthermore, no embodiment in this disclosure is required to achieve all the effects described above. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 A flowchart illustrating a food processing quality analysis and evaluation method based on multimodal data, provided in this embodiment of the disclosure.

[0020] Figure 2 A flowchart illustrating the time allocation matrix for the generated materials provided in this embodiment of the disclosure.

[0021] Figure 3 This is a schematic diagram of the quality assessment model structure provided in an embodiment of this disclosure.

[0022] Figure 4 A flowchart for calculating the contribution probability of a detection sample provided in an embodiment of this disclosure.

[0023] Figure 5 This is a schematic block diagram of a food processing quality analysis and evaluation system based on multimodal data, provided as an embodiment of the present disclosure.

[0024] Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] The food processing quality analysis and evaluation method based on multimodal data provided in this disclosure is applicable to continuous or semi-continuous food processing production lines with multiple process segments operating in series. Typical scenarios cover combined production lines with processes such as baking, frying, steaming, drying, cooling, shaping, and sterilization. The data sources required by this method include equipment operation data recorded by programmable logic controllers (PLCs) or supervisory control and data acquisition and monitoring systems (SCADA), batch and formula information recorded by manufacturing execution systems (MES), product images captured by industrial cameras, output results of existing testing devices such as online weighing, moisture detection, metal detection, and seal detection, as well as physicochemical testing, microbiological testing, sample re-inspection, and customer return re-inspection results generated in laboratory information management systems (LIMS) or quality management systems (QMS).

[0027] The implementation process of the method described in this application will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the scope of protection of this application.

[0028] Please see Figure 1 , Figure 1 This is a flowchart of a food processing quality analysis and evaluation method based on multimodal data according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes steps S1-S4: In step S1, a process segment data table is established based on the process flow configuration information of the production line, and the running time of each process segment is divided into time slices according to a fixed sampling interval. Equipment operating parameters, batch and formula information, online detection results and image features are collected for each time slice to form a multimodal data record.

[0029] In some embodiments, the system establishes a process segment data table based on the process flow configuration information of the production line. Each process segment corresponds to a physically continuous processing area in the production line. The data table records the identifier, start position, end position, conveying method and equipment type of each process segment. Taking a baking production line as an example, the process segments can be divided into feeding segment, preheating segment, heating segment, heat preservation segment, cooling segment, forming segment, and pre-packaging inspection segment, etc.

[0030] Based on this, the system divides the running time of each process segment into time slices according to a fixed sampling interval. The sampling interval can be configured to be consistent with the collection cycle of the existing data acquisition system. Each time slice has a unique identifier and records the process segment identifier, start and end timestamps, and batch number.

[0031] For example, for each time slice, the following types of data are collected to form a multimodal data record for that time slice: In terms of equipment operating parameters, numerical parameters such as conveyor belt speed, heating temperature, pressure, power, rotation speed, and start / stop status are combined into a numerical equipment parameter vector; For image features, color histogram distribution, surface texture description, edge integrity, product area ratio, and damaged area ratio can be extracted from images collected by industrial cameras to form an image feature vector; Optionally, high-level semantic features can also be obtained through a pre-trained convolutional neural network (CNN).

[0032] Discrete fields in batch and formula information, including formula number, equipment number, shift number, and raw material batch number, are converted into discrete information embedding vectors by the system, with configurable embedding dimensions. Online detection results include weighing values, moisture values, metal detection results, and sealing detection results, among which weighing values ​​and moisture values ​​are continuous detection quantities, while metal detection results and sealing detection results are discrete detection quantities. After numericalization or embedding processing, they form an online detection result vector.

[0033] The aforementioned numerical device parameter vectors, image feature vectors, discrete information embedding vectors, and online detection results are stored in association with a unique identifier for the same time slice, constituting a multimodal data record for that time slice.

[0034] In step S2, based on equipment operation data, equipment start-up and shutdown logs, speed change records, and batch boundaries, a material passage time allocation matrix is ​​generated and time-series constraints, batch boundary constraints, no-material-passage constraints, and normalization constraints are applied.

[0035] In a continuous food processing line, a product detected at the exit at a certain moment has actually undergone multiple processing stages, including feeding, heating, heat preservation, and cooling, before reaching the exit inspection point. If the production line experiences speed changes, short shutdowns, or batch switching during this period, the actual transit time of the material from upstream to the exit will change accordingly. Traditional methods map the exit inspection results to upstream process data using fixed timestamps or fixed delay values. In the above scenario, this can cause the exit quality results to be associated with irrelevant upstream data, resulting in a mismatch between the process data in the training samples and the actual materials. The material transit time allocation matrix is ​​used to quantify the correlation between each exit inspection sample and each upstream time slice, transforming this correspondence from a fixed value into a soft correlation that can be adjusted according to the production line's operating status.

[0036] The material processing time allocation matrix generated in this embodiment is arranged with the export test sample as the row and the upstream time slice as the column. Each matrix element takes a value between 0 and 1, which represents the correlation strength between the corresponding export sample and the corresponding upstream time slice. The larger the value, the more likely the material of the export sample is to come from the processing stage corresponding to the upstream time slice. The sum of all elements in the same row is equal to 1.

[0037] Specifically, please refer to Figure 2 , Figure 2 A flowchart illustrating the time allocation matrix of the generated materials provided in this embodiment of the present disclosure is shown, as follows: Figure 2 As shown in box 201, the reference transit time and the range of material dwell time on the conveyor belt conveying process segment are calculated.

[0038] The system reads equipment operation data segment by segment and calculates the reference time for material transit for different equipment types. For conveyor belt conveying segments, the reference transit time is obtained based on the ratio of the conveying distance to the conveyor belt speed. For equipment with material retention characteristics, such as screw extruders, mixing tanks, cooking equipment, and drying equipment, the material residence time inside the equipment is affected by multiple factors such as rotational speed, feed rate, and equipment volume, making it difficult to calculate accurately using a single parameter. Based on the accumulated operating records of the equipment under historical stable production conditions, the system statistically obtains the empirical distribution of residence time and takes the upper and lower percentile values ​​of this distribution as the initial boundaries of the residence time range, optionally using the 5th percentile and 95th percentile as boundary values. The cumulative transit time of material from an upstream segment to the outlet detection point is equal to the sum of the reference transit times of each intermediate segment.

[0039] In box 202, the upstream time slice subset is determined by combining shutdown events, speed reduction events, resumption of operation events, and batch switching events. The aforementioned baseline elapsed time reflects the estimated value under steady-state operation; once the production line operating status changes, it needs to be corrected event by event.

[0040] Specifically, the system reads the equipment start / stop logs and speed change records, and processes the following four types of events in chronological order: When a shutdown event occurs, the shutdown period is marked as a state of no material passing through, and this period is not included in the effective material propulsion time, but is still retained in the natural time axis to estimate the arrival time of the exit detection sample, because no new material enters or propagates into the equipment during the shutdown period; When a speed reduction event occurs, the elapsed time within the affected time slice is recalculated based on the actual speed after the speed reduction; When a resumption of operation event occurs, the elapsed time range corresponding to the affected exit sample is recalculated based on the resumption time and the actual speed after resumption; When a batch switching event occurs, the batch boundary is marked according to the batch start and end times recorded in the MES, and this boundary will be used in subsequent constraints.

[0041] After the above event-by-event correction, the system determines the corresponding upstream time slice subset for each export inspection sample; the time slices in this subset meet three conditions: they are within the corrected elapsed time range, belong to the same batch as the export sample, and are in the period when the production line is actually running rather than shut down.

[0042] In box 203, initial correlation strengths are assigned and normalized for time slices within the upstream time slice subset. For each exit detection sample, the system assigns initial correlation strengths to time slices within its upstream time slice subset. In one implementation, the estimated transit time center is used as the peak value, and initial values ​​are assigned according to a Gaussian function, with time slices closer to the center receiving higher correlation strengths. After assignment, all correlation strengths within the subset are normalized to a sum of 1. Time slices outside the subset have a correlation strength of zero. This generates an initial version of the material transit time assignment matrix, where non-zero elements are concentrated within the possible source intervals corresponding to each exit sample.

[0043] Specifically, let the testing time for the exported samples be... Upstream time slice The center time is The cumulative elapsed time range determined by the aforementioned steps is: Then the estimated time center is taken as The scale parameter is taken as ,in Fixed sampling interval; time slice The initial correlation strength is ; all within the subset After summing and normalizing, the initial correlation strength of the matrix row corresponding to the export detection sample is obtained.

[0044] In box 204, temporal constraints, batch boundary constraints, no-material-passage constraints, and normalization constraints are applied to the matrix. To ensure that the matrix always conforms to physical facts and production rules during subsequent training, the system establishes a mask matrix with the same dimensions as the material passage time allocation matrix. In the mask matrix, three types of upstream time slices are marked as restricted time slices and their corresponding mask values ​​are set to zero: the first type is upstream time slices whose time is later than the detection time of the exit detection sample, because the exit sample cannot originate from upstream data generated after its detection; the second type is upstream time slices whose batch number is different from that of the exit detection sample, as there is a clear material boundary between different batches and no cross-batch associations should be established; the third type is upstream time slices in a state where the equipment is stopped and no material passes through; the mask values ​​at the remaining positions are set to 1.

[0045] In each training iteration, the system multiplies the material passage time allocation matrix and the mask matrix element by element, and then performs normalization on each row of the matrix. For matrix rows where the sum of the elements-wise multiplication is greater than 0, row normalization is performed; for matrix rows where the sum of the elements-wise multiplication is equal to 0, the system re-determines candidate upstream time slots based on the conditions of being from the same batch, not experiencing downtime, and having a time no later than the detection time of the exit detection sample; if no candidate upstream time slots are found after re-determination, the exit detection sample is marked as temporarily excluded from this round of training until subsequent data collection can form a valid matrix row.

[0046] After the above two steps, the association strength of the restricted positions is reduced to zero, and the association strength of the remaining positions is redistributed proportionally so that the row sum equals 1. This operation is carried out throughout the entire training process, so that the temporal constraints, batch boundary constraints, no-material-passage constraints and normalization constraints remain effective at every step of parameter update.

[0047] Among them, the material passing time allocation matrix is ​​a trainable parameter, which is updated by gradient based on the quality prediction error in subsequent training; the initial version gives a reasonable starting point estimate based on the equipment operation data, and the matrix value is gradually adjusted within the physical constraints during the training process to make the correspondence between the exit sample and the upstream time slice closer to reality; the specific training and update process of the matrix will be explained in the subsequent joint update section, and will not be repeated here.

[0048] In step S3, based on the correlation strength of the material passing through the time allocation matrix, multimodal data records of the corresponding upstream time slices are aggregated for the exit detection samples to construct multimodal training samples; the multimodal training samples are input into the quality assessment model to output instantaneous quality scores, defect category probabilities, and confidence levels.

[0049] In some embodiments, after the material passes through the time allocation matrix, the system aggregates the upstream multimodal data that it actually passed through for each exit detection sample. The system reads the matrix row corresponding to a certain exit detection sample, extracts the upstream time slices with non-zero correlation strength, arranges the multimodal data records of these time slices in chronological order to form a time slice sequence, and binds the correlation strength in the allocation matrix with the corresponding time slice in the time slice sequence. The correlation strength value is input into the subsequent model along with the time slice sequence to perform differential weighting of the information of each time slice within the model.

[0050] A complete multimodal training sample consists of three parts. Specifically, the first part is the aforementioned time slice sequence and its associated correlation strength, which carries the processing information of the material corresponding to the export sample in each upstream process stage. The second part is the export inspection data, namely the image features, weighing value, moisture value, sealing inspection results, and appearance inspection results collected at the export inspection point. The third part is the traceability information, including batch number, formula number, equipment number, shift number, and packaging identification. In the training sample constructed in this way, the upstream process data is selected based on the correlation strength given by the material passing through the time allocation matrix, rather than a simple splicing of a fixed time point. This can reduce the mismatch between the export sample and the upstream data in scenarios such as production line speed change, shutdown, or batch switching.

[0051] In some embodiments, the quality assessment model receives the multimodal training samples described above and outputs an instantaneous quality score, defect category probability, and confidence level. Specifically, please refer to [link to relevant documentation]. Figure 3 , Figure 3 A schematic diagram of the quality assessment model structure provided in this disclosure embodiment is shown, such as... Figure 3 As shown, the model consists of five sequentially connected modules: input projection module, position encoding module, weighted attention encoding module, exit feature fusion module, and output head module.

[0052] The input projection module is responsible for unifying the heterogeneous features of each time slice in the time slice sequence into the same dimension. Specifically, this module receives the numerical device parameter vector, image feature vector, online detection result vector converted from the online detection results of the same time slice, and discrete information embedding vector of a single time slice. After concatenating the above vectors along the feature dimension into a long vector, it maps it to a fixed-dimensional sequence element representation through a fully connected network. The output dimension of the fully connected network is the hidden layer dimension of the model, which can be configured as an integer value between 128 and 512. The projected sequence element representation is then randomly input into the position encoding module.

[0053] The positional encoding module superimposes learnable positional embeddings onto each sequence element after projection. The embedding dimension is consistent with the hidden layer dimension, which is used to preserve the chronological order information of time slices in the sequence. After superposition, the sequence enters the weighted attention encoding module.

[0054] The weighted attention encoding module consists of multiple layers of multi-head self-attention layers and feedforward network layers stacked alternately. The number of layers can be configured according to the actual situation. Specifically, within each layer, the multi-head self-attention sub-layer is followed by residual connections and layer normalization, and the feedforward network sub-layer is also followed by residual connections and layer normalization. The basic principle of multi-head self-attention is to project the input sequence into three sets of vectors: query, key, and value. The attention score between each position is calculated by the dot product of the query and the key. Then, the value vector is weighted and summed using this score, so that each position in the sequence can aggregate information from other positions.

[0055] This embodiment, based on the above, after calculating the original attention score between the query vector and the key vector, incorporates the correlation strength corresponding to each time slice into the attention score normalization process. This allows time slices with higher correlation strength to receive a larger weight in information aggregation and have a more significant impact on the final output; time slices with zero or near-zero correlation strength have their attention weights reduced accordingly in the normalized attention weights. After multi-layer encoding, the sequence representation is input to the exit feature fusion module. Through this processing, the model incorporates material transit relationships while considering the original attention score, giving upstream time slices with stronger correlation to the exit detection sample a greater influence in information aggregation. Specifically, the normalized attention weights are:

[0056] in, Indicates the first in the weighted attention encoding module Each query position corresponds to a sequence element in the time slice sequence; Indicates the first A single upstream time slice location that has garnered attention; The time slice index used for normalized summation; This indicates the length of the upstream time slice sequence corresponding to the current export detection sample, i.e., the number of time slices involved in the attention calculation; Indicates the first The query position and the first The raw attention score, calculated between the key vectors of each time slice, is used to characterize the degree of correlation between them in the feature space. This indicates that the material passes through the time allocation matrix, and the outlet detection sample is related to the first... The correlation strength between upstream time slices is determined by factors such as equipment operating status, start / stop logs, speed change records, and batch boundaries, and is subject to time constraints, batch boundary constraints, no material passing constraints, and normalization constraints. This represents the normalized attention weights, i.e., the th... The query position is assigned to the first query position when aggregating information. The weights of each time slice; in the formula It is an exponential function. This is used to normalize the weights of all time slices corresponding to the same query position, so that the attention weights of each time slice form a comparable weight distribution.

[0057] The export feature fusion module performs a weighted pooling operation on the encoded sequence representation. The pooling weight is the correlation strength in the material passing through the time allocation matrix. After weighted summation, a summary vector of upstream process data is obtained. This summary vector is concatenated with the feature vector of export detection data and the traceability information embedding vector along the feature dimension. Then, it is mapped to a unified dimension fusion vector through a fully connected network and input to the output head module.

[0058] The output head module receives the fused vector and, after passing through a fully connected layer, splits it into three parallel output branches: the first branch outputs the instantaneous quality score, which is a single scalar value; the second branch outputs the defect category probability, which is a probability vector with a dimension equal to the number of predefined defect categories; the third branch outputs the confidence score, which is a single scalar value; in addition, the output head module also has an independent fully connected branch, which, also taking the fused vector as input, outputs an anomaly score scalar. This branch is specifically used for calculating the contribution probability in subsequent sample packets.

[0059] In step S4, a sample package is constructed based on the delayed quality inspection results, the contribution probability of each outbound inspection sample in the sample package is calculated, and a composite label and label credibility are generated based on the contribution probability; the quality assessment model and the material passage time allocation matrix are updated with the composite label and label credibility.

[0060] In some embodiments, the results of microbiological testing, physicochemical testing, and sample re-testing in food enterprises are typically available hours or even days after the product completes online testing, and can only be traced back to a specific batch, a certain production period, or the granularity of certain packaging labels. If a delayed non-conforming result is directly backfilled to all samples under that batch, a large number of actually qualified samples within the batch will be incorrectly labeled, thus reducing the accuracy of the training labels. This embodiment constructs a sample package and employs a multi-instance learning (MIL) mechanism to distribute the impact of delayed quality inspection results to potentially relevant samples.

[0061] Specifically, please refer to Figure 4 , Figure 4 A flowchart illustrating the calculation of the contribution probability of a detection sample provided in an embodiment of this disclosure is shown, as follows: Figure 4 As shown in box 401, fields are read from the delayed quality inspection results and the candidate sample set is filtered.

[0062] The system reads fields such as the test item name, test result value, judgment conclusion, corresponding batch number, corresponding production time range, and packaging label from a delayed quality inspection result. First, based on the corresponding batch number and corresponding production time range, it filters out export test samples belonging to that batch and whose test time falls within that time range from the export test sample database to obtain a candidate sample set. When the packaging label exists, the system further retains export test samples that match the packaging label in the candidate sample set, and removes the remaining samples from the candidate set to make the candidate range more accurate. If the packaging label does not exist, the candidate sample set filtered by batch number and time range remains unchanged.

[0063] In box 402, upstream data is traced based on the material transit time allocation matrix and a sample package is formed. The matrix row of each exit inspection sample in the candidate sample set is read from the material transit time allocation matrix, its associated upstream time slice is extracted, and the multimodal data records of these time slices are retrieved. This upstream data provides equipment status and online inspection information in key process segments for subsequent contribution probability calculation. The system combines all retained exit inspection samples, the corresponding upstream multimodal data records, and the inspection item name, inspection result value, and judgment conclusion of the delayed quality inspection result into a sample package. One sample package corresponds to one delayed quality inspection result, and the package contains several exit inspection samples and their upstream process information.

[0064] In box 403, the quality assessment model is input for each exit detection sample within the sample bag, and anomaly scores are calculated. The system constructs its multimodal training samples for each exit detection sample within the bag in the aforementioned manner and inputs them into the quality assessment model. After passing through four modules—input projection, position encoding, weighted attention encoding, and exit feature fusion—the fusion vector and predicted output of the sample are obtained. The fusion vector is passed to an independent fully connected branch in the output head module, which outputs an anomaly score scalar, representing the degree to which the sample exhibits delayed quality inspection anomaly features.

[0065] In box 404, the contribution probability is calculated based on the anomaly score to obtain the bag-level prediction result. Specifically, let's assume there are a total of [number missing] anomaly scores within the sample bag. The first export test sample, the first The anomaly score for each sample is The system calculates the exponential value for each outlier score, divides the exponential value of each outlier score by the sum of all exponential values, and obtains the contribution probability of each sample.

[0066] in It is an exponential function. The value is between 0 and 1, and the sum of all contribution probabilities equals 1; the above formula is essentially a normalized exponential function (Softmax), which converts the abnormal scores of each sample into a probability distribution; A higher value indicates that the model judges the sample to be more likely to be related to delayed quality inspection anomalies; the contribution probability is a derivative calculated from the anomaly score using a normalized exponential function; after the model parameters are updated, the anomaly score changes accordingly, and the contribution probability is recalculated and updated accordingly; after obtaining the contribution probability, the system weights and aggregates the prediction outputs of each exit detection sample within the packet according to the contribution probability to obtain the packet-level prediction result: in For the first Each sample outputs the probability of defect category or anomaly probability corresponding to the detection item name and judgment conclusion; the system predicts based on the package level results. The delayed label loss is calculated based on the difference between the category labels corresponding to the judgment conclusion of the sample package, using cross-entropy as the loss function. The training objective requires that the overall prediction of the package be consistent with the delayed quality inspection conclusion, but does not require that every sample within the package be labeled as the same category. During training, samples exhibiting anomalous characteristics in key process segments naturally receive a higher contribution probability, while samples with weaker correlation to anomalous characteristics are not significantly affected. This mitigates label contamination caused by batch-based, extensive backfilling while utilizing delayed quality inspection information.

[0067] In some embodiments, for export inspection samples with online inspection labels, the system generates or updates composite labels for the export inspection samples based on whether the export inspection samples are covered by delayed quality inspection results, and writes four fields—instant quality status, delayed quality inspection status, label source, and label credibility—into the composite label.

[0068] The label confidence level is a real number between 0 and 1, used to represent the confidence level of the current training label for the sample. For example, its value can be determined according to at least the following three scenarios. For export inspection samples that do not fall into the following three scenarios, the system keeps its existing composite label and label confidence level unchanged, or updates the label confidence level according to the consistency results of the covered online inspection labels and delayed quality inspection labels. Specifically, the first scenario is: the online inspection is qualified and no delayed quality inspection results cover the sample. At this time, the label confidence level is set to a preset benchmark value, which is configured during system initialization and has a value range between 0.8 and 1.0. It can be determined according to the historical false alarm rate statistics of the online inspection equipment on the production line.

[0069] The second scenario: The online test is qualified, but the delayed quality inspection is abnormal; the system will determine the contribution probability of this sample in the sample package. Compare with a preset threshold. Preset threshold Configured during system initialization, with a value range of 0 to 1; when using an adaptive threshold... Take the upper quartile of the contribution probability distribution within the current sample bag; set the preset benchmark value as... The minimum credibility is ,in .like Then the label confidence level of the sample is set to ;like Then maintain .

[0070] like If the value is higher than a preset threshold, it indicates that the model believes the sample is more likely to be related to latency anomalies, and the system accordingly lowers the confidence level of its label; if... The value should not exceed the preset threshold and should remain unchanged. The preset threshold is configured during system initialization and can optionally be the upper quartile of the contribution probability distribution. In actual applications, it can be adjusted according to the characteristics of the production line.

[0071] The third scenario: Online detection shows an anomaly, but delayed review shows it passes; the system reduces the weight of the defect label for that sample, specifically by setting the label confidence level used for defect category loss calculation to [value missing]. ,in The preset reduction coefficient has a range of values. At the same time, all data of the export inspection sample are retained; the purpose of this retention is to provide the model with information on the difference between the online detection results and the delayed review results of the sample, so that the model can learn to distinguish between real defects and online detection false alarms during training.

[0072] During model training, the system multiplies the loss value of each exit detection sample by its label confidence level and includes it in the total loss. Samples with lower confidence levels contribute less to the total loss, while samples with higher confidence levels contribute more to the total loss.

[0073] In some embodiments, during training, the system calculates two types of losses: an immediate label loss based on online detection labels, including regression loss of immediate quality scores and classification loss of defect category probabilities; and a delayed label loss based on delayed quality inspection labels through package-level prediction of the aforementioned sample packages. The system then weights and sums the two types of losses according to preset weight coefficients to obtain the total loss, where the weight coefficient for the immediate label loss is... The weighting coefficients for delayed label loss are: Both are preset positive real numbers, with values ​​ranging from 0 to 1, and .

[0074] When there are few delayed quality inspection results in the early stages of training, it is advisable to... Set to a large value, and increase it according to a predetermined strategy as delayed results accumulate. In one implementation, let the proportion of export inspection samples covered by delayed quality inspection results in the current training round be . ,and Then the weighting coefficients for the delayed label loss are taken as follows: The weighting coefficient for the instant label loss is taken as... ,in and These are all parameters configured during system initialization and satisfy... .

[0075] In one embodiment, the system jointly updates the model parameters of the quality assessment model and the material passage time allocation matrix through backpropagation based on the total loss; the contribution probability is recalculated by the updated anomaly score, and its value is updated as the model parameters are updated; the material passage time allocation matrix adopts an independent learning rate, which is usually set to one-tenth to one-hundredth of the model's main learning rate to avoid the matrix value shifting too quickly in the early stage of training because the model has not yet fully converged; after the material passage time allocation matrix is ​​updated, the system re-executes the element-wise multiplication of the mask matrix and the normalization of the matrix rows to ensure that the four physical constraints are continuously satisfied after each iteration.

[0076] In one embodiment, after each update, the system performs a version verification, which includes two items: the first item is to check whether the updated material time allocation matrix has any allocation that crosses the batch boundary, or whether there is any situation where a time slice in the equipment downtime state is allocated to the exit inspection sample; if any of the above situations occur, the item is deemed to fail; the second item is to evaluate the updated quality assessment model on the historical validation set and recent delayed re-inspection results, and if the prediction performance is lower than the previous version, it is also deemed to fail.

[0077] When version verification fails, the system refuses to write the updated quality assessment model and the updated material time allocation matrix, and retains the parameters of the previous version for continued use; when version verification passes, the updated quality assessment model and the updated material time allocation matrix are written to the model version library for online evaluation.

[0078] In one embodiment, after the model goes online, the system receives real-time collected multimodal data and generates real-time time slice multimodal data records according to the aforementioned fixed sampling interval. For products arriving at the exit inspection point, the system determines the corresponding upstream time slice based on the current material passage time allocation matrix, forms real-time evaluation samples according to the aforementioned training sample construction method, inputs them into the current quality evaluation model, and outputs instantaneous quality score, defect category probability, and confidence level.

[0079] After the delayed quality inspection results are generated, the system associates the delayed quality inspection results with historical export inspection samples according to the aforementioned sample package construction process to form a sample package, calculates the contribution probability, and then updates the composite label and label credibility of the corresponding sample according to the aforementioned composite label generation process. The updated historical samples are included in the next round of model training data, so that the system can gradually accumulate delayed quality inspection information during continuous operation, correct the label quality of historical training samples, and feed the corrected information back to the model parameters and material passage time allocation matrix through retraining.

[0080] Therefore, this method uses a material transit time allocation matrix to associate the export inspection samples with upstream process data according to the actual transit time of the materials. This allows the construction of training samples to adapt to changes in operating conditions such as production line speed changes, shutdowns, resumption of operation, and batch switching, reducing data mismatch. By using sample packages and a multi-instance learning mechanism, delayed quality inspection results are reasonably allocated to potentially relevant samples, avoiding label contamination caused by batch-by-batch backfilling of labels. The material transit time allocation matrix provides a more accurate traceability range for the sample packages, while delayed quality inspection information provides more realistic quality feedback for the training of the allocation matrix. The two cooperate with each other in joint training to improve the consistency between the training samples and the real material processing process and real quality conclusions, thereby improving the stability of food processing quality assessment under different shifts, different cycle times, and different raw material batches.

[0081] Please see Figure 5 , Figure 5 This is a structural block diagram of a food processing quality analysis and evaluation system based on multimodal data, provided in an embodiment of this application. For example... Figure 5 As shown, the system includes: The data recording module 501 is configured to establish a process segment data table based on the process flow configuration information of the production line, and divide the running time of each process segment into time slices according to a fixed sampling interval, and collect equipment operating parameters, batch and formula information, online detection results and image features for each time slice to form multimodal data records; The matrix generation module 502 is configured to generate a material passage time allocation matrix and apply timing constraints, batch boundary constraints, no material passage constraints, and normalization constraints based on equipment operation data, equipment start-up and stop logs, speed change records, and batch boundaries. The training sample construction and quality assessment module 503 is configured to construct multimodal training samples by aggregating multimodal data records of the upstream time slices corresponding to the material passing through the time allocation matrix based on the correlation strength of the material passage time. The multimodal training samples are then input into the quality assessment model, which outputs an instantaneous quality score, defect category probability, and confidence level. The delayed quality inspection processing and updating module 504 is configured to construct a sample package based on the delayed quality inspection results, calculate the contribution probability of each outbound inspection sample in the sample package, and generate a composite label and label credibility based on the contribution probability; and update the quality assessment model and the material passage time allocation matrix with the composite label and label credibility.

[0082] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0083] Based on the same inventive concept, this application also provides an electronic device, the method corresponding to which can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. For example... Figure 6 As shown, Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods and / or technical solutions of the foregoing embodiments of the present application.

[0084] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processor, it performs the functions defined in the methods of this application.

[0085] Another embodiment of this application provides a storage medium storing computer program instructions thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The above descriptions are merely preferred embodiments of this application and explanations of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by specific combinations of the above technical features, but should also cover other technical solutions formed by arbitrary combinations of the above technical features or their equivalent features without departing from the inventive concept.

Claims

1. A method for food processing quality analysis and evaluation based on multimodal data, characterized in that, include: Based on the process flow configuration information of the production line, a process segment data table is established, and the running time of each process segment is divided into time slices according to a fixed sampling interval. Equipment operating parameters, batch and formula information, online detection results and image features are collected for each time slice to form a multimodal data record. Based on equipment operation data, equipment start-up and shutdown logs, speed change records, and batch boundaries, a material passage time allocation matrix is ​​generated and time-series constraints, batch boundary constraints, no-material-passage constraints, and normalization constraints are applied. Based on the correlation strength of the material passing through the time allocation matrix, multimodal data records of the corresponding upstream time slices are aggregated for the exit inspection samples to construct multimodal training samples; the multimodal training samples are input into the quality assessment model to output instantaneous quality scores, defect category probabilities, and confidence levels; A sample package is constructed based on the delayed quality inspection results. The contribution probability of each outbound inspection sample in the sample package is calculated, and a composite label and label credibility are generated based on the contribution probability. The quality assessment model and the material time allocation matrix are updated with the composite label and label credibility.

2. The method according to claim 1, characterized in that, The generated material transit time allocation matrix includes: using the exit inspection sample as the matrix rows and the upstream time slice as the matrix columns; calculating the baseline transit time of the conveyor belt conveying process segment based on the conveying distance and conveyor belt speed; statistically analyzing the dwell time range of equipment with material retention characteristics based on historical stable production data; determining the upstream time slice subset corresponding to each exit inspection sample by combining shutdown events, speed reduction events, resumption of operation events, and batch switching events; assigning initial correlation strength and normalizing it to the time slices within the upstream time slice subset, and assigning a value of zero to the time slices outside the upstream time slice subset.

3. The method according to claim 1, characterized in that, Applying time-series constraints, batch boundary constraints, no-material-passage constraints, and normalization constraints includes: establishing a mask matrix with the same dimension as the material passage time allocation matrix; marking upstream time slices whose time is later than the detection time of the exit detection sample, upstream time slices whose batch number is different from the exit detection sample, and upstream time slices in a state of equipment shutdown and no material passage as restricted time slices; in each training iteration, multiplying the material passage time allocation matrix element by the mask matrix and performing normalization processing on the matrix rows.

4. The method according to claim 3, characterized in that, Constructing multimodal training samples includes: reading the matrix rows in the material passage time allocation matrix corresponding to the exit detection samples, and extracting upstream time slices with non-zero correlation strength; arranging the multimodal data records of the upstream time slices in chronological order to form a time slice sequence, and binding the correlation strength with the corresponding time slice in the time slice sequence; and merging the time slice sequence, exit detection data, and traceability information into the multimodal training samples.

5. The method according to claim 4, characterized in that, The quality assessment model comprises an input projection module, a location encoding module, a weighted attention encoding module, an exit feature fusion module, and an output head module connected in sequence. The input projection module concatenates and maps the numerical device parameter vector, image feature vector, and discrete information embedding vector of each time slice into a sequence element representation. The weighted attention encoding module incorporates the correlation strength into the attention score normalization process. The exit feature fusion module performs weighted pooling on the encoded sequence representation according to the correlation strength and concatenates it with the exit detection data and the traceability information to generate a fusion vector. The output head module outputs the instantaneous quality score, the defect category probability, and the confidence level based on the fusion vector.

6. The method according to claim 5, characterized in that, Updating the quality assessment model and the material transit time allocation matrix includes: calculating the immediate label loss based on online detection labels and the delayed label loss based on delayed quality inspection labels; weighting and summing the immediate label loss and the delayed label loss according to preset weight coefficients to obtain the total loss; jointly updating the model parameters of the quality assessment model, the material transit time allocation matrix, and the contribution probability based on the total loss; and re-performing the element-wise multiplication of the mask matrix and the normalization of the matrix rows after updating the material transit time allocation matrix.

7. The method according to claim 6, characterized in that, Constructing a sample package based on delayed quality inspection results includes: reading the inspection item name, inspection result value, judgment conclusion, corresponding batch number, corresponding production time range, and packaging label from the delayed quality inspection results; filtering export inspection samples based on the corresponding batch number and the corresponding production time range to obtain a candidate sample set; retaining export inspection samples that match the packaging label when the packaging label exists; reading the matrix row of each export inspection sample in the candidate sample set in the material transit time allocation matrix and extracting the multimodal data record of its associated upstream time slice; and combining the retained export inspection samples, the corresponding multimodal data record, and the judgment conclusion to form the sample package.

8. The method according to claim 7, characterized in that, The calculation of contribution probability includes: inputting the quality assessment model into each outgoing detection sample in the sample packet to obtain the predicted output and fusion vector of that outgoing detection sample; inputting the fusion vector into independent fully connected branches to obtain the anomaly score of that outgoing detection sample; taking the exponential value of all anomaly scores in the sample packet, and dividing the exponential value of each anomaly score by the sum of the exponential values ​​of all anomaly scores to obtain the contribution probability of each outgoing detection sample; weighting and aggregating the predicted outputs of each outgoing detection sample according to the contribution probability to obtain the packet-level prediction result; and calculating the delay label loss based on the packet-level prediction result and the judgment conclusion.

9. The method according to claim 8, characterized in that, Generating composite labels and label credibility includes: writing the immediate quality status, delayed quality inspection status, label source, and label credibility into the composite label; when online detection is qualified and there is no delayed quality inspection result coverage, setting the label credibility to a preset benchmark value; when online detection is qualified but delayed quality inspection is abnormal, adjusting the label credibility based on the comparison result of the contribution probability and a preset threshold; when online detection is abnormal but delayed review is qualified, reducing the weight of the defect label and retaining the corresponding exit detection sample; during training, multiplying the loss value of each exit detection sample by its label credibility and including it in the total loss.

10. A food processing quality analysis and evaluation system based on multimodal data, characterized in that, include: The data recording module is configured to establish a process segment data table based on the process flow configuration information of the production line, and divide the running time of each process segment into time slices according to a fixed sampling interval, and collect equipment operating parameters, batch and formula information, online detection results and image features for each time slice to form multimodal data records; The matrix generation module is configured to generate a material passage time allocation matrix and apply timing constraints, batch boundary constraints, no material passage constraints, and normalization constraints based on equipment operation data, equipment start-stop logs, speed change records, and batch boundaries. The training sample construction and quality assessment module is configured to construct multimodal training samples by aggregating multimodal data records of the upstream time slices corresponding to the material passing through the time allocation matrix based on the correlation strength of the material passage time. The multimodal training samples are then input into the quality assessment model, which outputs an instantaneous quality score, defect category probability, and confidence level. The delayed quality inspection processing and updating module is configured to construct a sample package based on the delayed quality inspection results, calculate the contribution probability of each outbound inspection sample in the sample package, and generate a composite label and label credibility based on the contribution probability; and update the quality assessment model and the material time allocation matrix with the composite label and label credibility.

Images