Food preservation box detection method and system based on neural network

Abstract

The invention provides a food preservation box detection method and system based on a neural network, and belongs to the technical field of neural networks and the technical field of food preservation box detection.The method comprises the steps that S1, multi-modal data are collected; s2, preprocessing and fusing the multi-modal data; s3, innovative neural network model processing; and S4, outputting a prediction result. According to the method, the innovative neural network model is combined, the feature information of the plastic food preservation box can be obtained from multiple dimensions, and the detection result is more accurate and reliable.

Description

Technical Field

[0001] This invention relates to the fields of neural network technology and food preservation box detection technology, specifically to a detection method and system for food preservation boxes based on neural networks. Background Technology

[0002] In the food packaging and storage sector, quality inspection of plastic food storage containers is crucial. Traditional inspection methods largely rely on manual visual inspection. Manual inspection is not only inefficient and unable to meet the needs of large-scale production, but the results are also heavily influenced by the subjective factors of the inspectors, easily leading to missed or false positives. For example, when inspecting for minute cracks on the surface of the food storage container, manual inspection may miss tiny defects due to visual fatigue or viewing angle issues.

[0003] With technological advancements, some companies are using a single camera combined with traditional image processing algorithms for inspection. However, a single camera (such as a high-definition camera) can only acquire visual information about the appearance of the food storage container, failing to detect defects within the material and struggling to identify potential problems under varying lighting, temperature, and humidity conditions. For instance, a high-definition camera cannot detect impurities or structural damage hidden within the plastic; furthermore, in low-light conditions, the image quality of the high-definition camera deteriorates, affecting inspection accuracy.

[0004] While deep learning-based detection methods have been applied, most have not fully utilized the advantages of multimodal data, and their simple model structures limit their ability to extract and process complex features. In the inspection of plastic food storage containers, using only single-modal data (such as image data alone) is insufficient to comprehensively acquire the feature information of the containers, resulting in insufficient detection accuracy and failing to meet the food industry's stringent quality control requirements for food storage containers. Summary of the Invention

[0005] This invention provides a detection method and system for food storage containers based on neural networks. By combining an innovative neural network model, it is possible to obtain feature information of plastic food storage containers from multiple dimensions, resulting in more accurate and reliable detection results.

[0006] One aspect of this specification discloses a detection method for food preservation containers based on neural networks, including:

[0007] S1. Multimodal data acquisition:

[0008] High-definition cameras, near-infrared cameras, and thermal imaging cameras are used to acquire images of food preservation boxes under different environmental conditions, obtaining multimodal data;

[0009] S2. Multimodal data preprocessing and fusion:

[0010] Multimodal data is image scaled and normalized, and features are extracted from the processed multimodal data. After feature extraction, a fusion method based on attention mechanism is used to fuse the features to obtain the multimodal fused feature map.

[0011] S3. Innovative Neural Network Model Processing:

[0012] The feature map after multimodal fusion is convolved by three convolutional layers with different kernel sizes and then stitched together to obtain a multi-scale fused feature map.

[0013] Dilated convolution is performed on the feature map fused from multiple scales to obtain a dilated convolution feature map;

[0014] Attention is calculated in both the spatial and channel dimensions of the dilated convolutional feature map to obtain a joint feature map;

[0015] The joint feature map is processed by a gated loop unit to obtain the feature representation;

[0016] S4. Prediction Results Output:

[0017] Based on labeled data, the feature representation is mapped and output through a fully connected layer to obtain a feature vector.

[0018] The Softmax function is used to convert the feature vector into a probability distribution vector.

[0019] Based on the probability distribution vector, the category with the highest probability is selected as the prediction result.

[0020] In this specification, the normalization process includes:

[0021] For high-definition images, use Normalization is performed, where It is the original high-definition image in The c-th channel pixel value at position c, These are the normalized pixel values; for near-infrared images and thermal imaging images, the normalization formula is: , These are the normalized pixel values. Before normalization The pixel value at the location, where [a, b] is the data range.

[0022] In this specification, during feature extraction:

[0023] For high-resolution images, a ResNet structure and attention mechanism module are used to extract appearance detail features, edge and contour features, and structural features;

[0024] For near-infrared images, convolution kernels sensitive to low-contrast images are used to extract internal material features and material composition features;

[0025] For thermal imaging images, convolutional kernels capable of capturing temperature change features are used to extract temperature distribution and heat conduction features.

[0026] In this specification, the formula for feature fusion using an attention-based fusion method is as follows: ;

[0027] in, This is the feature map after multimodal fusion. Features of high-definition images Near-infrared image features, Features of thermal imaging images, , , These are the weights calculated using a fully connected network.

[0028] In this specification, three convolutional layers with different kernel sizes are constructed in parallel, with kernel sizes of 3×3, 5×5, and 7×7 respectively. The output of the i-th convolutional layer... for: ;

[0029] in, This is the feature map after multimodal fusion. For convolution kernel, For bias, This represents the convolution operation. The ReLU function is used as the activation function, i.e. ;

[0030] If the outputs of the three convolutional layers are respectively , , The feature map of multi-scale fusion , This indicates splicing by channel dimension.

[0031] In this manual, the formula for dilated convolution is: ;

[0032] in, This is a dilated convolution feature map. This indicates a dilated convolution operation. For feature maps fused across multiple scales, For activation function, For hollow convolution kernels, The void ratio, For bias.

[0033] In this specification, for spatial dimension attention calculation, the dilated convolution feature map is first processed. Global average pooling and global max pooling are performed along the channel dimension to obtain two one-dimensional vectors. and Then, the two vectors are concatenated and passed through a convolutional layer with a kernel size of 7×7. , bias is Then, the spatial attention weight matrix S is obtained by using the Sigmoid activation function: ;

[0034] in, For the Sigmoid function, This represents a 7×7 convolution operation;

[0035] For channel-dimensional attention calculation, Global average pooling and global max pooling are performed in the spatial dimension to obtain two channel vectors. and After concatenating these two vectors, the mixture passes through two fully connected layers. The weight matrix of the first fully connected layer is... , bias is The weight matrix of the second fully connected layer is: , bias is Then, the channel attention weight vector C is obtained by using the Sigmoid activation function: ;

[0036] in, and These represent the first and second fully connected layers, respectively, with ReLU as the activation function.

[0037] Finally, the spatial attention weight matrix S and the channel attention weight vector C are respectively compared with... We perform weighted multiplication to obtain the joint feature map after spatial and channel joint attention processing. : .

[0038] In this specification, the gated loop unit adopts a bidirectional GRU structure, which first expands the joint feature map into a sequence by rows or columns. The feature sequence X is processed from both the forward and backward directions to obtain the forward hidden state sequence. and backward hidden state sequence ;

[0039] Concatenate the forward and backward hidden state sequences to obtain ;

[0040] Then, a self-attention mechanism is applied to H. Let the query matrix Q, key matrix K, and value matrix V of the self-attention mechanism all originate from H and are obtained through different linear transformations. The linear transformation matrices are as follows: , , The biases are respectively , , The feature representation output by the self-attention mechanism for: ;

[0041] in, K is the dimension after linear transformation. The softmax function is used to convert similarity scores into a probability distribution.

[0042] In this specification, the weights W for convolution and dilated convolution operations performed on three convolutional layers with different kernel sizes are initialized using a data distribution-based method:

[0043] First, the acquired images are labeled with data such as category, material composition, location, and state to obtain data labels. The feature map after multimodal fusion and the corresponding data labels are combined to form sample data.

[0044] Statistical analysis was performed on the images in the sample data to calculate the mean pixel value of different channels at different locations. and variance Statistical information was obtained;

[0045] Then, based on the statistical information, the weights are initialized using a truncated normal distribution, and the magnitude of the weight W is set to... , The kernel size is [size]. Input the number of channels. To output the number of channels, for each weight element , Here, m is the index of the position within the convolution kernel, m is the input channel index, and n is the output channel index, derived from a truncated normal distribution. Mid-sampling initialization, where and These are the mean and variance of the m-th input channel, with the cutoff range set to... .

[0046] Another aspect of this specification discloses a detection system for food storage containers based on neural networks, used to implement the detection method for food storage containers based on neural networks described in any one of the above-mentioned methods. The detection system for food storage containers based on neural networks includes:

[0047] The multimodal data acquisition module is used for:

[0048] High-definition cameras, near-infrared cameras, and thermal imaging cameras are used to acquire images of food preservation boxes under different environmental conditions, obtaining multimodal data;

[0049] The multimodal data preprocessing and fusion module is used for:

[0050] Multimodal data is image scaled and normalized, and features are extracted from the processed multimodal data. After feature extraction, a fusion method based on attention mechanism is used to fuse the features to obtain the multimodal fused feature map.

[0051] An innovative neural network model processing module is used for:

[0052] The feature map after multimodal fusion is convolved by three convolutional layers with different kernel sizes and then stitched together to obtain a multi-scale fused feature map.

[0053] Dilated convolution is performed on the feature map fused from multiple scales to obtain a dilated convolution feature map;

[0054] Attention is calculated in both the spatial and channel dimensions of the dilated convolutional feature map to obtain a joint feature map;

[0055] The joint feature map is processed by a gated loop unit to obtain the feature representation;

[0056] The prediction result output module is used for:

[0057] Based on labeled data, the feature representation is mapped and output through a fully connected layer to obtain a feature vector.

[0058] The Softmax function is used to convert the feature vector into a probability distribution vector.

[0059] Based on the probability distribution vector, the category with the highest probability is selected as the prediction result.

[0060] The embodiments described in this specification can achieve at least the following beneficial effects:

[0061] Multimodal data enhances detection accuracy: By acquiring multimodal data through conventional high-definition cameras, near-infrared cameras, and thermal imaging cameras, and combining this with an innovative neural network model, feature information of plastic food storage containers can be obtained from multiple dimensions. High-definition cameras capture exterior details, near-infrared cameras detect internal material defects, and thermal imaging cameras sense surface temperature changes, effectively improving the accuracy of detecting container type, damage status, degree of staining, and material composition, and reducing missed and false detection rates.

[0062] The innovative model enhances feature extraction capabilities: The constructed innovative neural network model incorporates structures such as a multi-scale feature fusion module, dilated convolutional layers, an enhanced attention mechanism, and recurrent memory units. The multi-scale feature fusion module can simultaneously acquire image features at different scales, dilated convolutional layers expand the receptive field, the attention mechanism allows the network to focus on important information, and the recurrent memory units capture long-term dependencies in feature sequences. These structures work synergistically to significantly enhance the network's ability to extract and process complex features from plastic food storage containers, improving the model's performance and generalization ability. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of a detection method for food preservation boxes based on neural networks, as described in some embodiments of the present invention.

[0065] Figure 2 This is a schematic diagram of the processing flow of the innovative neural network model involved in some embodiments of the present invention.

[0066] Figure 3 This is a schematic diagram of a detection system for a food preservation box based on a neural network, as described in some embodiments of the present invention. Detailed Implementation

[0067] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0068] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0069] like Figure 1 As shown, this embodiment provides a detection method for food preservation boxes based on neural networks, including:

[0070] S1. Multimodal data acquisition:

[0071] High-definition cameras, near-infrared cameras, and thermal imaging cameras are used to acquire images of food preservation boxes under different environmental conditions, obtaining multimodal data;

[0072] S2. Multimodal data preprocessing and fusion:

[0073] Multimodal data is image scaled and normalized, and features are extracted from the processed multimodal data. After feature extraction, a fusion method based on attention mechanism is used to fuse the features to obtain the multimodal fused feature map.

[0074] S3. For example Figure 2 As shown, the innovative neural network model processes:

[0075] The feature map after multimodal fusion is convolved by three convolutional layers with different kernel sizes and then stitched together to obtain a multi-scale fused feature map.

[0076] Dilated convolution is performed on the feature map fused from multiple scales to obtain a dilated convolution feature map;

[0077] Attention is calculated in both the spatial and channel dimensions of the dilated convolutional feature map to obtain a joint feature map;

[0078] The joint feature map is processed by a gated loop unit to obtain the feature representation;

[0079] S4. Prediction Results Output:

[0080] Based on labeled data, the feature representation is mapped and output through a fully connected layer to obtain a feature vector.

[0081] The Softmax function is used to convert the feature vector into a probability distribution vector.

[0082] Based on the probability distribution vector, the category with the highest probability is taken as the prediction result (i.e., the detection result).

[0083] In some embodiments, camera deployment and image acquisition involve deploying a set of acquisition devices including a conventional high-definition camera, a near-infrared camera, and a thermal imaging camera. The high-definition camera has a resolution of no less than 1920×1080 pixels and is used to acquire details of the appearance of the plastic food storage container, such as surface texture and labels. The near-infrared camera has a resolution of no less than 1280×720 pixels and can detect subtle differences inside the material of the storage container, such as the presence of internal cracks or impurities. The thermal imaging camera has an accuracy of 0.1°C and can sense the surface temperature distribution of the storage container to help determine whether there is potential damage or abnormal heating.

[0084] The positions and angles of the three cameras were adjusted to ensure that the plastic food storage container was in the same relative position in the images captured by all three cameras, and that the key parts of the container were covered in the image field of view. Images were acquired under different environmental conditions and angles, including varying humidity levels (e.g., 30%-80% relative humidity), temperatures (0℃-40℃), and lighting conditions (natural light, artificial light in different color temperature chambers). Under each combination of environmental conditions, each camera captured 1000 images to enrich the data diversity.

[0085] Images are stored in a unified HDF5 format, which can efficiently store various types of data and facilitate subsequent processing. Files are named according to the rule "plastic food container_status_environmental conditions_camera type_number.h5", and the files also record detailed metadata such as camera parameters and acquisition time.

[0086] In some embodiments, data labeling (data tagging) involves manually labeling the type of plastic food storage containers (e.g., different shapes and capacities), whether they are damaged (yes / no), and the degree of staining (none, slight, severe). Using professional material analysis instruments, such as Fourier transform infrared spectroscopy (FT-IR), the food storage container samples are analyzed to label the specific composition information of the plastic material, such as the proportion of polyethylene (PE), polypropylene (PP), etc.

[0087] Using improved annotation tools, such as the deep learning-assisted LabelMe tool, image annotation not only labels object categories and locations but also leverages models to automatically identify common features, improving annotation efficiency and accuracy. Annotation information is saved in JSON format, corresponding to the image filename, and the file records detailed annotation information, including location coordinates, category, and material composition. The annotation process must simultaneously consider the characteristics of the food storage container in images captured by three cameras to ensure consistency and completeness of the annotations.

[0088] In some embodiments, image scaling and normalization are performed as follows: For each type of image captured by a camera, the scaling ratio is dynamically adjusted based on the actual size and position of the plastic food storage container in the image to ensure that key features are not lost when the container is scaled to a fixed size (e.g., 224×224 pixels). For example, if the container occupies a small portion of the image, the scaling ratio is appropriately increased, while edge padding is used to maintain a uniform image size. The padding value is determined based on the image type (e.g., average padding is used for high-definition images, while specific background values ​​are used for near-infrared and thermal imaging images).

[0089] For normalization operations, it is no longer limited to simply normalizing pixel values ​​to the [0,1] range. For high-resolution images, it uses... Perform normalization (where It is the original high-definition image in The c-th channel pixel value at position c, (These are the normalized pixel values). For near-infrared images, if the data range is [a, b], then the normalization formula is: Similarly, thermal imaging images are normalized according to their data range.

[0090] In some embodiments, multimodal feature extraction and fusion are performed by constructing a multi-branch feature extraction network. For the high-resolution image branch, an improved ResNet structure is adopted, adding an attention mechanism module (such as a Squeeze-Excitation module) to the original residual block to enhance the network's focus on important features. For the near-infrared image branch and the thermal imaging image branch, convolutional neural network structures suitable for their data characteristics are designed respectively. For example, the near-infrared image branch uses convolutional kernels sensitive to low-contrast images, while the thermal imaging image branch uses convolutional kernels that can capture temperature change features.

[0091] After each branch network extracts features from its respective channel's image, the resulting features are fused. An attention-based fusion method is employed, assigning different weights to the feature maps of each branch. These weights are calculated using a small fully connected network based on the statistical information of the feature maps. For example, the feature map F after multimodal fusion is... ,in , , The weights are calculated using a fully connected network. Features of high-definition images Near-infrared image features, These are features of thermal imaging images.

[0092] In some embodiments, data partitioning and cross-validation preparation involves: The multimodal fusion feature map F and its corresponding data labels are combined to form sample data, which is then divided into training, validation, and test sets in a 70%:15%:15% ratio. During the partitioning process, stratified sampling is used to ensure that the proportion of various types of plastic food storage containers (different types, states, material compositions, etc.) in each set is similar to that of the overall data, thus guaranteeing data representativeness.

[0093] To further improve the model's stability and generalization ability, a 5-fold cross-validation strategy is adopted. During the training phase, the training set data is further divided into 5 subsets. Each training iteration uses 4 subsets as training data and the remaining subset as validation data. This process is repeated 5 times. Finally, the models obtained from the 5 training iterations are fused together (e.g., by averaging the model parameters) to obtain the final trained model.

[0094] In some embodiments, a multi-scale feature fusion module is constructed at the network front end. This module consists of three convolutional layers with different kernel sizes in parallel, namely 3×3, 5×5, and 7×7. The input is the multimodal fused feature map F, and the i-th convolutional layer (corresponding to a kernel size of 3×3, 5×5, and 7×7) is... The convolution kernel is , bias is The output of the convolutional layer for: ;

[0095] in, This represents the convolution operation. The ReLU function is used as the activation function, i.e. .

[0096] The outputs of convolutional layers with different kernel sizes are concatenated to obtain a multi-scale fused feature map. For example, if the outputs of the three convolutional layers are respectively , , ,but ,here This indicates concatenation along the channel dimension. This allows for the simultaneous acquisition of image features at different scales; small convolutional kernels focus on detailed features, while large convolutional kernels focus on global features.

[0097] In some embodiments, a dilated convolutional layer is added after the multi-scale feature fusion module. Dilated convolution can expand the receptive field of the convolutional kernel without increasing the number of parameters or computational cost. Let the input of the dilated convolutional layer be... The hollow convolution kernel is The void ratio is , bias is The output dilated convolution feature map of the dilated convolution layer. for: ;

[0098] in, This indicates a dilated convolution operation. Dilated convolution can better capture the contextual information of objects in an image. For tasks involving the detection of plastic food storage containers, it helps to more accurately identify the overall shape of the container and information about its surrounding environment.

[0099] In some embodiments, the spatial-channel attention module (SCAM) is constructed after the dilated convolutional layer. First, the input dilated convolutional feature map is processed... Attention calculations are performed separately for the spatial dimension and the channel dimension.

[0100] For spatial dimension attention calculation, first... Global average pooling and global max pooling are performed along the channel dimension to obtain two one-dimensional vectors. and Then, the two vectors are concatenated and passed through a convolutional layer with a kernel size of 7×7. , bias is Then, the spatial attention weight matrix S is obtained by using the Sigmoid activation function: ;

[0101] in, For the Sigmoid function, This represents a 7×7 convolution operation.

[0102] For channel-dimensional attention calculation, Global average pooling and global max pooling are performed in the spatial dimension to obtain two channel vectors. and After concatenating these two vectors, they are passed through two fully connected layers. The weight matrix of the first fully connected layer is... , bias is The weight matrix of the second fully connected layer is: , bias is Then, the channel attention weight vector C is obtained by using the Sigmoid activation function: ;

[0103] in, and These represent the first and second fully connected layers, respectively, with ReLU as the activation function.

[0104] Finally, the spatial attention weight matrix S and the channel attention weight vector C are respectively compared with the input feature map. Weighted multiplication yields the joint feature map after processing by the spatial and channel joint attention module. : ;

[0105] In this way, through the joint attention mechanism, the network can simultaneously focus on important information at different locations and in different channels of the image, thereby improving its ability to extract features from plastic food storage containers.

[0106] In some embodiments, a gated recurrent unit (GRU) is introduced after the spatial and channel joint attention module. The joint feature map processed by SCAM is then processed by the GRU. Expand into a sequence by row or column. This serves as the input to the GRU. Let the input weight matrix of the GRU cell be... The hidden layer weight matrix is The reset gate weight matrix is Update the gate weight matrix as follows The biases are respectively , , , .

[0107] For each time step t, first calculate the reset gate. and the update gate :

[0108] ;

[0109] ;

[0110] in, It is the hidden state of the previous time step. This is the Sigmoid function.

[0111] Then calculate the candidate hidden state. : ;

[0112] Where tanh is the hyperbolic tangent function. This represents element-wise multiplication.

[0113] Finally, calculate the hidden state at the current time step. : ;

[0114] GRU can capture long-term dependencies between feature sequences and has better processing capabilities for continuous feature changes that may exist in the detection of plastic food storage containers (such as the features of food storage container images taken from different angles).

[0115] In some embodiments, a bidirectional GRU structure is employed to further enhance the understanding of the feature sequence. The feature sequence X is processed from both the forward and backward directions to obtain the forward hidden state sequence. and backward hidden state sequence .

[0116] Concatenate the forward and backward hidden state sequences to obtain Then, a self-attention mechanism is applied to H. Suppose that the query matrix Q, key matrix K, and value matrix V of the self-attention mechanism are all derived from H through different linear transformations, where the linear transformation matrices are respectively... , , The biases are respectively , , The feature representation output by the self-attention mechanism for: ;

[0117] in, K is the dimension after linear transformation. The softmax function is used to convert similarity scores into a probability distribution.

[0118] Finally, the feature representation is obtained after processing by bidirectional GRU and self-attention fusion. It integrates information from before and after the sequence and highlights important features through a self-attention mechanism.

[0119] In some embodiments, the model parameters are finely tuned during initialization: for the weights W of the convolutional layers (including the multi-scale feature fusion module and the convolutional kernels in dilated convolutional layers), an initialization method based on data distribution is used. First, statistical analysis is performed on the images in the training dataset to calculate the mean pixel values ​​of different channels at different locations. and variance .

[0120] Then, based on this statistical information, the weights are initialized using a truncated normal distribution. Let the size of the weight matrix W be... ( The kernel size is [size]. Input the number of channels. (number of output channels), for each weight element ( (where m is the index of the position within the convolution kernel, m is the index of the input channel, and n is the index of the output channel), derived from the truncated normal distribution. Mid-sampling initialization, where and These are the mean and variance of the m-th input channel. The cutoff range is set to... This allows the initial weight values ​​to better reflect the actual distribution characteristics of the data, improving the initial performance of model training.

[0121] For the parameters of the GRU unit (input weight matrix) Hidden layer weight matrix Reset gate weight matrix Update the gate weight matrix (etc.), adopting an initialization strategy that combines pre-training and fine-tuning.

[0122] First, the GRU parameters in the current plastic food storage container detection model are initialized using the parameters of a GRU model pre-trained on a large-scale video action recognition dataset (such as the Kinetics dataset). This is because video action recognition tasks also involve processing time-series data, which is somewhat similar to using GRU to capture feature sequence dependencies in plastic food storage container detection.

[0123] Then, the initialized GRU parameters were fine-tuned on a dataset of plastic food storage container detection. During the fine-tuning process, a small learning rate (e.g., ...) was used. =0.0001), to avoid excessive changes to the pre-training parameters, and at the same time, to combine the characteristics of the training data to make the GRU parameters better adapt to the plastic food preservation box detection task, accelerate model convergence, and reduce training time.

[0124] In some embodiments, training parameters are set: a loss function is defined, and for multi-class problems (such as food container category, damage state, degree of staining, etc.), the cross-entropy loss function is selected. It is assumed that the model predicts the class probability distribution as follows: The actual category label is y (using one-hot encoding), and its calculation formula is: ;

[0125] Where n is the number of categories. This loss function measures the difference between the model's predictions and the true labels; the smaller the difference, the more accurate the model's predictions.

[0126] The optimizer chosen is the Adaptive Moment Estimation (Adam) optimizer. The Adam optimizer combines the advantages of the Adagrad and RMSProp algorithms, enabling it to adaptively adjust the learning rate. Its learning rate... The initial value is 0.001, and this value will be automatically adjusted according to the algorithm during training.

[0127] Set the number of training epochs to 100. In each training epoch, the training set data is divided into multiple batches for training, and the size of each batch is set to 32.

[0128] In some embodiments, the training process involves: in each round of training, sequentially retrieving each batch of data from the training set. For each batch, a feature map F (containing features fused from high-resolution, near-infrared, and thermal imaging images) and its corresponding label are generated. The input F is fed into the innovative neural network model built in S3. After forward propagation, the model outputs the prediction result. The prediction result is a probability distribution vector, where each element represents the probability that the corresponding category is correctly predicted.

[0129] According to the prediction results and real labels The loss value is calculated using the cross-entropy loss function. The Adam optimizer is based on the loss value. Update the model parameters, including the weights and biases of convolutional layers, the weight matrices of GRU units, and all other model parameters.

[0130] After each training round, the model is validated using validation set data. The multimodal fused image data from the validation set is input into the model to obtain prediction results on the validation set, and the loss value on the validation set is calculated accordingly. and accuracy These metrics, along with others, are used to evaluate the model's performance on unseen data by monitoring changes in these metrics on the validation set, thus preventing overfitting.

[0131] In some embodiments, the training termination condition is: setting an early stopping mechanism, which terminates training early when the loss value on the validation set no longer decreases for 10 consecutive rounds or the accuracy no longer improves.

[0132] At the same time, a maximum training epoch of 100 is set as a fallback condition. Even if the validation set metrics do not meet the early stopping condition, training will still stop when the training epoch reaches 100.

[0133] In some embodiments, model testing involves using a test set to perform a final evaluation of the trained model after training is complete. This includes calculating the loss value and metrics such as accuracy, precision, recall, and F1 score on the test set. For example, precision measures the proportion of samples that the model predicted as positive but were actually positive; recall measures the proportion of samples that were actually positive but were correctly predicted as positive; and the F1 score is the harmonic mean of precision and recall, taking both into account. These metrics provide a more comprehensive evaluation of the model's performance in real-world applications. Based on the evaluation results from the test set, a final performance assessment of the model is made. If all metrics meet the expected requirements, the model can be deployed in actual applications for detecting plastic food storage containers; if the metrics are unsatisfactory, the model architecture, training parameters, or data quality need to be re-examined, and the model needs to be improved and retrained.

[0134] In some embodiments, a square transparent plastic food storage box was tested, which measures 20 cm in length, 15 cm in width, and 10 cm in height. The main body of the box is colorless and transparent, and the lid has a light blue marking.

[0135] Camera settings and image acquisition:

[0136] The high-definition camera is set to a resolution of 1920×1080 pixels, the near-infrared camera to a resolution of 1280×720 pixels, and the thermal imaging camera to an accuracy of 0.1℃. In an indoor environment with 50% relative humidity and 25℃, using a combination of natural light and 4000K color temperature LED lighting, the three cameras simultaneously captured images of the food storage container from different angles. Ten sets of images were acquired from each angle, each set containing one high-definition, one near-infrared, and one thermal imaging image, for a total of 300 images. These images were stored in an HDF5 file in the format "Square Transparent Plastic Food Storage Container_Intact_50% Humidity 25℃_Camera Type_Number.h5", along with recording parameters such as camera focal length and exposure time.

[0137] Data annotation:

[0138] The material of the food storage container was analyzed using Fourier transform infrared spectroscopy, which determined that the main component was polypropylene (PP), accounting for 95%, with the remainder being additives. Using the LabelMe tool, which is based on deep learning, the container's location coordinates, category (square transparent plastic food storage container), condition (intact), degree of staining (no stains), and material composition information were manually labeled in the image. The labeling results were saved as a JSON file.

[0139] Image scaling and normalization:

[0140] An adaptive scaling algorithm was used to scale the main body of the food container in the high-definition image to 224x224 pixels. Since the food container occupies a small area in the image, the scaling ratio was 1.2, and the edges were filled with the mean value. The near-infrared and thermal imaging images were also scaled to 224x224 pixels and filled with a specific background value.

[0141] For example, the original value of a pixel (10,10,1) in an image is 128, and after normalization it is approximately 0.502; the data range of near-infrared images is [100,500], and if the value of a pixel is 200, it will be 0.25 after normalization; the same applies to thermal imaging images.

[0142] Multimodal feature extraction and fusion:

[0143] The high-resolution image branch uses an improved ResNet structure with 50 layers and adds a Squeeze-Excitation module to the residual block; the near-infrared image branch uses an 8-layer convolutional neural network with kernel sizes of 3x3, 5x5, etc.; and the thermal imaging image branch uses a 6-layer convolutional neural network to focus on capturing temperature change features.

[0144] After extracting features from the three branches, a 3-layer fully connected network is used to calculate the weights. The inputs are statistical information such as the mean and variance of the feature maps of each branch, and the outputs are three weight values. =0.4、 =0.3、 =0.3, fused feature map F=0.4 +0.3 +0.3 .

[0145] 3. Data partitioning and cross-validation preparation

[0146] The preprocessed 300 sets of data were divided into a training set (210 sets), a validation set (45 sets), and a test set (45 sets) according to a 70%:15%:15% ratio. Stratified sampling was used to ensure a balanced proportion of data in each category. Five-fold cross-validation was used, and the training set was further divided into 5 subsets. Four subsets were used for training and one subset was used for validation each time.

[0147] Innovative Neural Network Model Construction:

[0148] Multi-scale feature fusion module: Three parallel convolutional layers are configured with kernel sizes of 3x3, 5x5, and 7x7 respectively. The fused feature map F is used as input, and the first convolutional layer outputs... ,in It is a 3x3 convolution kernel. For bias, The ReLU activation function is used; similarly, we obtain... After splicing, a multi-scale fused feature map is obtained. .

[0149] Hollow convolution layer: Hollow convolution kernel Size is 3x3, void ratio d=2, input Output .

[0150] Spatial and channel attention computation: for Spatial and channel attention calculations are performed. In spatial attention calculation, global average pooling and max pooling are used to obtain... The spatial attention weight matrix S is obtained through a 7x7 convolutional layer and a sigmoid function; the channel attention weight vector C is obtained through global pooling, a fully connected layer, and a sigmoid function. .

[0151] GRU unit:

[0152] Will Expand the sequence into a row-wise sequence X, and input it into a GRU cell. The GRU cell parameters include the weight matrix. The size is 128x64 (assuming the input feature dimension is 128 and the hidden layer dimension is 64), calculate the reset gate. By employing a bidirectional GRU and a self-attention mechanism, the final feature representation is obtained. .

[0153] Model parameter initialization:

[0154] The convolutional layer weights are initialized by sampling from a truncated normal distribution N(127,64) based on the pixel values ​​of 127 and 64 obtained from the training data. The GRU units are first initialized using pre-trained parameters from the Kinetics dataset, and then fine-tuned on the food storage box detection dataset with a learning rate of 0.0001.

[0155] Model training:

[0156] Training parameters are set as follows: the loss function is the cross-entropy loss function, the number of classes n=8 (including different combinations of types, states, etc.); the optimizer is the Adam optimizer, the initial learning rate is 0.001; the number of training epochs E=100, and the batch size B=32.

[0157] Training process: Each training round takes batches of data, and F and the corresponding data labels are input into the model. The prediction results are obtained through forward propagation. If the probability of a food storage container being in good condition in a certain batch of prediction results is 0.85, calculate the loss value. The Adam optimizer updates the model parameters. At the end of each round, the model is evaluated using a validation set, and the validation loss and accuracy are calculated.

[0158] Training termination condition: Training stops when the validation set loss value no longer decreases for 10 consecutive rounds, or when the number of training rounds reaches 100.

[0159] Detection results output: Using test set data for detection, the input image is inferred by the model, and the predicted results are output. For example, for a test image, the model outputs a probability of 0.92 for the food storage container category as "square transparent plastic food storage container", a probability of 0.95 for the condition as "intact", a probability of 0.98 for the degree of stain as "none", and a probability of 0.9 for the material composition as polypropylene (PP).

[0160] This embodiment provides a neural network-based detection system for food preservation containers, used to implement the neural network-based detection method for food preservation containers described above, such as... Figure 3 As shown, the detection system for food preservation boxes based on neural networks includes:

[0161] The multimodal data acquisition module is used for:

[0162] High-definition cameras, near-infrared cameras, and thermal imaging cameras are used to acquire images of food preservation boxes under different environmental conditions, obtaining multimodal data;

[0163] The multimodal data preprocessing and fusion module is used for:

[0164] Multimodal data is image scaled and normalized, and features are extracted from the processed multimodal data. After feature extraction, a fusion method based on attention mechanism is used to fuse the features to obtain the multimodal fused feature map.

[0165] An innovative neural network model processing module is used for:

[0166] The feature map after multimodal fusion is convolved by three convolutional layers with different kernel sizes and then stitched together to obtain a multi-scale fused feature map.

[0167] Dilated convolution is performed on the feature map fused from multiple scales to obtain a dilated convolution feature map;

[0168] Attention is calculated in both the spatial and channel dimensions of the dilated convolutional feature map to obtain a joint feature map;

[0169] The joint feature map is processed by a gated loop unit to obtain the feature representation;

[0170] The prediction result output module is used for:

[0171] Based on labeled data, the feature representation is mapped and output through a fully connected layer to obtain a feature vector.

[0172] The Softmax function is used to convert the feature vector into a probability distribution vector.

[0173] Based on the probability distribution vector, the category with the highest probability is selected as the prediction result.

[0174] In summary, several specific embodiments of the present invention have been disclosed. Without contradiction, the various embodiments can be freely combined to form new embodiments. That is, embodiments that are alternative solutions can be freely substituted for each other, but cannot be combined with each other; embodiments that are not alternative solutions can be combined with each other. These new embodiments are also part of the substantive content of the present invention.

[0175] The above embodiments describe several specific implementations of the present invention. However, those skilled in the art should understand that various changes or modifications can be made to these implementations without departing from the principles and essence of the present invention, but all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A detection method for food preservation containers based on neural networks, characterized in that, include: S1. Multimodal data acquisition: High-definition cameras, near-infrared cameras, and thermal imaging cameras are used to acquire images of food preservation boxes under different environmental conditions, obtaining multimodal data; S2. Multimodal data preprocessing and fusion: Multimodal data is image scaled and normalized, and features are extracted from the processed multimodal data. After feature extraction, a fusion method based on attention mechanism is used to fuse the features to obtain the multimodal fused feature map. S3. Innovative Neural Network Model Processing: The feature map after multimodal fusion is convolved by three convolutional layers with different kernel sizes and then stitched together to obtain a multi-scale fused feature map. Dilated convolution is performed on the feature map fused from multiple scales to obtain a dilated convolution feature map; Attention is calculated in both the spatial and channel dimensions of the dilated convolutional feature map to obtain a joint feature map; The joint feature map is processed by a gated loop unit to obtain the feature representation; S4. Prediction Results Output: Based on labeled data, the feature representation is mapped and output through a fully connected layer to obtain a feature vector; The Softmax function is used to convert the feature vector into a probability distribution vector. Based on the probability distribution vector, the category with the highest probability is selected as the prediction result.

2. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, During normalization: For high-definition images, use Normalization is performed, where It is the original high-definition image in The c-th channel pixel value at position c, These are the normalized pixel values; For near-infrared images and thermal imaging images, the normalization formula is: , These are the normalized pixel values. Before normalization The pixel value at the location, where [a, b] is the data range.

3. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, Feature extraction in progress: For high-resolution images, a ResNet structure and attention mechanism module are used to extract appearance detail features, edge and contour features, and structural features; For near-infrared images, convolution kernels sensitive to low-contrast images are used to extract internal material features and material composition features; For thermal imaging images, convolutional kernels that can capture temperature change features are used to extract temperature distribution and heat conduction features.

4. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, The formula for feature fusion using an attention-based fusion method is as follows: ； in, This is the feature map after multimodal fusion. Features of high-definition images Near-infrared image features Features of thermal imaging images, , , These are the weights calculated using a fully connected network.

5. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, Three convolutional layers with different kernel sizes are constructed in parallel, with kernel sizes of 3×3, 5×5, and 7×7 respectively. The output of the i-th convolutional layer... for: ； in, This is the feature map after multimodal fusion. For convolution kernel, For bias, This represents the convolution operation. The activation function is the ReLU function, i.e. ; If the outputs of the three convolutional layers are respectively , , The feature map of multi-scale fusion , This indicates splicing by channel dimension.

6. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, The formula for dilated convolution is: ； in, This is a dilated convolution feature map. This indicates a dilated convolution operation. For feature maps fused across multiple scales, For activation function, For hollow convolution kernels, The void ratio, For bias.

7. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, For spatial dimension attention calculation, first process the dilated convolution feature map... Global average pooling and global max pooling are performed along the channel dimension to obtain two one-dimensional vectors. and Then, the two vectors are concatenated and passed through a convolutional layer with a kernel size of 7×7. , bias is Then, the spatial attention weight matrix S is obtained by using the Sigmoid activation function: ； in, For the Sigmoid function, This represents a 7×7 convolution operation; For channel-dimensional attention calculation, Global average pooling and global max pooling are performed in the spatial dimension to obtain two channel vectors. and After concatenating these two vectors, the mixture passes through two fully connected layers. The weight matrix of the first fully connected layer is... , bias is The weight matrix of the second fully connected layer is: , bias is Then, the channel attention weight vector C is obtained by using the Sigmoid activation function: ; in, and These represent the first and second fully connected layers, respectively, with ReLU as the activation function. Finally, the spatial attention weight matrix S and the channel attention weight vector C are respectively compared with... We perform weighted multiplication to obtain the joint feature map after spatial and channel joint attention processing. : .

8. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, The gated recurrent unit adopts a bidirectional GRU structure, which first expands the joint feature map into a sequence by rows or columns. The feature sequence X is processed from both the forward and backward directions to obtain the forward hidden state sequence. and backward hidden state sequence ; Concatenate the forward and backward hidden state sequences to obtain ; Then, a self-attention mechanism is applied to H. Let the query matrix Q, key matrix K, and value matrix V of the self-attention mechanism all originate from H and are obtained through different linear transformations. The linear transformation matrices are as follows: , , The biases are respectively , , The feature representation output by the self-attention mechanism for: ; in, K is the dimension after linear transformation. The softmax function is used to convert similarity scores into a probability distribution.

9. The detection method for food preservation boxes based on neural networks according to claim 1, characterized in that, The weights W for convolution and dilated convolution operations performed on three convolutional layers with different kernel sizes are initialized using a data distribution-based method: First, the acquired images are labeled with data such as category, material composition, location, and state to obtain data labels. The feature map after multimodal fusion and the corresponding data labels are combined to form sample data. Statistical analysis was performed on the images in the sample data to calculate the mean pixel value of different channels at different locations. and variance Statistical information was obtained; Then, based on the statistical information, the weights are initialized using a truncated normal distribution, and the magnitude of the weight W is set to... , The kernel size is [size]. Input the number of channels. To output the number of channels, for each weight element , Here, m is the index of the position within the convolution kernel, m is the input channel index, and n is the output channel index, derived from a truncated normal distribution. Mid-sampling initialization, where and These are the mean and variance of the m-th input channel, with the cutoff range set to... .

10. A detection system for food preservation boxes based on neural networks, characterized in that, The method for detecting food preservation boxes based on neural networks according to any one of claims 1 to 9, the detection system for food preservation boxes based on neural networks includes: The multimodal data acquisition module is used for: High-definition cameras, near-infrared cameras, and thermal imaging cameras are used to acquire images of food preservation boxes under different environmental conditions, obtaining multimodal data; The multimodal data preprocessing and fusion module is used for: Multimodal data is image scaled and normalized, and features are extracted from the processed multimodal data. After feature extraction, a fusion method based on attention mechanism is used to fuse the features to obtain the multimodal fused feature map. An innovative neural network model processing module is used for: The feature map after multimodal fusion is convolved by three convolutional layers with different kernel sizes and then stitched together to obtain a multi-scale fused feature map. Dilated convolution is performed on the feature map fused from multiple scales to obtain a dilated convolution feature map; Attention is calculated in both the spatial and channel dimensions of the dilated convolutional feature map to obtain a joint feature map; The joint feature map is processed by a gated loop unit to obtain the feature representation; The prediction result output module is used for: Based on labeled data, the feature representation is mapped and output through a fully connected layer to obtain a feature vector. The Softmax function is used to convert the feature vector into a probability distribution vector. Based on the probability distribution vector, the category with the highest probability is selected as the prediction result.

Images