A fresh food identification method based on temporal reordering

Abstract

The present invention relates to a fresh produce recognition method based on temporal reordering. On the basis of the traditional fresh produce recognition method, a timestamp buffer is set and maintained during the training and inference process to record the correct category and timestamp of each sample in the past recognition sequence. In addition to image feature extraction, we additionally query the most recent appearance time and frequency of all categories from the timestamp buffer to obtain additional temporal features of all categories. We splice image features and temporal features, and re-predict the confidence of all categories through the temporal reordering network, so that the fresh produce recognition algorithm can learn the features of the changes between different codes of the same product, cope with the immediate changes in product codes in real applications, and effectively improve the recognition rate of the fresh produce recognition model.

Description

Technical Field

[0001] The present invention relates to a fresh food identification method based on time sequence reordering. Background Art

[0002] In recent years, with the continuous development of neural network technology, fresh produce recognition technology has been widely used in self-service weighing and identification of fresh produce in various stores. However, because the same category of supermarket products (such as apples) may have multiple product codes due to different sources or different quality, the codes of apples sold every day may be different. Current fresh produce recognition methods are mostly based solely on image classification, which has difficulty in overcoming the periodic changes in codes within the same category, resulting in low recognition accuracy for products with different codes within the same category. Summary of the Invention

[0003] In response to the above-mentioned problems in the prior art, the present invention provides an online fresh produce identification method based on time sequence reordering, which can quickly adapt to changes in commodity codes and effectively improve the accuracy of fresh produce identification.

[0004] One objective of this invention is to provide a fresh produce identification method based on temporal reordering. This method can accurately predict the confidence levels of thousands of fresh produce products, assisting supermarket cashiers in quickly identifying the types of fresh produce. This temporal reordering strategy improves the model's adaptability to changes in product codes, effectively boosting the recognition rate of fresh produce identification models in supermarkets.

[0005] To achieve these objectives and other advantages of the present invention, a method for identifying fresh produce based on time-series reordering is provided, comprising: establishing and maintaining a fixed-size category timestamp cache array for recording the category to which the identified product data stream belongs and the timestamp of each product identification during the identification process. Unlike traditional fresh produce identification, the training and testing processes of the fresh produce identification method based on time-series reordering are both based on the identification data stream. The fresh produce identification model we designed based on time-series reordering enables the fresh produce identification algorithm to better adapt to changes in the same product code in the data stream, while at the same time perceiving the sales frequency of different products, further improving the recognition rate.

[0006] The training loss function in the training process is the cross entropy loss function. We calculate the gradients of all parameters of the neural network based on the Adam optimizer and the total loss value, and optimize the network parameters accordingly to obtain

[0007] Based on the optimized neural network, during testing, the recognition logic of the data stream during training is still followed. For each fresh food input image tensor and timestamp, the recognition result is the confidence level of each predicted fresh food category. The categories with the highest confidence levels are the predicted fresh food categories, completing the fresh food recognition.

[0008] Preferably, the input image is any image I in the recognition sequence S. First, it passes through the feature extraction network N to obtain the image feature M. At the same time, the recognition timestamp cache C of each category j is read from the timestamp cache, and the most recent occurrence time R and the most recent occurrence frequency E of each category are obtained through statistics, and then the time series feature T is obtained. By merging the image feature vector M and the time series feature vector T, we input the merged feature into the time series reordering network O. The output of network O is the prediction result Y of picture I. In the inference mode, we return the prediction result Y to the cashier, and the cashier judges whether the prediction result contains the correct category based on the prediction result Y and gives feedback on a correct category F. In the training mode, we calculate the loss function L based on the prediction result Y and the feedback result F, and optimize the parameters of the network by calculating the gradient. Subsequently, we store the timestamp ti in the timestamp buffer C.

[0009] The feature extraction network N belongs to common image recognition networks, such as MobileNetV2, ResNet18, etc.

[0010] The size of the recognition timestamp buffer C is K. Here, K is the maximum cache number of categories. When the buffer is full, the category corresponding to the farthest timestamp will be automatically removed. The most recent occurrence time Rj is the most recent occurrence time of each category in C, and the most recent occurrence frequency Ej corresponds to the occurrence frequency of each category in C.

[0011] The time series reordering network O is composed of multiple fully connected layers. The output prediction result Y of O = P j , 0 <= j < J. Here, Pj is the recognition confidence of each category j.

[0012] The loss function L is default to the cross-entropy loss function, and the parameters of the feature extraction network N and the time series reordering network O are optimized by calculating the gradient based on the loss L.

[0013] The present invention has at least the following beneficial effects: Since the recognition method based on time series reordering is used, the fresh food algorithm can quickly adapt to the changes in commodity coding in the continuous time dimension, reduce the negative impact of different encodings of the same commodity on recognition, and effectively improve the accuracy of fresh food recognition.

[0014] Other advantages, objectives and features of the present invention will be partially reflected by the following description, and partially will be understood by those skilled in the art through the research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a time series reordering method according to an embodiment of the present invention.

[0016] Figure 2 It is a process diagram of an experiment according to an embodiment of the present invention. Detailed implementation manners

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0018] Figure 1 A fresh food recognition method based on timing reordering according to an embodiment of the present invention is shown, which includes:

[0019] Training and inference of the fresh food recognition model, including setting and maintaining a timestamp buffer. When processing each picture in the recognition sequence, the most recent appearance time and frequency of each category are considered simultaneously, so as to reorder the category confidence based on the timing information and the timing reordering network. The timing reordering network is composed of a non-linear multi-layer perceptron, and a recognition result optimized based on timing is predicted based on the image features and timing information simultaneously;

[0020] I. The training process of the specific fresh food recognition algorithm based on timing reordering is as follows: Randomly select a sequence S, initialize an empty timestamp buffer C with a maximum buffer quantity of K. Assume that the total number of categories of all images in the sequence S is J. Feature extraction and category prediction are performed on each image in the recognition sequence, specifically including:

[0021] Step 1, as Figure 1 , extract the image feature M of the selected input image through the feature extraction network N. The size of the feature M is (1, J);

[0022] Step 2, as Figure 1 , query the most recent appearance time R and the most recent appearance frequency E of all categories from the timestamp buffer C. Assume that category j appears n times and the last appearance position is m (m < K), then R j = m / K and E j = n / K. The final timing feature is composed of E and R of each category, and the size is (2, J);

[0023] Step 3, splice the image feature and the timing feature to obtain a mixed feature with a size of (3, J), and input the mixed feature into the timing reordering network O to obtain the final prediction result Y with a size of (1, J). Among them, the structure of O is a three-layer fully connected layer, and there is an additional ReLU activation layer after the first and second fully connected layers;

[0024] Step 4, as Figure 1, based on the cashier's confirmation result of whether the correct category F is in Y based on the recognition result Y, the correct category F is fed back to the timestamp buffer area C. During the training process, the recognition result Y and the correct category F will be used to calculate the loss value.

[0025] 2. Perform model optimization, including:

[0026] In step one, parallelizing training is difficult based on recognition sequence optimization, significantly impacting training efficiency. To accelerate training, we propose a training data preloading strategy. Assuming no recognition occurs, but still retaining the feedback process and updating the timestamp cache, we iterate over all images in each recognition sequence in timestamp order, recording each image's path, the corresponding timestamp cache state, and the image's true category. This cache of three components enables efficient parallel training of the fresh food recognition model.

[0027] Step 2: Select the cross entropy loss function to calculate the final classification loss value, calculate the gradient of the parameters of the feature extraction network and the time series reordering network, and optimize the parameters of the neural network in combination with the Adam optimizer.

[0028] 3. Conduct model testing and reasoning, including:

[0029] like Figure 1 , initialize the timestamp buffer for a set of fresh produce image recognition sequences. Resample and normalize the pixel values ​​of each fresh produce recognition image to obtain an input vector of size (1,3,224,224). This is input into the temporal reordering-based fresh produce recognition method to obtain an image feature M of size (1,J). The baseline method directly regards M as the final prediction result, that is, the recognition confidence of each category among a total of J categories. However, our proposed temporal reordering-based fresh produce recognition method queries the most recent appearance time R and the most recent appearance frequency E of each category from the timestamp buffer C to obtain the temporal feature T. We concatenate M and T and input them into the temporal reordering network O to obtain the final fresh produce recognition prediction result Y.

[0030] To further illustrate the present invention, two embodiments are listed below:

[0031] A comparative experiment on fresh produce recognition was conducted using fresh produce data from two different supermarket chains. To verify the effectiveness of the temporal reordering-based fresh produce recognition method, the data was collected from two large supermarket chains in Beijing (A) and Hangzhou (B), each with significantly different fresh produce types. The number of samples collected from the datasets for supermarket chains A and B is shown in Table 1.

[0032] Table 1 Dataset composition

[0033] A chain supermarket B chain supermarket Number of training set sequences 10 10 Number of training set images 105,000 63,000 Number of validation set sequences 2 2 Number of validation set images 22,000 15,000 Number of test set sequences 5 5 Number of test set images 80,000 40,000 Total number of categories (J) 300 260

[0034] like Figure 2 As shown, in the training and inference of the baseline method, the output of the feature extraction network is directly used as the prediction result Y. The fresh food identification method based on time series reordering according to the present invention additionally considers the time series features T of each category under the time series data stream by maintaining a timestamp buffer. The time series features are spliced ​​with the image features, and the time series reordering network O is called to re-predict the confidence of each category. During the training process, the classification loss between the predicted result Y and the true result F is calculated based on the cross entropy function. The Adam optimizer is used to optimize the parameters of the feature extraction network and the time series reordering network. The model obtained after each round of training will be tested on the validation set to obtain the accuracy of the current fresh food identification method on the validation set. Finally, the model with the highest accuracy on the validation set is selected. According to one embodiment of the present invention, the default number of training rounds is 80 rounds, and the maximum size K of the timestamp buffer defaults to 5000.

[0035] In order to simultaneously verify the advantages of the fresh food identification method based on temporal reordering and the generalization ability across models, three commonly used benchmark neural networks MobileNetV2, ResNet18, and ResNet50 were selected according to an embodiment of the present invention. In Table 2, the embodiment of the fresh food identification method based on temporal reordering according to the present invention is marked with "+C". Based on the same training / validation / test experimental configuration, the inventors conducted comparative experiments with / without temporal reordering on the validation set and the test set. Since fresh food identification has the situation of multiple codes for a single category, the top1 and top5 recognition rates are considered, that is, the probability of the first hit in the recognition result or the first five containing the correct result. All indicator values ​​in Table 2 are the average values ​​of all 5 test sequences.

[0036] Table 2 Comparative experimental results with / without temporal reordering on different datasets

[0037]

[0038] As shown in Table 2, popular and frequently sold commodities (such as pears and apples) often have more than ten different commodity codes, and traditional fresh food recognition neural networks often have low accuracy. Taking ResNet18 as an example, it only achieved a top1 recognition accuracy of 79.5% and 81.5% on the validation sets A and B, and the recognition accuracy on the test set decreased by 0.9% and 0.8%, respectively. In contrast, after adopting the fresh food recognition method based on temporal reordering of the present invention, the recognition accuracy of ResNet18 on the validation set was greatly improved by 4.4% in chain supermarket A and 4.2% in chain restaurant B. For ResNet50, after adopting the fresh food recognition method based on temporal reordering, the top1 and top5 on the validation set of supermarket B increased by 5.4% and 1.7%, respectively. It is worth mentioning that on the test set of chain supermarket A, the temporal reordering method proposed in the present invention achieved similar results to the validation set. For example, for ResNet50, the top1 and top5 only decreased by 0.3% and 0.2%, respectively. The small Kyoto difference between the validation and test sets indicates that the proposed temporal reordering method is also effective in improving the robustness of fresh produce recognition. Similarly, a similar trend is observed for another feature extraction network, MobileNetV2, confirming that the temporal reordering method has a broad effect on improving fresh produce recognition accuracy.

[0039] The magnitude and scale of the data sets used for training / testing are merely examples for illustrating the present invention. Applications, modifications and variations of the present invention will be apparent to those skilled in the art.

[0040] As described above, according to the present invention, due to the use of a fresh food identification method based on time series reordering, the fresh food identification algorithm can adaptively adjust the prediction confidence of each category according to the time series characteristics of the fresh food products sold, so that the fresh food identification algorithm can effectively overcome the fresh food identification problem of single-variety multi-product coding, and can improve the recognition robustness of the fresh food identification neural network. In addition, the identification method can be widely used in classification problems similar to fresh food identification, such as one product with multiple categories and the identification sequences are related in time series.

Claims

1. A fresh food identification method based on time sequence reordering, characterized by include: A) Neural network training steps, including: A1) For a recognition sequence S consisting of a large number of randomly selected images, initialize an empty timestamp buffer C. Assume that the total number of categories of all images in the sequence is J; A2) For each image in the recognition sequence S, extract image features through the feature extraction network N, with size (1, J); A3) querying the most recent occurrence time R and the most recent occurrence frequency E of all categories from the timestamp buffer C; A4) Concatenate the image features and the temporal features to obtain a mixed feature of size (3, J), which is input into the temporal reordering network O to obtain the final prediction result Y of size (1, J); A5) Based on the cashier's feedback on the correct fresh food category F based on the recognition result Y, cache the category F and timestamp of the current image in C. B) Model optimization steps, including: B1) Preloading training data: For each recognition sequence S, all data is traversed in timestamp order. Each image's path, the state of the corresponding timestamp cache, and the image's true category are recorded and cached for parallel training of the fresh produce recognition model. B2) Select the cross entropy loss function to calculate the final classification loss value, calculate the gradient of the parameters of the feature extraction network and the time series reordering network, and optimize the parameters of the neural network in combination with the Adam optimizer. C) Model testing and inference steps, including: Similar to the neural network training steps, for each image in the recognition sequence, the image features are first extracted, and the time series features are obtained by querying the timestamp buffer. The image features and time series features are spliced ​​and input into the time series reordering network to obtain the prediction results of the recognition confidence of each category. Then, the cashier determines whether there is a correct category in the prediction result and / or determines and feeds back the correct category by clicking or querying the fresh food category information, so that the correct category is updated to the timestamp buffer area, thereby completing the recognition of the current image.

2. The method for identifying fresh produce based on time sequence reordering according to claim 1, characterized in that Further including: By setting a timestamp cache area of ​​a certain length to store and update the sales frequency of recently sold goods, and re-sorting the recognition results based on the recognition frequency, the problem of different codes for the same product in the fresh food recognition scenario can be solved, thereby improving the accuracy of fresh food recognition.

3. The method for identifying fresh produce based on time sequence reordering according to claim 1, characterized in that Further including: By setting time series features and splicing image features, the artificial intelligence model's perception of time series information is enhanced, thereby improving the accuracy of fresh food identification.

Images