Convolutional neural network-based fruit and vegetable image identification system and method for smart refrigerator

Abstract

The present application discloses a convolutional neural network-based fruit and vegetable image identification system and method for a smart refrigerator. By using an image processing technology and an artificial intelligence technology based on a deep learning algorithm, image multi-dimensional semantic features are extracted and learned from fruit and vegetable images, and the multi-dimensional semantic feature information is interacted and fused to form complete comprehensive representation features about categories of fruits and vegetables, thereby achieving intelligent identification of the categories of fruits and vegetables. In this way, by means of a fruit and vegetable identification function, the smart refrigerator can help a user quickly identify the types of fruits and vegetables stored in the refrigerator, facilitate the user knowing the inventory of food materials, and help the user reasonably arrange the use order of the food materials and reduce the waste of the food materials, thereby improving the intelligent level of food material management, helping the user better utilize the food materials and reduce the waste of the food materials, and also promoting the practice of food safety awareness and healthy diet.

Description

A Smart Refrigerator Fruit and Vegetable Image Recognition System and Method Based on Convolutional Neural Networks Technical Field

[0001] This application relates to the field of intelligent recognition, and more specifically, to an intelligent refrigerator fruit and vegetable image recognition system and method based on convolutional neural networks. Background Technology

[0002] Refrigerators play a vital role in people's daily lives. Specifically, refrigerators extend the shelf life of food by maintaining a low-temperature environment, slowing down spoilage and helping people preserve food for longer periods, thus reducing food waste. At the same time, refrigerators provide organized storage space, enabling people to effectively organize and store various ingredients and foods for convenient daily access. With the rapid development of artificial intelligence technology, image recognition technology based on convolutional neural networks has been widely applied in various fields. In the smart home field, smart refrigerators, as an indispensable part of the home, are constantly having their functions upgraded.

[0003] In traditional refrigerators, users need to manually inspect and identify the food inside, which is not only time-consuming but also inefficient. Therefore, there is a need for an optimized smart refrigerator fruit and vegetable image recognition system and method, so that smart refrigerators can help users make better use of food and reduce food waste. Technical issues

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a smart refrigerator fruit and vegetable image recognition system and method based on convolutional neural networks. This system utilizes image processing technology and artificial intelligence technology based on deep learning algorithms to extract and learn multi-dimensional semantic features from fruit and vegetable images. These multi-dimensional semantic features are then interacted with and fused to form a comprehensive description of the fruit and vegetable category, thereby achieving intelligent identification of fruit and vegetable categories. Through this fruit and vegetable recognition function, the smart refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, facilitating understanding of food inventory and helping users rationally arrange the order of food use, reducing food waste. This improves the intelligence level of food management, helps users better utilize food, reduces food waste, and also promotes food safety awareness and healthy eating practices. Technical solutions

[0005] According to one aspect of this application, a method for recognizing fruits and vegetables in a smart refrigerator based on a convolutional neural network is provided, comprising:

[0006] Acquire images of the fruits and vegetables to be analyzed;

[0007] Calculate the gradient orientation histogram of the fruit and vegetable image to be analyzed;

[0008] Calculate the color gradient histogram of the fruit and vegetable image to be analyzed;

[0009] Image semantic features are extracted from the fruit and vegetable image to be analyzed, the gradient orientation histogram of the fruit and vegetable image to be analyzed, and the gradient orientation histogram of the fruit and vegetable image to be analyzed to obtain the corrected fruit and vegetable image semantic feature map, fruit and vegetable gradient semantic feature map, and fruit and vegetable color semantic feature map.

[0010] The fruit and vegetable recognition result is determined based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

[0011] According to another aspect of this application, a smart refrigerator fruit and vegetable image recognition system based on a convolutional neural network is provided, comprising:

[0012] The image acquisition module is used to acquire images of the fruits and vegetables to be analyzed.

[0013] The gradient orientation histogram calculation module is used to calculate the gradient orientation histogram of the fruit and vegetable image to be analyzed.

[0014] The color gradient histogram module is used to calculate the color gradient histogram of the fruit and vegetable image to be analyzed.

[0015] The image semantic feature extraction module is used to extract image semantic features from the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed, so as to obtain the corrected fruit and vegetable image semantic feature map, fruit and vegetable gradient semantic feature map, and fruit and vegetable color semantic feature map.

[0016] The recognition result generation module is used to determine the fruit and vegetable recognition result based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. Beneficial effects

[0017] Compared with existing technologies, this application provides a smart refrigerator fruit and vegetable image recognition system and method based on convolutional neural networks. It utilizes image processing technology and artificial intelligence technology based on deep learning algorithms to extract and learn multi-dimensional semantic features from fruit and vegetable images. These multi-dimensional semantic features are then interacted and fused to form a comprehensive description of the fruit and vegetable categories, thereby achieving intelligent identification of fruit and vegetable categories. Through this fruit and vegetable recognition function, the smart refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, making it easier for users to understand their food inventory and to rationally arrange the order of food use, reducing food waste. This improves the intelligence level of food management, helps users better utilize food, reduces food waste, and also promotes food safety awareness and healthy eating practices. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 is a flowchart of a smart refrigerator fruit and vegetable image recognition method based on a convolutional neural network according to an embodiment of this application;

[0020] Figure 2 is a system architecture diagram of the smart refrigerator fruit and vegetable image recognition method based on convolutional neural network according to an embodiment of this application;

[0021] Figure 3 is a flowchart of the training phase of the smart refrigerator fruit and vegetable image recognition method based on convolutional neural network according to an embodiment of this application;

[0022] Figure 4 is a flowchart of sub-step S4 of the smart refrigerator fruit and vegetable image recognition method based on convolutional neural network according to an embodiment of this application;

[0023] Figure 5 is a flowchart of sub-step S5 of the smart refrigerator fruit and vegetable image recognition method based on convolutional neural network according to an embodiment of this application;

[0024] Figure 6 is a block diagram of a smart refrigerator fruit and vegetable image recognition system based on a convolutional neural network according to an embodiment of this application. Embodiments of the present invention

[0025] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0026] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0027] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0028] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0029] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0030] In traditional refrigerators, users need to manually inspect and identify the food inside, which is not only time-consuming but also inefficient. Therefore, there is a need for an optimized smart refrigerator fruit and vegetable image recognition system and method, so that smart refrigerators can help users make better use of food and reduce food waste.

[0031] This application proposes a method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network. Figure 1 is a flowchart of the method according to an embodiment of this application. Figure 2 is a system architecture diagram of the method according to an embodiment of this application. As shown in Figures 1 and 2, the method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of this application includes the following steps: S1, acquiring an image of fruit and vegetable to be analyzed; S2, calculating the gradient direction histogram of the image of fruit and vegetable to be analyzed; S3, calculating the color gradient histogram of the image of fruit and vegetable to be analyzed; S4, extracting image semantic features from the image of fruit and vegetable to be analyzed, the gradient direction histogram of the image of fruit and vegetable to be analyzed, and the gradient direction histogram of the image of fruit and vegetable to obtain a corrected semantic feature map of the fruit and vegetable image, a gradient semantic feature map of the fruit and vegetable image, and a color semantic feature map of the fruit and vegetable image; S5, determining the fruit and vegetable recognition result based on the feature interaction information between the corrected semantic feature map of the fruit and vegetable image, the gradient semantic feature map of the fruit and vegetable image, and the color semantic feature map of the fruit and vegetable image.

[0032] Specifically, steps S1 and S2 involve acquiring an image of the fruit and vegetable to be analyzed and calculating a gradient orientation histogram (HHMA) of the image. The gradient orientation histogram is a method used in image processing to describe local gradient features of an image. In computer vision and image processing, gradient typically refers to the direction in which pixel values ​​change most rapidly in an image. The gradient orientation histogram statistically analyzes and distributes the gradient direction information of each pixel in the image to describe local features such as texture and edges. In the technical solution of this application, by calculating the gradient orientation histogram of the image of the fruit and vegetable to be analyzed, local features such as edges and textures in the image can be extracted, thereby helping the system better understand the structure and features of the fruit and vegetable image. Specifically, the gradient orientation histogram can help identify edge information in the image of the fruit and vegetable to be analyzed. Edge information in fruit and vegetable images is usually very important for identifying different types of fruits and vegetables because it often represents the shape and contour features of the fruits and vegetables. The gradient orientation histogram can effectively capture this edge information, thereby helping the system better understand the structural features of the fruit and vegetable image. For example, if the image of the fruit and vegetable to be analyzed contains apples and oranges, and the two fruits are mixed together... By calculating the histogram of gradient directions, the model can extract edge information from an image, specifically the boundary between apples and oranges. This edge information helps the system distinguish between apples and oranges because their shapes and outlines are often different. Furthermore, the histogram of gradient directions can also be used to describe the texture features of an image. Different textures typically correspond to different gradient direction distributions; by statistically analyzing gradient information in different directions, the texture features of the image can be reflected, which helps distinguish between different types of fruits and vegetables.

[0033] Specifically, in step S3, a color gradient histogram of the fruit and vegetable image to be analyzed is calculated. A color gradient histogram is a method for describing color changes in an image; it combines color information and gradient information to represent the distribution of different colors in different directions within the image. By calculating the color gradient histogram of the fruit and vegetable image to be analyzed, color change information can be extracted from the image, helping the system better understand the color distribution and changes in the image. Specifically, the color gradient histogram reflects the distribution of different colors in an image, which helps in extracting color features. For fruit and vegetable image recognition systems, color is one of the important features for distinguishing different types of fruits and vegetables; calculating the color gradient histogram helps the system better understand the color information of fruit and vegetable images. Specifically, in a specific example of this application, calculating the color gradient histogram of the fruit and vegetable image to be analyzed includes: converting the fruit and vegetable image to be analyzed into a Lab color space image; dividing the Lab color space image into cells to obtain multiple cells; calculating the color gradient of the multiple cells and generating multiple cell color gradient histograms based on the color gradient distribution; and generating the color gradient histogram based on the multiple cell color gradient histograms.

[0034] Specifically, in step S4, image semantic features are extracted from the image of the fruit and vegetable to be analyzed, its gradient orientation histogram, and its gradient orientation histogram to obtain a corrected semantic feature map of the fruit and vegetable image, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map. In a specific example of this application, as shown in Figure 4, step S4 includes: S41, using a deep learning network model to extract features from the image of the fruit and vegetable to be analyzed, its gradient orientation histogram, and its gradient orientation histogram to obtain a semantic feature map of the fruit and vegetable image, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map; and S42, using a nonlinear response compensator to process the semantic feature map of the fruit and vegetable image to obtain the corrected semantic feature map of the fruit and vegetable image.

[0035] Specifically, in step S41, a deep learning network model is used to extract features from the analyzed fruit and vegetable image, the gradient orientation histogram of the fruit and vegetable image to be analyzed, and the gradient orientation histogram of the fruit and vegetable image to be analyzed to obtain a semantic feature map of the fruit and vegetable image, a gradient semantic feature map of the fruit and vegetable image, and a color semantic feature map of the fruit and vegetable image. In a specific example of this application, the fruit and vegetable image to be analyzed is processed by an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the fruit and vegetable image. A convolutional neural network (CNN) is an artificial neural network specifically designed to process data with a grid-like structure. Its characteristic is that it extracts features from the image layer by layer through convolutional layers and pooling layers. Thus, CNN can learn the feature representations contained in the fruit and vegetable image to be analyzed, from low-level features (such as edges and textures) to high-level semantic features (such as shape and category), which helps the system better understand the semantic information of the fruit and vegetable image. Simultaneously, the gradient orientation histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the fruit and vegetable gradient semantic feature map; and the color gradient histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the fruit and vegetable color semantic feature map. That is, the convolutional neural network model is used to further extract the gradient latent features and color latent features from the gradient orientation histogram and color gradient histogram of the fruit and vegetable image to be analyzed, so that the fruit and vegetable gradient semantic feature map and the fruit and vegetable color semantic feature map have more abstract and representative feature representations. More specifically, the image of the fruit and vegetables to be analyzed is processed by an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the fruit and vegetables image. This includes: using each layer of the image feature extractor based on the convolutional neural network model to process the input data in the forward pass of the layer, performing convolution processing on the input data to obtain a convolutional feature map; performing pooling on the convolutional feature map based on a local feature matrix to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activation feature map; wherein, the output of the last layer of the image feature extractor based on the convolutional neural network model is the semantic feature map of the fruit and vegetables image, and the input of the first layer of the image feature extractor based on the convolutional neural network model is the image of the fruit and vegetables to be analyzed.

[0036] Specifically, in step S42, a nonlinear response compensator is used to process the semantic feature map of the fruit and vegetable image to obtain the corrected semantic feature map of the fruit and vegetable image. It should be understood that a nonlinear response compensator is typically used to correct images to eliminate distortion caused by the nonlinear response of devices such as sensors or displays. In the technical solution of this application, considering that the semantic feature map of the fruit and vegetable image is obtained by convolution processing of the fruit and vegetable image to be analyzed through a convolutional neural network model, its feature distribution manifold in high-dimensional space is limited by the convolution kernel. That is, the feature values ​​at each position in the semantic feature map of the fruit and vegetable image represent the correlation relationships in the local neighborhood space of the fruit and vegetable image to be analyzed. Since the weights in the convolution kernel are the same, these local neighborhood spatial correlation relationships in the semantic feature map of the fruit and vegetable image have the same degree of importance. This processing method may lead to the neglect of important features and the overemphasis on secondary features, affecting the accuracy of the final fruit and vegetable category judgment. Therefore, it is expected that a nonlinear response compensator can be used to make this nonlinear correlation explicit and stronger, thereby compensating for the nonlinear response mapping correlation between the image semantic features of the fruit and vegetable images to be analyzed and the fruit and vegetable categories, so that the semantic features of the fruit and vegetable images can be more accurately characterized and expressed. More specifically, processing the semantic feature map of the fruit and vegetable images using a nonlinear response compensator to obtain the corrected semantic feature map of the fruit and vegetable images includes: processing the semantic feature map of the fruit and vegetable images using the following nonlinear response compensation formula to obtain the corrected semantic feature map of the fruit and vegetable images; wherein, the nonlinear response compensation formula is: ;in, Let A, B, C, and D be the pixel values ​​at each location in the semantic feature map of the fruit and vegetable image, where A, B, C, and D are adjustment parameters with different values. The pixel values ​​at each position of the semantic feature map of the corrected vegetable image.

[0037] Specifically, step S5 determines the fruit and vegetable recognition result based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. In a specific example of this application, as shown in Figure 5, step S5 includes: S51, passing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map; S52, passing the fruit and vegetable multi-channel image semantic fusion feature map through a fruit and vegetable recognition decision based on a classification function to obtain a fruit and vegetable recognition result, wherein the fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed.

[0038] Specifically, in step S51, the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are processed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map. That is, in the technical solution of this application, the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are processed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map. In this way, the multi-dimensional features contained in the fruit and vegetable image to be analyzed, namely edge and texture information, color and hue information, and more abstract image semantic information, are fused through the multi-channel feature fusion network to obtain a more comprehensive and integrated feature representation. Unlike simply summing pixels or concatenating channels in the feature maps, the multi-channel feature fusion network captures the differences in multi-source heterogeneous features among the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map to facilitate mutual supplementation and exchange. Specifically, the multi-channel feature fusion network concatenates the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. Batch normalization is then used to balance the data distribution of the features, thereby utilizing the semantic information provided by high-level features to guide and induce multi-channel feature fusion. This allows the network to correctly focus on the important information of the fruit and vegetable image, thus generating more discriminative fusion features. More specifically, passing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map through the multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map includes: processing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map using the following multi-channel feature fusion formula to obtain the fruit and vegetable multi-channel image semantic fusion feature map; wherein, the multi-channel feature fusion formula is: ;in, The semantic feature map of the corrected vegetable image. The gradient semantic feature map of the fruits and vegetables, This is the semantic feature map of the fruit and vegetable colors. Represents a cascade function. This indicates convolution processing. This represents the batch normalization function. This represents the ReLU activation function. For multi-channel fused feature maps, This indicates global average pooling. This represents the Sigmoid activation function. For weighted feature vectors, The vector element of the b-channel corresponding to the fruit and vegetable gradient semantic feature map in the weighted feature vector. The vector element of the c-channel corresponding to the semantic feature map of fruit and vegetable color in the weighted feature vector. Represents the weighting function. This indicates element-wise addition. This is the semantic fusion feature map of the multi-channel image of the fruits and vegetables.

[0039] Specifically, in step S52, the semantic fusion feature map of the multi-channel fruit and vegetable image is passed through a fruit and vegetable recognition decision based on a classification function to obtain a fruit and vegetable recognition result. The fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed. That is, in the technical solution of this application, the semantic fusion feature map of the multi-channel fruit and vegetable image is passed through a fruit and vegetable recognition decision based on a classification function to obtain a fruit and vegetable recognition result. The fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed. More specifically, the semantic fusion feature map of the multi-channel fruit and vegetable image is expanded into a classification feature vector based on row vectors or column vectors; the classification feature vector is fully connected and encoded using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and the encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

[0040] It should be understood that before using the aforementioned neural network model for inference, the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the fruit and vegetable recognition decision based on the classification function need to be trained. In other words, the intelligent refrigerator fruit and vegetable image recognition method based on convolutional neural networks in this application also includes a training phase for training the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the fruit and vegetable recognition decision based on the classification function.

[0041] Figure 3 is a flowchart of the training phase of the smart refrigerator fruit and vegetable image recognition method based on convolutional neural network according to an embodiment of this application. As shown in Figure 3, the smart refrigerator fruit and vegetable image recognition method based on convolutional neural network according to an embodiment of this application includes: a training phase, including: S110, acquiring training data, the training data including training images of fruits and vegetables to be analyzed, and the true values ​​of the fruit and vegetable category labels of the training images of fruits and vegetables to be analyzed; S120, calculating the gradient direction histogram of the training images of fruits and vegetables to be analyzed; S130, calculating the color gradient histogram of the training images of fruits and vegetables to be analyzed; S140, passing the training images of fruits and vegetables to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a semantic feature map of the training images of fruits and vegetables; S150, processing the semantic feature map of the training images of fruits and vegetables using the nonlinear response compensator to obtain a semantic feature map of the training corrected images of fruits and vegetables; S160, passing the gradient direction histogram of the training images of fruits and vegetables to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a gradient semantic feature map of the training images of fruits and vegetables; S170, passing the training images of fruits and vegetables to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a gradient semantic feature map of the training images of fruits and vegetables; The color gradient histogram of the image is processed by the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable color semantic feature map; S180, the trained and corrected fruit and vegetable image semantic feature map, the trained fruit and vegetable gradient semantic feature map, and the trained fruit and vegetable color semantic feature map are processed by the multi-channel feature fusion network to obtain a training fruit and vegetable multi-channel image semantic fusion feature map; S190, the training fruit and vegetable multi-channel image semantic fusion feature map is processed by the fruit and vegetable recognition decision based on the classification function to obtain a classification loss function value; S200, the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the fruit and vegetable recognition decision based on the classification function are trained using the classification loss function value, wherein the training fruit and vegetable multi-channel image semantic fusion feature map is optimized each time the training iteration of the training fruit and vegetable multi-channel image semantic fusion feature map is performed for classification regression by the fruit and vegetable recognition decision based on the classification function.

[0042] Specifically, in the technical solution of this application, the semantic feature map of the trained and corrected fruit and vegetable image expresses the image semantic features of the trained fruit and vegetable image to be analyzed corrected by the local nonlinear feature response. The semantic feature map of the trained fruit and vegetable gradient and the semantic feature map of the trained fruit and vegetable color respectively express the semantic features of the gradient direction and the semantic features of the color gradient of the trained fruit and vegetable image to be analyzed. Furthermore, considering that the nonlinear response compensator's correction of the local nonlinear feature response will further strengthen the main trend of the image semantic feature distribution, this will lead to a change in the semantic feature map of the trained and corrected fruit and vegetable image, the semantic feature map of the trained fruit and vegetable gradient, and the semantic feature map of the trained fruit and vegetable color. The differences in image semantic representation based on feature matrices in color semantic feature maps mean that after passing the trained and corrected fruit and vegetable image semantic feature maps, the trained fruit and vegetable gradient semantic feature maps, and the trained fruit and vegetable color semantic feature maps through a multi-channel feature fusion network, the resulting trained fruit and vegetable multi-channel image semantic fusion feature map will also have differences in feature distribution among the various feature matrices. This affects the class probability regression effect of each feature matrix of the trained fruit and vegetable multi-channel image semantic fusion feature map based on the overall numerical distribution of eigenvalues, thereby affecting the training speed and accuracy of the classification results obtained by classifying the fruit and vegetable recognition decision based on the classification function.

[0043] Therefore, in each training iteration of the training fruit and vegetable multi-channel image semantic fusion feature map through the fruit and vegetable recognition decision based on the classification function, this application optimizes the training fruit and vegetable multi-channel image semantic fusion feature map. Specifically, this includes: firstly, expanding the training fruit and vegetable multi-channel image semantic fusion feature map into a training fruit and vegetable multi-channel image semantic fusion feature vector. Then, the semantic fusion feature vector of the trained fruit and vegetable multi-channel images is processed. Specifically, the optimization involves optimizing the semantic fusion feature vectors of the trained multi-channel fruit and vegetable images. The optimization includes the following steps: calculating the autocorrelation matrix of the training fruit and vegetable multi-channel image semantic fusion feature vector and its transpose, and calculating the first autocorrelation matrix. and the The inner product of the row vectors, as the weight matrix. The matrix values ​​of the positions are used to obtain the weight matrix. Then, the weight matrix is ​​multiplied by the autocorrelation matrix, and then multiplied by the training fruit and vegetable multi-channel image semantic fusion feature vector to obtain the correction vector. Finally, the correction vector is multiplied by the training fruit and vegetable multi-channel image semantic fusion feature vector to obtain the optimized training fruit and vegetable multi-channel image semantic fusion feature vector, and the optimized training fruit and vegetable multi-channel image semantic fusion feature vector is restored to the optimized training fruit and vegetable multi-channel image semantic fusion feature map.

[0044] The specific optimization steps are as follows: ;in, This represents the semantic fusion feature vector of multi-channel fruit and vegetable images trained. Represents the transpose of a vector. Represents the autocorrelation matrix. and The first element of the autocorrelation matrix represents the... and the row vectors Represents the weight matrix. The weight matrix represents the first... The matrix values ​​of the position, Indicates the height of the autocorrelation matrix. Represents the set of real numbers. Represents matrix multiplication. Dot product, This represents the optimized training semantic fusion feature vector of multi-channel fruit and vegetable images.

[0045] In other words, based on the self-association dimension of the training fruit and vegetable multi-channel image semantic fusion feature vector, which is the object to be modulated, the association is extended based on the spatial sub-dimensional complexity of the high-dimensional feature space of the feature distribution. This introduces the decomposable association dimension offset of the training fruit and vegetable multi-channel image semantic fusion feature vector into the heterogeneous association embedding space. Thus, the association self-consistency is enhanced by the joint fine-tuning of the decomposable dimension set represented by the heterogeneous association embedding space. This improves the feature set of the training fruit and vegetable multi-channel image semantic fusion feature map in the feature class probability regression effect based on the predetermined category consistency in the association target classification domain. That is, it improves the training speed and the accuracy of the classification results obtained by classifying the fruit and vegetable recognition decision based on the classification function.

[0046] In summary, the intelligent refrigerator fruit and vegetable image recognition method based on convolutional neural networks according to the embodiments of this application is explained. It utilizes image processing technology and artificial intelligence technology based on deep learning algorithms to extract and learn multi-dimensional semantic features from fruit and vegetable images. These multi-dimensional semantic features are then interacted and fused to form a complete comprehensive description of the fruit and vegetable category, thereby achieving intelligent identification of fruit and vegetable categories. Thus, through the fruit and vegetable recognition function, the intelligent refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, making it easier for users to understand the food inventory and to rationally arrange the order of food use, reducing food waste. This improves the level of intelligent food management, helps users better utilize food, reduces food waste, and also promotes food safety awareness and healthy eating practices.

[0047] Furthermore, a smart refrigerator fruit and vegetable image recognition system based on convolutional neural networks is also provided.

[0048] Figure 6 is a block diagram of a smart refrigerator fruit and vegetable image recognition system based on a convolutional neural network according to an embodiment of this application. As shown in Figure 6, the smart refrigerator fruit and vegetable image recognition system 300 based on a convolutional neural network according to an embodiment of this application includes: an image acquisition module 310, used to acquire fruit and vegetable images to be analyzed; a gradient orientation histogram calculation module 320, used to calculate the gradient orientation histogram of the fruit and vegetable images to be analyzed; a color gradient histogram module 330, used to calculate the color gradient histogram of the fruit and vegetable images to be analyzed; an image semantic feature extraction module 340, used to extract image semantic features from the fruit and vegetable images to be analyzed, the gradient orientation histogram of the fruit and vegetable images to be analyzed, and the gradient orientation histogram of the fruit and vegetable images to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map; and a recognition result generation module 350, used to determine the fruit and vegetable recognition result based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

[0049] As described above, the intelligent refrigerator fruit and vegetable image recognition system 300 based on convolutional neural networks according to embodiments of this application can be implemented in various wireless terminals, such as servers with intelligent refrigerator fruit and vegetable image recognition algorithms based on convolutional neural networks. In one possible implementation, the intelligent refrigerator fruit and vegetable image recognition system 300 based on convolutional neural networks according to embodiments of this application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the intelligent refrigerator fruit and vegetable image recognition system 300 based on convolutional neural networks can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the intelligent refrigerator fruit and vegetable image recognition system 300 based on convolutional neural networks can also be one of many hardware modules of the wireless terminal.

[0050] Alternatively, in another example, the convolutional neural network-based smart refrigerator fruit and vegetable image recognition system 300 and the wireless terminal can also be separate devices, and the convolutional neural network-based smart refrigerator fruit and vegetable image recognition system 300 can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with the agreed data format.

[0051] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network, characterized in that, include: Acquire images of the fruits and vegetables to be analyzed; Calculate the gradient orientation histogram of the fruit and vegetable image to be analyzed; Calculate the color gradient histogram of the fruit and vegetable image to be analyzed; Image semantic features are extracted from the fruit and vegetable image to be analyzed, the gradient orientation histogram of the fruit and vegetable image to be analyzed, and the gradient orientation histogram of the fruit and vegetable image to be analyzed to obtain the corrected fruit and vegetable image semantic feature map, fruit and vegetable gradient semantic feature map, and fruit and vegetable color semantic feature map. The fruit and vegetable recognition result is determined based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

2. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 1, characterized in that, Calculating the color gradient histogram of the fruit and vegetable image to be analyzed includes: The fruit and vegetable images to be analyzed are converted into Lab color space images; The Lab color space map is divided into cells to obtain multiple cells; Calculate the color gradient of the multiple cells, and generate a color gradient histogram of the multiple cells based on the color gradient distribution; The color gradient histogram is generated based on the multiple cell color gradient histograms.

3. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 2, characterized in that, Image semantic feature extraction is performed on the fruit and vegetable image to be analyzed, its gradient orientation histogram, and its gradient orientation histogram to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map, including: The deep learning network model is used to extract features from the analyzed fruit and vegetable image, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain the fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map; The semantic feature map of the fruit and vegetable image is processed using a nonlinear response compensator to obtain the corrected semantic feature map of the fruit and vegetable image.

4. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 3, characterized in that, Feature extraction is performed on the analyzed fruit and vegetable image, the gradient orientation histogram of the fruit and vegetable image to be analyzed, and the gradient orientation histogram of the fruit and vegetable image to be analyzed using a deep learning network model to obtain a semantic feature map of the fruit and vegetable image, a gradient semantic feature map of the fruit and vegetable image, and a color semantic feature map of the fruit and vegetable image, including: The fruit and vegetable images to be analyzed are processed by an image feature extractor based on a convolutional neural network model to obtain semantic feature maps of the fruit and vegetable images. The gradient orientation histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the gradient semantic feature map of the fruit and vegetable; The color gradient histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the fruit and vegetable color semantic feature map.

5. The intelligent refrigerator fruit and vegetable image recognition method based on convolutional neural networks according to claim 4, characterized in that, The semantic feature map of the fruit and vegetable image is processed using a nonlinear response compensator to obtain the corrected semantic feature map of the fruit and vegetable image, including: The semantic feature map of the fruit and vegetable image is processed using the following nonlinear response compensation formula to obtain the corrected semantic feature map of the fruit and vegetable image; wherein, the nonlinear response compensation formula is: ;in, Let A, B, C, and D be the pixel values ​​at each location in the semantic feature map of the fruit and vegetable image, where A, B, C, and D are adjustment parameters with different values. The pixel values ​​at each position of the semantic feature map of the corrected vegetable image.

6. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 5, characterized in that, The fruit and vegetable recognition result is determined based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map, including: The corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map. The semantic fusion feature map of the multi-channel fruit and vegetable image is passed through a fruit and vegetable recognition decision based on a classification function to obtain the fruit and vegetable recognition result, which is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed.

7. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 6, characterized in that, The corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map, including: The semantic feature map of the corrected fruit and vegetable image, the semantic feature map of the fruit and vegetable gradient, and the semantic feature map of the fruit and vegetable color are processed using the following multi-channel feature fusion formula to obtain the multi-channel image semantic fusion feature map of the fruit and vegetable; wherein, the multi-channel feature fusion formula is: ;in, The semantic feature map of the corrected vegetable image. The gradient semantic feature map of the fruits and vegetables, This is the semantic feature map of the fruit and vegetable colors. Represents a cascade function. This indicates convolution processing. This represents the batch normalization function. This represents the ReLU activation function. For multi-channel fused feature maps, This indicates global average pooling. This represents the Sigmoid activation function. For weighted feature vectors, The vector element of the b-channel corresponding to the fruit and vegetable gradient semantic feature map in the weighted feature vector. The vector element of the c-channel corresponding to the semantic feature map of fruit and vegetable color in the weighted feature vector. Represents the weighting function. This indicates element-wise addition. This is the semantic fusion feature map of the multi-channel image of the fruits and vegetables.

8. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 7, characterized in that, It also includes a training step: training the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the fruit and vegetable recognition decision device based on the classification function.

9. The method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to claim 8, characterized in that, The training steps include: Acquire training data, which includes training images of fruits and vegetables to be analyzed, and the true values ​​of the fruit and vegetable category labels of the training images of fruits and vegetables to be analyzed; Calculate the gradient orientation histogram of the training images of fruits and vegetables to be analyzed; Calculate the color gradient histogram of the training images of fruits and vegetables to be analyzed; The training fruit and vegetable images to be analyzed are passed through the image feature extractor based on the convolutional neural network model to obtain the semantic feature map of the training fruit and vegetable images; The nonlinear response compensator is used to process the semantic feature map of the trained fruit and vegetable image to obtain the semantic feature map of the trained and corrected fruit and vegetable image. The gradient orientation histogram of the training fruit and vegetable images to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the training fruit and vegetable gradient semantic feature map. The color gradient histogram of the training fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the training fruit and vegetable color semantic feature map. The semantic feature map of the trained and corrected fruit and vegetable image, the semantic feature map of the trained fruit and vegetable gradient, and the semantic feature map of the trained fruit and vegetable color are passed through the multi-channel feature fusion network to obtain the semantic fusion feature map of the trained fruit and vegetable multi-channel image. The semantic fusion feature map of the trained multi-channel fruit and vegetable images is passed through the fruit and vegetable recognition decision device based on the classification function to obtain the classification loss function value; The image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the fruit and vegetable recognition decision based on the classification function are trained using the classification loss function value. In each training iteration of the trained fruit and vegetable multi-channel image semantic fusion feature map through the classification function-based fruit and vegetable recognition decision, the trained fruit and vegetable multi-channel image semantic fusion feature map is optimized.

10. A smart refrigerator fruit and vegetable image recognition system based on convolutional neural networks, characterized in that, include: The image acquisition module is used to acquire images of the fruits and vegetables to be analyzed. The gradient orientation histogram calculation module is used to calculate the gradient orientation histogram of the fruit and vegetable image to be analyzed. The color gradient histogram module is used to calculate the color gradient histogram of the fruit and vegetable image to be analyzed. The image semantic feature extraction module is used to extract image semantic features from the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed, so as to obtain the corrected fruit and vegetable image semantic feature map, fruit and vegetable gradient semantic feature map, and fruit and vegetable color semantic feature map. The recognition result generation module is used to determine the fruit and vegetable recognition result based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

Images