Intelligent refrigerator fruit and vegetable image recognition system and method based on convolutional neural network

Abstract

The present application discloses a convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system and method, which uses image processing technology and artificial intelligence technology based on deep learning algorithms to extract and learn multi-dimensional semantic features from fruit and vegetable images, and interacts and integrates this multi-dimensional semantic feature information to form a complete comprehensive description feature about the fruit and vegetable category, thereby realizing intelligent identification of fruit and vegetable categories. In this way, through the fruit and vegetable recognition function, the smart refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, facilitate users to understand the inventory status of ingredients, and help users reasonably arrange the order of using ingredients and reduce food waste, thereby improving the intelligent level of food management, helping users to better utilize ingredients and reduce food waste, while also promoting food safety awareness and the practice of healthy eating.

Description

Technical Field

[0001] The present application relates to the field of intelligent recognition, and more specifically, to an intelligent refrigerator fruit and vegetable image recognition system and method based on convolutional neural networks. Background Art

[0002] Refrigerators play a vital role in our daily lives. Specifically, they extend the shelf life of food by maintaining a low temperature, slowing spoilage and helping people preserve food longer and reducing food waste. Furthermore, refrigerators provide organized storage space, enabling people to effectively organize and store a variety of ingredients and foods for convenient daily use. With the rapid development of artificial intelligence (AI), image recognition technology based on convolutional neural networks has been widely applied in various fields. In the smart home sector, smart refrigerators, as an indispensable part of the home, are constantly evolving in functionality.

[0003] In traditional refrigerators, users need to manually check and identify the food in the refrigerator, which is not only time-consuming but also inefficient. Therefore, an optimized smart refrigerator fruit and vegetable image recognition system and method is desired, so that smart refrigerators can help users better utilize food and reduce food waste. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, the present application is proposed. The embodiment of the present application provides a smart refrigerator fruit and vegetable image recognition system and method based on convolutional neural networks, which extracts and learns the multi-dimensional semantic features of fruit and vegetable images from fruit and vegetable images by utilizing image processing technology and artificial intelligence technology based on deep learning algorithms, and interacts and fuses this multi-dimensional semantic feature information to form a complete comprehensive expression feature about the fruit and vegetable categories, thereby realizing intelligent identification of fruit and vegetable categories. In this way, through the fruit and vegetable recognition function, the smart refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, facilitate users to understand the inventory status of ingredients, and help users reasonably arrange the order of using ingredients and reduce food waste, thereby improving the intelligence level of food management, helping users to better utilize ingredients and reduce food waste, while also promoting food safety awareness and the practice of healthy eating.

[0005] According to one aspect of the present application, a method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network is provided, comprising:

[0006] Obtain images of fruits and vegetables to be analyzed;

[0007] Calculating a gradient direction histogram of the fruit and vegetable image to be analyzed;

[0008] Calculating a color gradient histogram of the fruit and vegetable image to be analyzed;

[0009] Performing image semantic feature extraction on the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map;

[0010] The fruit and vegetable recognition result is determined based on feature interaction information among the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

[0011] According to another aspect of the present application, a convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system is provided, which includes:

[0012] An image acquisition module, used to acquire images of fruits and vegetables to be analyzed;

[0013] A gradient direction histogram calculation module, used to calculate the gradient direction histogram of the fruit and vegetable image to be analyzed;

[0014] A color gradient histogram module, used to calculate the color gradient histogram of the fruit and vegetable image to be analyzed;

[0015] An image semantic feature extraction module is used to extract image semantic features from the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map;

[0016] The recognition result generation module is used to determine the fruit and vegetable recognition result based on the feature interaction information among the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map and the fruit and vegetable color semantic feature map.

[0017] Compared with the existing technology, the present application provides a convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system and method, which uses image processing technology and artificial intelligence technology based on deep learning algorithms to extract and learn multi-dimensional semantic features from fruit and vegetable images, and interacts and integrates this multi-dimensional semantic feature information to form a complete comprehensive description feature about the fruit and vegetable category, thereby realizing intelligent identification of fruit and vegetable categories. In this way, through the fruit and vegetable recognition function, the smart refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, facilitate users to understand the inventory status of ingredients, and help users reasonably arrange the order of using ingredients and reduce food waste, thereby improving the intelligent level of food management, helping users to better utilize ingredients and reduce food waste, while also promoting food safety awareness and the practice of healthy eating. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 Flowchart of a method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of the present application;

[0020] Figure 2 This is a system architecture diagram of a method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of the present application;

[0021] Figure 3 Flowchart of the training phase of the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition method according to an embodiment of the present application;

[0022] Figure 4 Flowchart of sub-step S4 of the method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of the present application;

[0023] Figure 5 Flowchart of sub-step S5 of the method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of the present application;

[0024] Figure 6 This is a block diagram of a smart refrigerator fruit and vegetable image recognition system based on a convolutional neural network according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0026] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0027] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0028] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0030] In traditional refrigerators, users need to manually check and identify the food in the refrigerator, which is not only time-consuming but also inefficient. Therefore, an optimized smart refrigerator fruit and vegetable image recognition system and method is desired, so that smart refrigerators can help users better utilize food and reduce food waste.

[0031] In the technical solution of this application, a method for intelligent refrigerator fruit and vegetable image recognition based on convolutional neural network is proposed. Figure 1 This is a flowchart of a method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of the present application. Figure 2 : This is a system architecture diagram of the method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, a method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network includes the following steps: S1, obtaining a fruit and vegetable image to be analyzed; S2, calculating a gradient direction histogram of the fruit and vegetable image to be analyzed; S3, calculating a color gradient histogram of the fruit and vegetable image to be analyzed; S4, performing image semantic feature extraction on the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map; S5, determining a fruit and vegetable recognition result based on feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

[0032] Specifically, S1 and S2 acquire a fruit and vegetable image to be analyzed; and calculate a gradient directional histogram of the fruit and vegetable image to be analyzed. The gradient directional histogram is a method used in image processing to describe the local gradient features of an image. In the fields of computer vision and image processing, a gradient generally refers to the direction in which pixel values ​​in an image change most rapidly. A gradient directional histogram statistically analyzes and distributes the gradient directional information of each pixel in an image to describe local features such as texture and edges. In the technical solution of the present application, by calculating the gradient directional histogram of the fruit and vegetable image to be analyzed, local features such as edges and texture can be extracted, thereby helping the system better understand the structure and features of the fruit and vegetable image. Specifically, the gradient directional histogram can help identify edge information in the fruit and vegetable image to be analyzed. Edge information in fruit and vegetable images is often crucial for identifying different types of fruits and vegetables, as it often represents their shape and outline. The gradient directional histogram can effectively capture this edge information, thereby helping the system better understand the structural features of the fruit and vegetable image. For example, if the fruit and vegetable image to be analyzed contains apples and oranges, the two fruits are mixed together. By calculating the gradient direction histogram, the model can extract edge information in the image, such as the boundary between an apple and an orange. This edge information helps the system distinguish between apples and oranges, as they often have different shapes and contours. Furthermore, the gradient direction histogram can also be used to describe the texture characteristics of an image. Different textures typically correspond to different gradient direction distributions. By counting gradient information in different directions, the texture characteristics of the image can be reflected, helping to distinguish different types of fruits and vegetables.

[0033] In particular, S3 calculates the color gradient histogram of the fruit and vegetable image to be analyzed. A color gradient histogram is a method for describing color changes in an image. It combines color information and gradient information to represent the distribution of different colors in the image in different directions. By calculating the color gradient histogram of the fruit and vegetable image to be analyzed, color change information can be extracted from the fruit and vegetable image to be analyzed, helping the system better understand the color distribution and changes of the fruit and vegetable image. Specifically, the color gradient histogram can reflect the distribution of different colors in the image and help extract the color features of the image. For a fruit and vegetable image recognition system, color is one of the important features for distinguishing different types of fruits and vegetables. Calculating the color gradient histogram can help the system better understand the color information of the fruit and vegetable image. Specifically, in a specific example of the present application, calculating the color gradient histogram of the fruit and vegetable image to be analyzed includes: converting the fruit and vegetable image to be analyzed into a Lab color space diagram; dividing the Lab color space diagram into cells to obtain multiple cells; calculating the color gradients of the multiple cells and generating multiple cell color gradient histograms based on the color gradient distribution; and generating the color gradient histogram based on the multiple cell color gradient histograms.

[0034] In particular, the S4 is to extract image semantic features from the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map. In particular, in a specific example of the present application, if Figure 4 As shown, the S4 includes: S41, using a deep learning network model to perform feature extraction on the analyzed fruit and vegetable image, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map; S42, using a nonlinear response compensator to process the fruit and vegetable image semantic feature map to obtain the corrected fruit and vegetable image semantic feature map.

[0035] Specifically, S41 utilizes a deep learning network model to perform feature extraction on the analysis fruit and vegetable image, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. In particular, in one specific example of the present application, the fruit and vegetable image to be analyzed is passed through an image feature extractor based on a convolutional neural network model to obtain a fruit and vegetable image semantic feature map. A convolutional neural network (CNN) is an artificial neural network specifically designed for processing data with a grid-like structure. Its characteristic is that it extracts features from an image layer by layer through convolutional and pooling layers. In this way, the CNN can learn the feature representations contained in the fruit and vegetable image to be analyzed, from low-level features (such as edges and texture) to high-level semantic features (such as shape and category), which helps the system better understand the semantic information of the fruit and vegetable image. At the same time, the gradient direction histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain a fruit and vegetable gradient semantic feature map; and the color gradient histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain a fruit and vegetable color semantic feature map. In other words, the convolutional neural network model is used to further extract the gradient implicit features and color implicit features from the gradient direction histogram and the color gradient histogram of the fruit and vegetable image to be analyzed, so that the fruit and vegetable gradient semantic feature map and the fruit and vegetable color semantic feature map have more abstract and representative feature representations. More specifically, the fruit and vegetable image to be analyzed is passed through an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the fruit and vegetable image, including: using each layer of the image feature extractor based on the convolutional neural network model to perform the following on the input data in the forward pass of the layer: convolution processing on the input data to obtain a convolution feature map; pooling the convolution feature map based on a local feature matrix to obtain a pooled feature map; and nonlinear activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the image feature extractor based on the convolutional neural network model is the semantic feature map of the fruit and vegetable image, and the input of the first layer of the image feature extractor based on the convolutional neural network model is the fruit and vegetable image to be analyzed.

[0036] Specifically, in S42, a nonlinear response compensator is used to process the semantic feature map of the fruit and vegetable image to obtain the corrected semantic feature map of the fruit and vegetable image. It should be understood that a nonlinear response compensator is typically used to correct images to eliminate distortion caused by the nonlinear response of devices such as sensors or displays. In the technical solution of the present application, considering that the semantic feature map of the fruit and vegetable image is obtained by convolution processing of the fruit and vegetable image to be analyzed using a convolutional neural network model, its feature distribution manifold in high-dimensional space is restricted by the convolution kernel. In other words, the feature values ​​at each position in the semantic feature map of the fruit and vegetable image represent the correlation relationship of the local neighborhood space in the fruit and vegetable image to be analyzed. Since the weights in the convolution kernel are the same, the local neighborhood space correlation relationship in the semantic feature map of the fruit and vegetable image has the same degree of importance. This processing method may lead to the neglect of important features and the overemphasis of secondary features, affecting the accuracy of the final fruit and vegetable category judgment. Therefore, it is expected that a nonlinear response compensator can be used to manifest and strengthen this nonlinear association relationship, thereby compensating for the nonlinear response mapping association between the image semantic features of the fruit and vegetable images to be analyzed and the fruit and vegetable categories, so that the semantic features of the fruit and vegetable images can be more accurately portrayed and expressed. More specifically, the nonlinear response compensator is used to process the semantic feature map of the fruit and vegetable images to obtain the semantic feature map of the corrected fruit and vegetable images, including: processing the semantic feature map of the fruit and vegetable images using the following nonlinear response compensation formula to obtain the semantic feature map of the corrected fruit and vegetable images; wherein the nonlinear response compensation formula is:

[0037] ;in, is the pixel value at each position of the semantic feature map of the fruit and vegetable image, A, B, C and D are adjustment parameters with different values, It is the pixel value at each position of the semantic feature map of the fruit and vegetable image after correction.

[0038] In particular, the S5 determines the fruit and vegetable recognition result based on the feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. In particular, in a specific example of the present application, Figure 5 As shown, the S5 includes: S51, passing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map; S52, passing the fruit and vegetable multi-channel image semantic fusion feature map through a fruit and vegetable recognition decision device based on a classification function to obtain a fruit and vegetable recognition result, and the fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed.

[0039] Specifically, in S51, the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map. That is, in the technical solution of the present application, the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map. In this way, the multi-dimensional features contained in the fruit and vegetable image to be analyzed, namely edge and texture information, color and color information, and more abstract image semantic information, are fused through the multi-channel feature fusion network to obtain a more comprehensive and integrated feature representation. Among them, unlike simply summing pixels or splicing channels of the feature map, the multi-channel feature fusion network complements and communicates with each other by capturing the difference information of multi-source heterogeneous features between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. Specifically, the multi-channel feature fusion network connects the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map in series, and then uses batch normalization to balance the data distribution of the features, thereby using the semantic information provided by the high-level features to guide and induce the multi-channel feature fusion, so that the network can correctly focus on the important information of the fruit and vegetable image, thereby generating more discriminative fusion features. More specifically, the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map, including: processing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map with the following multi-channel feature fusion formula to obtain the fruit and vegetable multi-channel image semantic fusion feature map; wherein, the multi-channel feature fusion formula is:

[0040] ;in, is the semantic feature map of the fruit and vegetable image after correction, is the fruit and vegetable gradient semantic feature map, is the semantic feature map of fruit and vegetable color, Represents a cascade function, represents convolution processing, represents the batch normalization function, represents the ReLu activation function, is a multi-channel fusion feature map, represents global average pooling processing, represents the Sigmoid activation function, is the weight feature vector, is the vector element of the b channel corresponding to the fruit and vegetable gradient semantic feature map in the weighted feature vector, is the vector element of the c channel corresponding to the fruit and vegetable color semantic feature map in the weighted feature vector, represents the weighting function, It means element-wise addition. It is the semantic fusion feature map of the fruit and vegetable multi-channel image.

[0041] Specifically, in S52, the semantic fusion feature map of the fruit and vegetable multi-channel image is passed through a fruit and vegetable recognition decision maker based on a classification function to obtain a fruit and vegetable recognition result, and the fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed. That is, in the technical solution of the present application, the semantic fusion feature map of the fruit and vegetable multi-channel image is passed through a fruit and vegetable recognition decision maker based on a classification function to obtain a fruit and vegetable recognition result, and the fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed. More specifically, the semantic fusion feature map of the fruit and vegetable multi-channel image is expanded into a classification feature vector based on a row vector or a column vector; the classification feature vector is fully connected encoded using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and the encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

[0042] It should be understood that before using the above-mentioned neural network model for inference, the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the classification function-based fruit and vegetable identification and decision maker need to be trained. In other words, the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition method of this application also includes a training phase for training the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the classification function-based fruit and vegetable identification and decision maker.

[0043] Figure 3 Flowchart of the training phase of the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition method according to an embodiment of the present application. Figure 3As shown, the method for recognizing fruit and vegetable images of a smart refrigerator based on a convolutional neural network according to an embodiment of the present application includes: a training stage, including: S110, obtaining training data, the training data including training fruit and vegetable images to be analyzed, and the true value of the fruit and vegetable category label of the training fruit and vegetable images to be analyzed; S120, calculating the gradient direction histogram of the training fruit and vegetable images to be analyzed; S130, calculating the color gradient histogram of the training fruit and vegetable images to be analyzed; S140, passing the training fruit and vegetable images to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable image semantic feature map; S150, using the nonlinear response compensator to process the training fruit and vegetable image semantic feature map to obtain a training corrected fruit and vegetable image semantic feature map; S160, passing the gradient direction histogram of the training fruit and vegetable images to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable gradient semantic feature map; S170, passing the training fruit and vegetable images to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable gradient semantic feature map. The color gradient histogram of the image is passed through the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable color semantic feature map; S180, the training corrected fruit and vegetable image semantic feature map, the training fruit and vegetable gradient semantic feature map and the training fruit and vegetable color semantic feature map are passed through the multi-channel feature fusion network to obtain a training fruit and vegetable multi-channel image semantic fusion feature map; S190, the training fruit and vegetable multi-channel image semantic fusion feature map is passed through the fruit and vegetable recognition decision device based on the classification function to obtain a classification loss function value; S200, the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network and the fruit and vegetable recognition decision device based on the classification function are trained with the classification loss function value, wherein each time the training fruit and vegetable multi-channel image semantic fusion feature map is subjected to a training iteration of classification regression by the fruit and vegetable recognition decision device based on the classification function, the training fruit and vegetable multi-channel image semantic fusion feature map is optimized.

[0044] In particular, in the technical solution of the present application, the semantic feature map of the fruit and vegetable image after training correction expresses the image semantic features of the training fruit and vegetable image to be analyzed through the correction of local nonlinear feature response, and the training fruit and vegetable gradient semantic feature map and the training fruit and vegetable color semantic feature map respectively express the gradient direction semantic features and color gradient semantic features of the training fruit and vegetable image to be analyzed, and, considering that the correction of the local nonlinear feature response by the nonlinear response compensator will further strengthen the main trend of the image semantic feature distribution, this will result in the semantic feature map of the fruit and vegetable image after training correction, the training fruit and vegetable gradient semantic feature map and the training fruit and vegetable color semantic feature map. The image semantic expression difference based on the feature matrix of the color semantic feature map, so that after the training corrected fruit and vegetable image semantic feature map, the training fruit and vegetable gradient semantic feature map and the training fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network, the obtained training fruit and vegetable multi-channel image semantic fusion feature map will also have feature distribution differences between the various feature matrices, affecting the class probability regression effect based on the overall numerical distribution of the eigenvalues ​​of each feature matrix of the training fruit and vegetable multi-channel image semantic fusion feature map, thereby affecting the training speed of the classification training performed by the fruit and vegetable recognition decision device based on the classification function and the accuracy of the classification results obtained.

[0045] Therefore, the present application optimizes the training fruit and vegetable multi-channel image semantic fusion feature map each time the training fruit and vegetable multi-channel image semantic fusion feature map is subjected to classification regression training iteration by the fruit and vegetable recognition decision device based on the classification function, specifically comprising: first, expanding the training fruit and vegetable multi-channel image semantic fusion feature map into a training fruit and vegetable multi-channel image semantic fusion feature vector , and then the semantic fusion feature vector of the training fruit and vegetable multi-channel image Optimize, specifically, the semantic fusion feature vector of the training fruit and vegetable multi-channel image The optimization includes the following optimization steps: calculating the autocorrelation matrix of the semantic fusion feature vector of the training fruit and vegetable multi-channel image and its own transposition, and calculating the first Hedi The inner product of the row vectors is used as the first The matrix value of the position is obtained to obtain a weight matrix, and then the weight matrix is ​​matrix-multiplied by the autocorrelation matrix, and then further matrix-vector multiplied by the training fruit and vegetable multi-channel image semantic fusion feature vector to obtain a correction vector, and finally the correction vector is dot-multiplied by the training fruit and vegetable multi-channel image semantic fusion feature vector to obtain an optimized training fruit and vegetable multi-channel image semantic fusion feature vector, and the optimized training fruit and vegetable multi-channel image semantic fusion feature vector is restored to an optimized training fruit and vegetable multi-channel image semantic fusion feature map.

[0046] The specific optimization steps are expressed as:

[0047] ;in, Represents the semantic fusion feature vector of training fruit and vegetable multi-channel images, represents the transpose of a vector, represents the autocorrelation matrix, and The first Hedi row vector, represents the weight matrix, The weight matrix The matrix value of the position, represents the height of the autocorrelation matrix, represents the set of real numbers, represents matrix multiplication, represents dot product, Represents the optimized semantic fusion feature vector of training fruit and vegetable multi-channel images.

[0048] That is, based on the self-correlation dimension of the semantic fusion feature vector of the training fruit and vegetable multi-channel image to be modulated, the correlation expansion is performed based on the spatial sub-dimensional complexity of the high-dimensional feature space of the feature distribution, thereby introducing the decomposable correlation dimension offset of the semantic fusion feature vector of the training fruit and vegetable multi-channel image into the heterogeneous correlation embedding space, thereby enhancing the correlation self-consistency through the joint fine-tuning of the decomposable dimension set represented by the heterogeneous correlation embedding space, so as to improve the feature class probability regression effect based on the predetermined category harmony of the feature set of the semantic fusion feature map of the training fruit and vegetable multi-channel image in the associated target classification domain, that is, to improve the training speed of the classification training through the fruit and vegetable recognition decision maker based on the classification function and the accuracy of the classification results obtained.

[0049] In summary, according to the embodiment of the present application, the method for recognizing fruit and vegetable images in a smart refrigerator based on a convolutional neural network is explained. It extracts and learns the multi-dimensional semantic features of fruit and vegetable images from images by utilizing image processing technology and artificial intelligence technology based on deep learning algorithms, and interacts and integrates this multi-dimensional semantic feature information to form a complete comprehensive description feature about the fruit and vegetable categories, thereby realizing intelligent identification of fruit and vegetable categories. In this way, through the fruit and vegetable recognition function, the smart refrigerator can help users quickly identify the types of fruits and vegetables stored in the refrigerator, facilitate users to understand the inventory status of ingredients, and help users reasonably arrange the order of using ingredients and reduce food waste, thereby improving the intelligent level of food management, helping users to better utilize ingredients and reduce food waste, while also promoting food safety awareness and the practice of healthy eating.

[0050] Furthermore, a smart refrigerator fruit and vegetable image recognition system based on convolutional neural network is also provided.

[0051] Figure 6 FIG is a block diagram of a smart refrigerator fruit and vegetable image recognition system based on a convolutional neural network according to an embodiment of the present application. Figure 6 As shown, the intelligent refrigerator fruit and vegetable image recognition system 300 based on a convolutional neural network according to an embodiment of the present application includes: an image acquisition module 310 for acquiring a fruit and vegetable image to be analyzed; a gradient direction histogram calculation module 320 for calculating the gradient direction histogram of the fruit and vegetable image to be analyzed; a color gradient histogram module 330 for calculating the color gradient histogram of the fruit and vegetable image to be analyzed; an image semantic feature extraction module 340 for performing image semantic feature extraction on the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the gradient direction histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map; and a recognition result generation module 350 for determining a fruit and vegetable recognition result based on feature interaction information between the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map.

[0052] As described above, the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server equipped with a convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition algorithm. In one possible implementation, the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the convolutional neural network-based intelligent refrigerator fruit and vegetable image recognition system 300 can also be one of the many hardware modules of the wireless terminal.

[0053] Alternatively, in another example, the convolutional neural network-based smart refrigerator fruit and vegetable image recognition system 300 and the wireless terminal may also be separate devices, and the convolutional neural network-based smart refrigerator fruit and vegetable image recognition system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0054] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network, characterized in that: include: Obtain images of fruits and vegetables to be analyzed; Calculating a gradient direction histogram of the fruit and vegetable image to be analyzed; Calculating a color gradient histogram of the fruit and vegetable image to be analyzed; Performing image semantic feature extraction on the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the color gradient histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map, including: using a deep learning network model to perform feature extraction on the analysis fruit and vegetable image, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the color gradient histogram of the fruit and vegetable image to be analyzed to obtain a fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map; using a nonlinear response compensator to process the fruit and vegetable image semantic feature map to obtain the corrected fruit and vegetable image semantic feature map; The method comprises the following steps: determining a fruit and vegetable recognition result based on feature interaction information among the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map, comprising: passing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map; and passing the fruit and vegetable multi-channel image semantic fusion feature map through a fruit and vegetable recognition decision device based on a classification function to obtain a fruit and vegetable recognition result, wherein the fruit and vegetable recognition result is used to represent a fruit and vegetable category label of the fruit and vegetable image to be analyzed.

2. The method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network according to claim 1, characterized in that: Calculating the color gradient histogram of the fruit and vegetable image to be analyzed includes: Converting the fruit and vegetable image to be analyzed into a Lab color space image; Performing cell division on the Lab color space graph to obtain a plurality of cells; Calculating color gradients of the plurality of cells, and generating a plurality of cell color gradient histograms based on the color gradient distribution; The color gradient histogram is generated based on the multiple cell color gradient histograms.

3. The method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network according to claim 2, characterized in that: Using a deep learning network model, feature extraction is performed on the analysis fruit and vegetable image, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the color gradient histogram of the fruit and vegetable image to be analyzed to obtain a fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map, including: Passing the fruit and vegetable image to be analyzed through an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the fruit and vegetable image; Passing the gradient direction histogram of the fruit and vegetable image to be analyzed through the image feature extractor based on the convolutional neural network model to obtain the fruit and vegetable gradient semantic feature map; The color gradient histogram of the fruit and vegetable image to be analyzed is passed through the image feature extractor based on the convolutional neural network model to obtain the fruit and vegetable color semantic feature map.

4. The method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network according to claim 3, characterized in that: Processing the fruit and vegetable image semantic feature map using a nonlinear response compensator to obtain the corrected fruit and vegetable image semantic feature map includes: The semantic feature map of the fruit and vegetable image is processed using the following nonlinear response compensation formula to obtain the corrected semantic feature map of the fruit and vegetable image; wherein the nonlinear response compensation formula is: ;in, is the pixel value at each position of the semantic feature map of the fruit and vegetable image, A, B, C and D are adjustment parameters with different values, It is the pixel value at each position of the semantic feature map of the fruit and vegetable image after correction.

5. The method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network according to claim 4, characterized in that: The corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are passed through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map, including: The corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map are processed using the following multi-channel feature fusion formula to obtain the fruit and vegetable multi-channel image semantic fusion feature map; wherein the multi-channel feature fusion formula is: ;in, is the semantic feature map of the fruit and vegetable image after correction, is the fruit and vegetable gradient semantic feature map, is the semantic feature map of fruit and vegetable color, Represents a cascade function, represents convolution processing, represents the batch normalization function, represents the ReLu activation function, is a multi-channel fusion feature map, represents global average pooling processing, represents the Sigmoid activation function, is the weight feature vector, is the vector element of the b channel corresponding to the fruit and vegetable gradient semantic feature map in the weighted feature vector, is the vector element of the c channel corresponding to the fruit and vegetable color semantic feature map in the weighted feature vector, represents the weighting function, It means element-wise addition. It is the semantic fusion feature map of the fruit and vegetable multi-channel image.

6. The method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network according to claim 5, characterized in that: The method further includes a training step of training the image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network and the fruit and vegetable recognition decision device based on the classification function.

7. The method for recognizing fruit and vegetable images in an intelligent refrigerator based on a convolutional neural network according to claim 6, characterized in that: The training step comprises: Acquire training data, the training data including training images of fruits and vegetables to be analyzed, and true values ​​of fruit and vegetable category labels of the training images of fruits and vegetables to be analyzed; Calculating a gradient direction histogram of the training fruit and vegetable image to be analyzed; Calculating a color gradient histogram of the training fruit and vegetable image to be analyzed; Passing the training fruit and vegetable image to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a semantic feature map of the training fruit and vegetable image; Processing the training fruit and vegetable image semantic feature map using the nonlinear response compensator to obtain a training corrected fruit and vegetable image semantic feature map; Passing the gradient direction histogram of the training fruit and vegetable image to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable gradient semantic feature map; Passing the color gradient histogram of the training fruit and vegetable image to be analyzed through the image feature extractor based on the convolutional neural network model to obtain a training fruit and vegetable color semantic feature map; The training corrected fruit and vegetable image semantic feature map, the training fruit and vegetable gradient semantic feature map, and the training fruit and vegetable color semantic feature map are passed through the multi-channel feature fusion network to obtain a training fruit and vegetable multi-channel image semantic fusion feature map; Passing the semantic fusion feature map of the training fruit and vegetable multi-channel image through the fruit and vegetable recognition decision device based on the classification function to obtain a classification loss function value; The image feature extractor based on the convolutional neural network model, the nonlinear response compensator, the multi-channel feature fusion network, and the classification function-based fruit and vegetable recognition decision device are trained using the classification loss function value, wherein each time the training fruit and vegetable multi-channel image semantic fusion feature map undergoes classification regression training iterations through the classification function-based fruit and vegetable recognition decision device, the training fruit and vegetable multi-channel image semantic fusion feature map is optimized.

8. An intelligent refrigerator fruit and vegetable image recognition system based on convolutional neural network, characterized in that: include: An image acquisition module, used to acquire images of fruits and vegetables to be analyzed; A gradient direction histogram calculation module, used to calculate the gradient direction histogram of the fruit and vegetable image to be analyzed; A color gradient histogram module, used to calculate the color gradient histogram of the fruit and vegetable image to be analyzed; An image semantic feature extraction module is configured to perform image semantic feature extraction on the fruit and vegetable image to be analyzed, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the color gradient histogram of the fruit and vegetable image to be analyzed to obtain a corrected fruit and vegetable image semantic feature map, a fruit and vegetable gradient semantic feature map, and a fruit and vegetable color semantic feature map, comprising: utilizing a deep learning network model to perform feature extraction on the analyzed fruit and vegetable image, the gradient direction histogram of the fruit and vegetable image to be analyzed, and the color gradient histogram of the fruit and vegetable image to be analyzed to obtain a fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map; and utilizing a nonlinear response compensator to process the fruit and vegetable image semantic feature map to obtain the corrected fruit and vegetable image semantic feature map. The recognition result generation module is used to determine the fruit and vegetable recognition result based on the feature interaction information among the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map. The module comprises: passing the corrected fruit and vegetable image semantic feature map, the fruit and vegetable gradient semantic feature map, and the fruit and vegetable color semantic feature map through a multi-channel feature fusion network to obtain a fruit and vegetable multi-channel image semantic fusion feature map; passing the fruit and vegetable multi-channel image semantic fusion feature map through a fruit and vegetable recognition decision device based on a classification function to obtain a fruit and vegetable recognition result, wherein the fruit and vegetable recognition result is used to represent the fruit and vegetable category label of the fruit and vegetable image to be analyzed.

Images