Methods, devices, and media for fruit and vegetable identification based on neural networks and feature matching

Abstract

This invention relates to the field of image processing technology, specifically providing a method, apparatus, and medium for fruit and vegetable identification based on neural networks and feature matching, aiming to solve the problem of difficulty in identifying fruit and vegetable categories during automatic settlement. To this end, the fruit and vegetable identification method based on neural networks and feature matching of this invention includes: acquiring fruit and vegetable images of the category to be identified; performing image preprocessing on the fruit and vegetable images; performing image segmentation on the preprocessed fruit and vegetable images of the category to be identified; inputting the segmented fruit and vegetable images of the category to be identified into a trained first feature extraction model and a second feature extraction model respectively to obtain features of the fruit and vegetable images of the category to be identified; acquiring the features of fruit and vegetable categories stored in a feature database; and calculating the cosine similarity between the features of the fruit and vegetable images of the category to be identified and the features in the feature database to obtain the category of the fruit and vegetable images of the category to be identified.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically providing a method, apparatus, and medium for fruit and vegetable recognition based on neural networks and feature matching. Background Technology

[0002] With the growth of the population, people's demand for fruits and vegetables has increased significantly. Currently, the sales of fruits and vegetables are mainly concentrated in fruit and vegetable stores or supermarkets. Fruit and vegetable stores or supermarkets have a wide variety of fruits and vegetables. Their payment method is mainly that the cashier selects the fruits and vegetables for the customer, weighs them, and calculates the price of the fruits and vegetables purchased by the customer. This payment method not only requires the store staff to stand next to the weighing and payment equipment and repeatedly enter the prices of fruits and vegetables, but may also cause customers to queue.

[0003] While neural network-based methods for fruit and vegetable recognition exist, most rely solely on neural networks. Firstly, these methods have high data acquisition requirements, needing to collect images of fruits and vegetables in different environments, states, and arrangements. Furthermore, fruit and vegetable classification networks trained on images from individual fruit and vegetable supermarkets are often inapplicable to other supermarkets or stalls. Secondly, these methods don't consider the issue of fruit and vegetable bagging. Fruit and vegetable shops, supermarkets, and farmers' markets typically sell fruits and vegetables in bags, but most bags are not completely transparent, thus obscuring the characteristics of the produce and reducing the recognition success rate.

[0004] Accordingly, there is a need in this field for a new fruit and vegetable identification scheme based on neural networks and feature matching to solve the above problems. Summary of the Invention

[0005] The present invention aims to solve the above-mentioned technical problems, namely, the difficulty in identifying the categories of fruits and vegetables when performing automatic settlement of fruits and vegetables. The present invention provides a fruit and vegetable identification method and device, control device and storage medium based on neural networks and feature matching.

[0006] In a first aspect, the present invention provides a fruit and vegetable identification method based on neural networks and feature matching, the method comprising:

[0007] Acquire images of fruits and vegetables of the type to be identified, placed on a shelf;

[0008] Image preprocessing is performed on the fruit and vegetable images to be identified to obtain the preprocessed images of the fruit and vegetable categories to be identified.

[0009] Image segmentation is performed on the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing to obtain the segmented fruit and vegetable images of the fruit and vegetable categories to be identified.

[0010] The fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation are input into the trained first feature extraction model and second feature extraction model respectively. Based on the trained first feature extraction model and second feature extraction model, the features of the fruit and vegetable images of the fruit and vegetable categories to be identified are obtained.

[0011] Retrieve the features of fruit and vegetable categories stored in the feature database;

[0012] The features of the fruit and vegetable image of the fruit and vegetable category to be identified are compared with the features of the fruit and vegetable categories stored in the feature database. The fruit and vegetable category in the feature database corresponding to the highest cosine similarity value is identified as the category of the fruit and vegetable image of the fruit and vegetable category to be identified.

[0013] In one technical solution of the above-mentioned fruit and vegetable recognition method based on neural networks and feature matching, the step of preprocessing the fruit and vegetable image of the fruit and vegetable category to be identified to obtain the preprocessed fruit and vegetable image of the fruit and vegetable category to be identified includes:

[0014] The fruit and vegetable images of the fruit and vegetable categories to be identified are subjected to dark channel dehazing to obtain the fruit and vegetable images of the fruit and vegetable categories to be identified after dark channel dehazing.

[0015] The fruit and vegetable images to be identified, after undergoing dark channel dehazing, are then subjected to gamma correction processing to obtain gamma-corrected images of the fruit and vegetable categories to be identified.

[0016] In one technical solution of the above-mentioned fruit and vegetable recognition method based on neural networks and feature matching, the step of performing image segmentation on the fruit and vegetable image of the fruit and vegetable category to be identified after image preprocessing to obtain the segmented fruit and vegetable image of the fruit and vegetable category to be identified includes:

[0017] The preprocessed fruit and vegetable images of the fruit and vegetable categories to be identified are input into a trained semantic segmentation network, which then segments the fruit and vegetable images of the corresponding fruit and vegetable categories to be identified.

[0018] In one technical solution of the above-mentioned fruit and vegetable identification method based on neural networks and feature matching, the training process of the first feature extraction model is as follows:

[0019] Obtain the first training dataset, wherein the first training dataset is a dataset of hundreds of fruit and vegetable images collected through the network, not limited to weighing scenarios, and each category of fruit and vegetable images includes dozens to hundreds of images.

[0020] The first feature extraction model is trained using the first training dataset. When the first target loss value determined during the training process is lower than the first threshold, the training of the first feature extraction model is completed.

[0021] In one technical solution of the above-mentioned fruit and vegetable identification method based on neural networks and feature matching, the training process of the second feature extraction model is as follows:

[0022] Obtain a second training dataset, which is a dataset of dozens of fruit and vegetable images limited to weighing scenarios, collected through the network or actively photographed. Each category of fruit and vegetable images includes several to dozens of images.

[0023] The second feature extraction model is trained using the second training dataset. When the second target loss value determined during the training process is lower than the second threshold, the training of the second feature extraction model is complete.

[0024] In one technical solution of the above-mentioned fruit and vegetable recognition method based on neural networks and feature matching, the step of inputting the fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation into the trained first feature extraction model and second feature extraction model respectively, and obtaining the features of the fruit and vegetable images of the fruit and vegetable categories to be identified based on the trained first feature extraction model and second feature extraction model, includes:

[0025] The fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation are input into the trained first feature extraction model. Based on the trained first feature extraction model, the first feature is extracted from the fruit and vegetable images of the fruit and vegetable categories to be identified, and the first feature of the fruit and vegetable images of the fruit and vegetable categories to be identified is obtained.

[0026] The fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation are input into the trained second feature extraction model. Based on the trained second feature extraction model, the second feature of the fruit and vegetable images of the fruit and vegetable categories to be identified is extracted to obtain the second features of the fruit and vegetable images of the fruit and vegetable categories to be identified.

[0027] The first feature and the second feature are spliced ​​together, and the spliced ​​feature is used as the feature of the fruit and vegetable image to be identified as the fruit and vegetable category.

[0028] In one technical solution of the above-mentioned fruit and vegetable identification method based on neural networks and feature matching, the first feature extraction model and the second feature extraction model are RepVgg-A0 neural networks.

[0029] In a second aspect, the present invention provides a fruit and vegetable identification device based on neural networks and feature matching, the device comprising:

[0030] The first acquisition module is used to acquire images of fruits and vegetables of the type to be identified placed on the shelf;

[0031] The image preprocessing module is used to preprocess the fruit and vegetable images of the fruit and vegetable categories to be identified, and obtain the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing.

[0032] The image segmentation module is used to segment the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing, and obtain the segmented fruit and vegetable images of the fruit and vegetable categories to be identified.

[0033] The feature extraction module is used to input the fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation into the trained first feature extraction model and the second feature extraction model respectively, and obtain the features of the fruit and vegetable images of the fruit and vegetable categories to be identified based on the trained first feature extraction model and the second feature extraction model.

[0034] The second acquisition module is used to acquire the features of fruit and vegetable categories stored in the feature database;

[0035] The identification module is used to calculate the cosine similarity between the features of the fruit and vegetable image of the fruit and vegetable category to be identified and the features of the fruit and vegetable categories stored in the feature database, and to identify the fruit and vegetable category in the feature database corresponding to the highest cosine similarity value as the category of the fruit and vegetable image of the fruit and vegetable category to be identified.

[0036] In a third aspect, the present invention provides a control device comprising a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the method described in any of the above-described technical solutions of the fruit and vegetable recognition method based on neural networks and feature matching.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium storing a plurality of program codes adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions of the fruit and vegetable identification method based on neural networks and feature matching.

[0038] The present invention comprises one or more of the following technical solutions: Beneficial effects:

[0039] In implementing the technical solution of this invention, fruits and vegetables can be identified using a neural network and feature matching method: First, images of fruits and vegetables of the category to be identified are acquired, placed on a shelf; image preprocessing is performed on the images of the categories to be identified to obtain preprocessed images of the categories to be identified; image segmentation is performed on the preprocessed images of the categories to be identified to obtain segmented images of the categories to be identified; then, the segmented images of the categories to be identified are input into a trained first feature extraction model and a second feature extraction model, respectively, and features of the images of the categories to be identified are obtained based on the trained first feature extraction model and the second feature extraction model; then, features of fruit and vegetable categories stored in a feature database are acquired; finally, cosine similarity calculation is performed between the features of the images of the categories to be identified and the features of the categories stored in the feature database, and the category of the fruit and vegetable in the feature database corresponding to the highest cosine similarity value is identified as the category of the image of the category to be identified.

[0040] Through the above implementation methods, image preprocessing of fruit and vegetable images to be identified reduces the impact of environmental factors such as plastic bags on the fruits and vegetables and varying light intensities at different times on fruit and vegetable identification; image segmentation of the fruit and vegetable images to be identified accurately segments the fruit and vegetable portions within the images, thereby reducing the impact of fruit and vegetable location and quantity on identification; feature extraction of the fruit and vegetable images to be identified, and calculation of the cosine similarity between the features of the fruit and vegetable images to be identified and the features in the feature database, allows for quick and accurate identification of the fruit and vegetable categories, facilitating the rapid and accurate acquisition of the prices of the fruits and vegetables purchased by customers. Attached Figure Description

[0041] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0042] Figure 1 This is a schematic diagram of the main steps of a fruit and vegetable identification method based on neural networks and feature matching according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic flowchart of the main steps of step S102 according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram illustrating the segmentation of a fruit and vegetable portion of an image using a semantic segmentation network according to an embodiment of the present invention;

[0045] Figure 4 This is a schematic flowchart of the main steps of step S106 according to an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of a fruit and vegetable identification device based on neural networks and feature matching according to an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of a control device according to an embodiment of the present invention. Detailed Implementation

[0048] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0049] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0050] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a fruit and vegetable identification method based on neural networks and feature matching according to an embodiment of the present invention. Figure 1 As shown, the fruit and vegetable identification method based on neural networks and feature matching in this embodiment of the invention mainly includes the following steps S101-S106.

[0051] Step S101: Obtain images of fruits and vegetables of the type to be identified placed on the shelf.

[0052] In this embodiment of the invention, the fruits and vegetables that need to be weighed are bagged and placed on a shelf. The fruits and vegetables of the type to be identified are photographed on the shelf to obtain images of the fruits and vegetables of the type to be identified.

[0053] Step S102: Perform image preprocessing on the fruit and vegetable images of the fruit and vegetable categories to be identified, to obtain the preprocessed fruit and vegetable images of the fruit and vegetable categories to be identified.

[0054] Step S103: Perform image segmentation on the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing to obtain image segmented fruit and vegetable images of the fruit and vegetable categories to be identified.

[0055] Step S104: Input the fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation into the trained first feature extraction model and second feature extraction model respectively, and obtain the features of the fruit and vegetable images of the fruit and vegetable categories to be identified based on the trained first feature extraction model and second feature extraction model.

[0056] In this embodiment of the invention, both the first feature extraction model and the second feature extraction model are RepVgg-A0 neural networks. The RepVgg-A0 neural network is a feature extraction network with fewer parameters, faster inference speed, and higher classification accuracy.

[0057] To extract image features more accurately, this embodiment of the invention employs two feature extraction models—a first feature extraction model and a second feature extraction model—to extract features from fruit and vegetable images of the categories to be identified. The results of the two models are then concatenated, and the concatenated features are used as the overall features of the fruit and vegetable image for the category to be identified. Compared to using a single feature extraction model, this method can improve the accuracy of image feature extraction by approximately 3.5%, with a particularly significant improvement in accuracy for fruit and vegetable images covered in plastic bags in weighing scenarios.

[0058] Step S105: Obtain the features of fruit and vegetable categories stored in the feature database.

[0059] In this embodiment of the invention, the feature database stores the features of the fruit and vegetable categories currently sold in the fruit and vegetable store. These features are obtained based on the collected images of the fruit and vegetable currently sold in the fruit and vegetable store and the trained first feature extraction model and second feature extraction model.

[0060] It should be noted that the features of fruit and vegetable categories stored in the feature database are updated in real time according to the number of fruit and vegetable varieties. For example, if a fruit and vegetable store currently sells 30 kinds of fruits and vegetables, the feature database will store the features corresponding to these 30 kinds of fruits and vegetables. As the number of fruit and vegetable varieties currently sold in the fruit and vegetable store increases or decreases, the feature database will add or delete features accordingly.

[0061] Step S106: Perform cosine similarity calculation between the features of the fruit and vegetable image of the fruit and vegetable category to be identified and the features of the fruit and vegetable categories stored in the feature database, and identify the fruit and vegetable category in the feature database corresponding to the highest cosine similarity value as the category of the fruit and vegetable image of the fruit and vegetable category to be identified.

[0062] In a specific example, the feature A of the fruit and vegetable image to be identified is compared with the feature B of the fruit and vegetable category stored in the feature database. m Cosine similarity is calculated using the following formula:

[0063]

[0064] Where m is the number of fruit and vegetable categories stored in the feature database; A i B is the i-th feature of the fruit and vegetable image for which the fruit and vegetable category to be identified; mi Let i be the i-th feature of the fruit and vegetable category stored in the m-th feature database.

[0065] Based on the above cosine similarity calculation formula, m cosine similarities are obtained. The m cosine similarities are sorted in descending order, and the cosine similarity at the first position is the highest cosine similarity value. The fruit and vegetable category in the feature database corresponding to the highest cosine similarity value is identified as the category of the fruit and vegetable image to be identified.

[0066] By employing the methods described in steps S101 to S106 above, image preprocessing of the fruit and vegetable images to be identified reduces the impact of environmental factors such as plastic bags on the fruits and vegetables and varying light intensities at different times on fruit and vegetable identification; image segmentation of the fruit and vegetable images to be identified accurately segments the fruit and vegetable portions within the images, thereby reducing the influence of fruit and vegetable location and quantity on fruit and vegetable identification; and feature extraction of the fruit and vegetable images to be identified, along with calculating the cosine similarity between the features of the fruit and vegetable images to be identified and the features in the feature database, allows for rapid and accurate identification of the fruit and vegetable categories, facilitating the quick and accurate acquisition of the prices of the fruits and vegetables purchased by customers.

[0067] The following provides a further explanation of step S102 in the above method embodiment.

[0068] In some embodiments of step S102 above, such as Figure 2 As shown, the step of preprocessing the fruit and vegetable images to be identified, to obtain preprocessed images of the fruit and vegetable categories to be identified, includes:

[0069] Step S1021: Perform dark channel dehazing on the fruit and vegetable images of the fruit and vegetable categories to be identified, and obtain the fruit and vegetable images of the fruit and vegetable categories to be identified after dark channel dehazing.

[0070] Step S1022: Perform gamma correction processing on the fruit and vegetable images of the fruit and vegetable categories to be identified after dark channel dehazing to obtain gamma-corrected fruit and vegetable images of the fruit and vegetable categories to be identified.

[0071] In this embodiment of the invention, when customers buy fruits and vegetables in fruit and vegetable stores or supermarkets, they often put bags on the fruits and vegetables before weighing them. Although plastic bags have a certain degree of transparency, they still have a significant impact on the recognition of image features. Therefore, in order to reduce the impact of putting plastic bags on the extraction of fruit and vegetable image features, it is necessary to first perform an image preprocessing on the obtained fruit and vegetable images of the category to be identified after bagging. The image preprocessing method is to reduce the occlusion of fruit and vegetable features by the plastic bag through dark channel dehazing, making the fruit and vegetable features more obvious.

[0072] Subsequently, due to changes in time and other factors throughout the day, the lighting conditions vary at different times, affecting the recognition of image features. Therefore, to improve image brightness, another image preprocessing is required for the fruit and vegetable images to be identified. After performing dark channel dehazing to reduce the occlusion of fruit and vegetable features by the plastic bag, another image preprocessing method is to further improve image brightness through gamma correction to balance the lighting. Finally, the image size of the fruit and vegetable images to be identified after image preprocessing is adjusted, for example, maintaining the aspect ratio between the original and adjusted image sizes, before proceeding to step S103.

[0073] In some embodiments of step S103 above, the step of performing image segmentation on the fruit and vegetable image of the fruit and vegetable category to be identified after image preprocessing to obtain the segmented fruit and vegetable image of the fruit and vegetable category to be identified includes:

[0074] The preprocessed fruit and vegetable images of the fruit and vegetable categories to be identified are input into a trained semantic segmentation network, which then segments the fruit and vegetable images of the corresponding fruit and vegetable categories to be identified.

[0075] In this embodiment of the invention, due to the different quantities of fruits and vegetables purchased by each customer and the inconsistent placement of the fruits and vegetables on the trays when weighing them, directly extracting features from the entire image of the fruit and vegetable category to be identified after image preprocessing may still result in many interference factors. Furthermore, it may be difficult to identify the features of smaller fruits and vegetables such as tangerines and cherry tomatoes from the entire image.

[0076] To address these issues, this invention inputs preprocessed fruit and vegetable images (representing the fruit and vegetable categories to be identified) into a trained semantic segmentation network. Semantic segmentation refers to the semantic segmentation of an image, that is, dividing the image into regions with specific semantic meanings using a certain method, and identifying the semantic category of each region to obtain a pixelated dense classification. In the prior art, semantic segmentation network technology is relatively mature, and this part is not the focus of this invention, so it will not be described further.

[0077] Taking an embodiment of the present invention as an example, a semantic segmentation network can be used to identify fruit and vegetable regions. This semantic segmentation network is used to segment fruits and vegetables. By inputting images of fruits and vegetables of the category to be identified and the background image of the tray on the shelf, the semantic segmentation network can accurately segment the corresponding fruit and vegetable parts in the images of the category to be identified, and to a certain extent remove the influence of plastic bags. Figure 3 This diagram illustrates how a semantic segmentation network is used to segment fruit and vegetable portions of an image.

[0078] In this embodiment of the invention, after segmenting the fruit and vegetable portion of the fruit and vegetable image to be identified, the portion with the largest connected component is selected. After resizing the image, the resized image is sent to the image feature extraction part, that is, to the trained first feature extraction model and the trained second feature extraction model. Then, based on the first feature and the second feature output by the trained first feature extraction model and the second feature extraction model respectively, the features of the fruit and vegetable image to be identified are obtained.

[0079] In one embodiment of the present invention, before obtaining the trained first feature extraction model, the training of the first feature extraction model and the second feature extraction model is further included.

[0080] In one embodiment of the present invention, the training process of the first feature extraction model is as follows:

[0081] Obtain the first training dataset, wherein the first training dataset is a dataset of hundreds of fruit and vegetable images collected through the network, not limited to weighing scenarios, and each category of fruit and vegetable images includes dozens to hundreds of images.

[0082] The first feature extraction model is trained using the first training dataset. When the first target loss value determined during the training process is lower than the first threshold, the training of the first feature extraction model is completed.

[0083] In a specific example, the first target loss value can use cross-entropy as the loss function, or it can use additive angular margin loss (ArcFace) or ternary loss, etc. This embodiment of the invention uses cross-entropy as the loss function as an example. Cross-entropy can be used as a loss function in a neural network, where p represents the distribution of the true labels and q is the distribution of the predicted labels of the trained model. The cross-entropy loss function can measure the similarity between p and q. When the first target loss value determined during training is lower than a first threshold, the distribution of the predicted labels q of the trained model in the cross-entropy loss approximates the distribution p of the true labels, at which point the training of the first feature extraction model is complete.

[0084] In this example, the first training dataset used for training is a dataset of hundreds of categories of fruits and vegetables, such as nearly 300 categories, collected from the network. This dataset is rich in categories and has a large number of images, with dozens to hundreds of images for each category of fruits and vegetables. However, some of the images were taken in scenes that are very different from weighing scenes. That is, the image data in the first training dataset is not limited to fruit and vegetable images in weighing scenes. For example, it also includes fruit and vegetable images taken in scenes such as planting fruit trees and harvesting vegetables.

[0085] In one embodiment of the present invention, before obtaining the trained second feature extraction model, the process further includes training the second feature extraction model. The training process of the second feature extraction model is as follows:

[0086] Obtain a second training dataset, which is a dataset of dozens of fruit and vegetable images limited to weighing scenarios, collected through the network or actively photographed. Each category of fruit and vegetable images includes several to dozens of images.

[0087] The second feature extraction model is trained using the second training dataset. When the second target loss value determined during the training process is lower than the second threshold, the training of the second feature extraction model is complete.

[0088] In a specific example, the second target loss value is similar to the first target loss value. It can also use cross-entropy as the loss function, additive angular margin loss (ArcFace) as the loss function, or ternary loss as the loss function, etc. In this embodiment of the invention, the loss function can be selected according to the specific circumstances of training the second feature extraction model. When the second target loss value determined during training is lower than the second threshold, the training of the second feature extraction model is complete.

[0089] In this example, the second training dataset differs significantly from the first training dataset. Specifically, the second training dataset consists of dozens of fruit and vegetable images collected online or staged, limited to weighing scenarios. This dataset has fewer categories, approximately 50, and all images are from fruit and vegetable weighing scenarios in supermarkets or fruit and vegetable stores. During the training of the second feature extraction model, because some images in the second training dataset were taken without bags, special processing was added. For example, images in the second training dataset were randomly fogged to simulate being covered by plastic bags; and images containing various empty plastic bags were collected online and segmented, then randomly pasted onto images in the second training dataset with varying degrees of transparency. Experiments showed that these special processing methods significantly improved the detection accuracy of fruits and vegetables covered by plastic bags in the images, increasing the accuracy by 5.5%.

[0090] Following the above implementation method, after the first feature extraction model and the second feature extraction model are trained, and after image preprocessing and image segmentation are performed on the fruit and vegetable images of the fruit and vegetable categories to be identified, the fruit and vegetable images of the fruit and vegetable categories to be identified obtained after image preprocessing and image segmentation need to be input into the trained first feature extraction model and the second feature extraction model to obtain the features of the fruit and vegetable images of the fruit and vegetable categories to be identified.

[0091] In some embodiments of step S104 above, such as Figure 4 As shown, the step of calculating the cosine similarity between the features of the fruit and vegetable image of the fruit and vegetable category to be identified and the features of the fruit and vegetable categories stored in the feature database, and identifying the fruit and vegetable category in the feature database corresponding to the highest cosine similarity value as the category of the fruit and vegetable image of the fruit and vegetable category to be identified, includes:

[0092] Step S1041: Input the fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation into the trained first feature extraction model. Based on the trained first feature extraction model, perform first feature extraction on the fruit and vegetable images of the fruit and vegetable categories to be identified to obtain the first features of the fruit and vegetable images of the fruit and vegetable categories to be identified.

[0093] Step S1042: Input the fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation into the trained second feature extraction model. Based on the trained second feature extraction model, perform second feature extraction on the fruit and vegetable images of the fruit and vegetable categories to be identified to obtain the second features of the fruit and vegetable images of the fruit and vegetable categories to be identified.

[0094] Step S1043: The first feature and the second feature are spliced ​​together, and the spliced ​​feature is used as the feature of the fruit and vegetable image of the fruit and vegetable category to be identified.

[0095] In a specific example, the first feature extraction model outputs the first feature of the fruit and vegetable image of the fruit and vegetable category to be identified, which is a 512-dimensional feature. The second feature extraction model outputs the second feature of the fruit and vegetable image of the fruit and vegetable category to be identified, which is also a 512-dimensional feature. The two 512-dimensional features are concatenated to obtain a 1024-dimensional feature, which is the feature of the fruit and vegetable image of the fruit and vegetable category to be identified.

[0096] In some embodiments of step S105 above, obtaining the features of fruit and vegetable categories stored in the feature database includes:

[0097] Collect images of fruits and vegetables for each category currently sold in the fruit and vegetable store;

[0098] Image preprocessing is performed on the fruit and vegetable images of the categories of fruits and vegetables currently sold in the fruit and vegetable store to obtain the preprocessed images of the fruit and vegetable images of the categories of fruits and vegetables currently sold in the fruit and vegetable store.

[0099] Image segmentation is performed on the fruit and vegetable images of the currently sold fruit and vegetable categories in the fruit and vegetable store after image preprocessing to obtain the segmented fruit and vegetable images of the currently sold fruit and vegetable categories in the fruit and vegetable store.

[0100] The fruit and vegetable images of the currently sold fruit and vegetable categories in the fruit and vegetable store are segmented and then input into the trained first feature extraction model and second feature extraction model, respectively. Based on the trained first feature extraction model and second feature extraction model, the features of the fruit and vegetable images of the currently sold fruit and vegetable categories in the fruit and vegetable store are obtained.

[0101] In a specific example, the steps to obtain the features of fruit and vegetable images representing the fruit and vegetable categories currently sold in the fruit and vegetable store are as follows: Based on the trained first feature extraction model, perform first feature extraction on the collected fruit and vegetable images representing the fruit and vegetable categories currently sold in the fruit and vegetable store, obtaining a 512-dimensional feature; based on the trained second feature extraction model, perform second feature extraction on the fruit and vegetable images representing the fruit and vegetable categories currently sold in the fruit and vegetable store, obtaining another 512-dimensional feature; concatenate these two 512-dimensional features to obtain a 1024-dimensional feature, which is the feature of the fruit and vegetable images representing the fruit and vegetable categories currently sold in the fruit and vegetable store. Collect images representing the number of fruit and vegetable categories currently sold in the fruit and vegetable store, repeat the above process, and obtain the features of fruit and vegetable images representing all fruit and vegetable categories currently sold in the fruit and vegetable store, storing them in a feature database.

[0102] To verify the effectiveness of this method, the inventors collected a dataset of 60 types of fruits and vegetables placed on trays, with approximately 100 images for each type. Using the method proposed in this invention, the accuracy rate for identifying fruits and vegetables without plastic bags was 99.7%, and the accuracy rate for identifying fruits and vegetables with plastic bags was 93.9%. The MIOU value of the image segmentation network was 0.973. MIOU is an important indicator of image segmentation accuracy and can be interpreted as the average intersection-union ratio. This invention employs a reparameterized RepVgg-A0 network as the feature extraction network. On the HiSilicon HI3559A chip, this RepVgg-A0 network can achieve a speed of over 500 images per second. Even on devices with limited computing power, the entire process can achieve real-time detection speed, fully meeting the needs of practical applications.

[0103] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that, in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in this order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention. For example, in the embodiments of the present invention, step S105 is executed first, followed by steps S101-S104 and step S106. For example, in the sub-steps of step S102 in the embodiments of the present invention, step S1022 is executed first, followed by step S1021.

[0104] Furthermore, the present invention also provides a fruit and vegetable identification device based on neural networks and feature matching.

[0105] See appendix Figure 5 , Figure 5 This is a main structural block diagram of a fruit and vegetable identification device based on neural networks and feature matching according to an embodiment of the present invention. Figure 5As shown, the fruit and vegetable recognition device based on neural networks and feature matching in this embodiment of the invention mainly includes a first acquisition module 11, an image preprocessing module 12, an image segmentation module 13, a feature extraction module 14, a second acquisition module 15, and a recognition module 16. In some embodiments, the first acquisition module 11 can be configured to acquire images of fruits and vegetables of the category to be identified placed on a shelf. The image preprocessing module 12 can be configured to perform image preprocessing on the images of fruits and vegetables of the category to be identified, obtaining preprocessed images of fruits and vegetables of the category to be identified. The image segmentation module 13 can be configured to perform image segmentation on the preprocessed images of fruits and vegetables of the category to be identified, obtaining segmented images of fruits and vegetables of the category to be identified. The feature extraction module 14 can be configured to input the segmented images of fruits and vegetables of the category to be identified into a trained first feature extraction model and a second feature extraction model, respectively, and obtain features of the images of fruits and vegetables of the category to be identified based on the trained first feature extraction model and second feature extraction model. The second acquisition module 15 can be configured to acquire features of fruit and vegetable categories stored in a feature database. The recognition module 16 can be configured to perform cosine similarity calculation on the features of the fruit and vegetable image of the fruit and vegetable category to be identified and the features of the fruit and vegetable categories stored in the feature database, and identify the fruit and vegetable category in the feature database corresponding to the highest cosine similarity value as the category of the fruit and vegetable image of the fruit and vegetable category to be identified.

[0106] In one implementation, a description of the specific functions can be found in steps S101-S106.

[0107] The aforementioned fruit and vegetable recognition device based on neural networks and feature matching is used to perform... Figure 1 The illustrated embodiments of the fruit and vegetable identification method based on neural networks and feature matching are similar in their technical principles, the technical problems they solve, and the technical effects they produce. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the fruit and vegetable identification device based on neural networks and feature matching can be found in the embodiments of the fruit and vegetable identification method based on neural networks and feature matching, and will not be repeated here.

[0108] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0109] Furthermore, the present invention also provides a control device. For example... Figure 6 As shown, in one embodiment of the control device according to the present invention, the control device includes a processor and a storage device. The storage device can be configured to store a program for executing the fruit and vegetable identification method based on neural networks and feature matching of the above-described method embodiments. The processor can be configured to execute the program in the storage device, which includes, but is not limited to, the program for executing the fruit and vegetable identification method based on neural networks and feature matching of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The control device can be a control device device comprising various electronic devices.

[0110] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program that executes the fruit and vegetable identification method based on neural networks and feature matching described in the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described fruit and vegetable identification method based on neural networks and feature matching. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0111] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0112] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0113] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A fruit and vegetable identification method based on neural networks and feature matching, characterized in that, include: Acquire images of fruits and vegetables of the type to be identified, placed on a shelf; Image preprocessing is performed on the fruit and vegetable images to be identified to obtain the preprocessed images of the fruit and vegetable categories to be identified. Image segmentation is performed on the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing to obtain the segmented fruit and vegetable images of the fruit and vegetable categories to be identified. The fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation are input into the trained first feature extraction model and second feature extraction model respectively. Based on the trained first feature extraction model and second feature extraction model, the features of the fruit and vegetable images of the fruit and vegetable categories to be identified are obtained. Retrieve the features of fruit and vegetable categories stored in the feature database; The features of the fruit and vegetable image of the fruit and vegetable category to be identified are compared with the features of the fruit and vegetable categories stored in the feature database. The fruit and vegetable category in the feature database corresponding to the highest cosine similarity value is identified as the category of the fruit and vegetable image of the fruit and vegetable category to be identified. The training process of the first feature extraction model is as follows: Obtain the first training dataset, wherein the first training dataset is a dataset of hundreds of fruit and vegetable images collected through the network, not limited to weighing scenarios, and each category of fruit and vegetable images includes dozens to hundreds of images. The first feature extraction model is trained using the first training dataset. When the first target loss value determined during the training process is lower than the first threshold, the training of the first feature extraction model is completed. The training process of the second feature extraction model is as follows: Obtain a second training dataset, which is a dataset of dozens of fruit and vegetable images limited to weighing scenarios, collected through the network or actively photographed. Each category of fruit and vegetable images includes several to dozens of images. The second feature extraction model is trained using the second training dataset. When the second target loss value determined during the training process is lower than the second threshold, the training of the second feature extraction model is completed. During the training process of the second feature extraction model using the second training dataset, data augmentation processing is performed on the second training dataset. The data augmentation processing includes: randomly adding fog to the images in the second training dataset to simulate the situation of being covered by a plastic bag; and / or randomly pasting images containing various empty plastic bags onto the images in the second training dataset with a certain transparency.

2. The fruit and vegetable identification method based on neural networks and feature matching according to claim 1, characterized in that, The step of preprocessing the fruit and vegetable images to be identified, resulting in preprocessed images of the fruit and vegetable categories to be identified, includes: The fruit and vegetable images of the fruit and vegetable categories to be identified are subjected to dark channel dehazing to obtain the fruit and vegetable images of the fruit and vegetable categories to be identified after dark channel dehazing. The fruit and vegetable images to be identified, after undergoing dark channel dehazing, are then subjected to gamma correction processing to obtain gamma-corrected images of the fruit and vegetable categories to be identified.

3. The fruit and vegetable identification method based on neural networks and feature matching according to claim 1 or 2, characterized in that, The step of performing image segmentation on the fruit and vegetable images of the to-be-identified fruit and vegetable categories after image preprocessing to obtain segmented fruit and vegetable images of the to-be-identified fruit and vegetable categories includes: The preprocessed fruit and vegetable images of the fruit and vegetable categories to be identified are input into a trained semantic segmentation network, which then segments the fruit and vegetable images of the corresponding fruit and vegetable categories to be identified.

4. The fruit and vegetable identification method based on neural networks and feature matching according to claim 1, characterized in that, The process involves inputting the segmented fruit and vegetable images of the desired category into a pre-trained first feature extraction model and a second feature extraction model, respectively. Based on these models, features of the fruit and vegetable images representing the desired category are obtained, including: The fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation are input into the trained first feature extraction model. Based on the trained first feature extraction model, the first feature is extracted from the fruit and vegetable images of the fruit and vegetable categories to be identified, and the first feature of the fruit and vegetable images of the fruit and vegetable categories to be identified is obtained. The fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation are input into the trained second feature extraction model. Based on the trained second feature extraction model, the second feature of the fruit and vegetable images of the fruit and vegetable categories to be identified is extracted to obtain the second features of the fruit and vegetable images of the fruit and vegetable categories to be identified. The first feature and the second feature are spliced ​​together, and the spliced ​​feature is used as the feature of the fruit and vegetable image to be identified as the fruit and vegetable category.

5. The fruit and vegetable identification method based on neural networks and feature matching according to claim 1, characterized in that, The first feature extraction model and the second feature extraction model are RepVgg-A0 neural networks.

6. A fruit and vegetable identification device based on neural networks and feature matching, characterized in that, include: The first acquisition module is used to acquire images of fruits and vegetables of the type to be identified placed on the shelf; The image preprocessing module is used to preprocess the fruit and vegetable images of the fruit and vegetable categories to be identified, and obtain the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing. The image segmentation module is used to segment the fruit and vegetable images of the fruit and vegetable categories to be identified after image preprocessing, and obtain the segmented fruit and vegetable images of the fruit and vegetable categories to be identified. The feature extraction module is used to input the fruit and vegetable images of the fruit and vegetable categories to be identified after image segmentation into the trained first feature extraction model and the second feature extraction model respectively, and obtain the features of the fruit and vegetable images of the fruit and vegetable categories to be identified based on the trained first feature extraction model and the second feature extraction model. The second acquisition module is used to acquire the features of fruit and vegetable categories stored in the feature database; The identification module is used to calculate the cosine similarity between the features of the fruit and vegetable image of the fruit and vegetable category to be identified and the features of the fruit and vegetable categories stored in the feature database, and to identify the fruit and vegetable category in the feature database corresponding to the highest cosine similarity value as the category of the fruit and vegetable image of the fruit and vegetable category to be identified. The training process of the first feature extraction model is as follows: Obtain the first training dataset, wherein the first training dataset is a dataset of hundreds of fruit and vegetable images collected through the network, not limited to weighing scenarios, and each category of fruit and vegetable images includes dozens to hundreds of images. The first feature extraction model is trained using the first training dataset. When the first target loss value determined during the training process is lower than the first threshold, the training of the first feature extraction model is completed. The training process of the second feature extraction model is as follows: Obtain a second training dataset, which is a dataset of dozens of fruit and vegetable images limited to weighing scenarios, collected through the network or actively photographed. Each category of fruit and vegetable images includes several to dozens of images. The second feature extraction model is trained using the second training dataset. When the second target loss value determined during the training process is lower than the second threshold, the training of the second feature extraction model is completed. During the training process of the second feature extraction model using the second training dataset, data augmentation processing is performed on the second training dataset. The data augmentation processing includes: randomly adding fog to the images in the second training dataset to simulate the situation of being covered by a plastic bag; and / or randomly pasting images containing various empty plastic bags onto the images in the second training dataset with a certain transparency.

7. A control device, comprising a processor and a storage device, said storage device being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to perform the fruit and vegetable identification method based on neural networks and feature matching as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the fruit and vegetable identification method based on neural networks and feature matching as described in any one of claims 1 to 5.

Images