Fruit and vegetable intelligent settlement method and system based on machine vision

Abstract

The invention discloses a fruit and vegetable intelligent settlement method and system based on machine vision, and relates to the technical field of machine vision, and the method comprises the steps: obtaining a fruit and vegetable image in a fruit and vegetable placement region; acquiring actual weights of all fruits and vegetables; determining the types of fruits and vegetables in the fruit and vegetable placement area through a preset fruit and vegetable type prediction model; determining a first weight value of each fruit and vegetable type based on the fruit and vegetable image; determining a second weight value of each fruit and vegetable type based on the fruit and vegetable density; determining a comprehensive weight value of each fruit and vegetable type according to the first weight value and the second weight value; according to the comprehensive weight value and the actual weight of each fruit and vegetable type, the settlement weight of each fruit and vegetable type is distributed; and taking the sum of the products of the settlement weights of all the fruit and vegetable types and the corresponding unit price amounts as a settlement price and outputting the settlement price. The problem of low manual settlement efficiency can be effectively solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine vision technology, and in particular to a method and system for intelligent settlement of fruits and vegetables based on machine vision. Background Art

[0002] In retail places such as fresh food supermarkets and farmers' markets, the settlement process of fruits and vegetables usually involves multiple links such as weighing, identifying product types, and calculating prices. Specialized staff are required to perform weighing, settlement and other operations, which has high labor costs. In addition, during peak hours, there is a demand for customers to weigh a variety of fruits and vegetables, but because staff need to weigh, pack and code each type of fruit and vegetable separately, the processing speed of weighing and settlement is particularly slow, which cannot meet the needs of such customers for fast settlement. At the same time, staff may need to serve multiple customers and products at the same time, resulting in low efficiency of manual settlement, which in turn affects the operational efficiency of the retail industry.

[0003] Therefore, a solution is urgently needed to solve the above problems. Summary of the invention

[0004] The embodiments of the present application provide a method and system for intelligent settlement of fruits and vegetables based on machine vision, which are used to solve the problem of low efficiency of manual settlement.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for intelligent settlement of fruits and vegetables based on machine vision is provided, which is applied to an intelligent settlement system for fruits and vegetables. The intelligent settlement system for fruits and vegetables includes a touch screen, a weighing module, and an image acquisition module. The method includes:

[0007] In response to a settlement instruction from a user through the touch screen, the image acquisition module acquires images of fruits and vegetables in the fruit and vegetable placement area;

[0008] Obtaining the actual weight of all fruits and vegetables in the fruit and vegetable placement area through the weighing module;

[0009] Inputting the fruit and vegetable image into a preset fruit and vegetable category prediction model to determine at least one fruit and vegetable category in the fruit and vegetable placement area, wherein each of the fruit and vegetable categories corresponds to at least one fruit and vegetable;

[0010] Determine a first weight value for each type of the fruit and vegetable based on the fruit and vegetable image;

[0011] Determine the density of fruits and vegetables corresponding to each of the fruit and vegetable types in the first preset database, and determine a second weight value for each of the fruit and vegetable types based on the density of fruits and vegetables;

[0012] Determining a comprehensive weight value of each type of fruit and vegetable according to the first weight value and the second weight value;

[0013] Allocate the settlement weight of each type of fruits and vegetables according to the comprehensive weight value and the actual weight of each type of fruits and vegetables;

[0014] Determine the unit price amount corresponding to each type of fruit and vegetable in the second preset database;

[0015] The sum of the products of the settlement weights of all the fruit and vegetable types and the corresponding unit prices is output as the settlement price.

[0016] In another possible implementation manner of the first aspect, determining the first weight value of each type of fruit and vegetable based on the fruit and vegetable image includes:

[0017] For each type of fruit and vegetable, extract the contour area of ​​each fruit and vegetable through the fruit and vegetable image, and summarize the contour areas of all the fruits and vegetables to obtain the contour area of ​​the fruits and vegetables corresponding to each type of fruit and vegetable;

[0018] Summarizing the contour areas of the fruits and vegetables corresponding to all the fruit and vegetable types to obtain a total contour area;

[0019] Based on the fruit and vegetable contour area corresponding to each of the fruit and vegetable types and the sum of the contour areas, a first weight value of each of the fruit and vegetable types is determined.

[0020] In another possible implementation manner of the first aspect, determining the second weight value of each type of fruit and vegetable based on the fruit and vegetable density includes:

[0021] Summarizing the density of the fruits and vegetables corresponding to all the types of fruits and vegetables to obtain a total density;

[0022] Based on the fruit and vegetable density corresponding to each of the fruit and vegetable types and the total density, a second weight value of each of the fruit and vegetable types is determined.

[0023] In another possible implementation manner of the first aspect, determining the comprehensive weight value of each type of fruit and vegetable according to the first weight value and the second weight value includes:

[0024] Acquire fruit and vegetable area data, fruit and vegetable density data, and fruit and vegetable quality data corresponding to the fruit and vegetable types in the first preset database;

[0025] Using a preset first correlation coefficient formula, the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data is calculated to obtain a first weight coefficient of the first weight value;

[0026] Using a preset second correlation coefficient formula, the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data is calculated to obtain a second weight coefficient of the second weight value;

[0027] Based on the first weight value, the first weight coefficient, the second weight value and the second weight coefficient, a comprehensive weight of each type of fruits and vegetables is calculated.

[0028] In another possible implementation manner of the first aspect, after acquiring the images of fruits and vegetables in the fruit and vegetable placement area by the image acquisition module, the method includes:

[0029] Extracting edge features and color features of the fruit and vegetable images;

[0030] When the color feature is a preset color and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, a reminder message is issued and displayed on the touch screen, wherein the reminder message is used to prompt the user to remove the packaging bag or box of the fruits and vegetables.

[0031] In another possible implementation of the first aspect, the step of constructing the fruit and vegetable variety prediction model includes:

[0032] Determine the image size in the fruit and vegetable feature training set;

[0033] Determining the spatial dimension of the input layer of the fruit and vegetable variety prediction model by using the image size in the fruit and vegetable feature training set;

[0034] Determining the number of color channels of the input layer of the fruit and vegetable variety prediction model according to the fruit and vegetable color features;

[0035] Determine the number of neurons in the output layer of the fruit and vegetable variety prediction model through all the fruit and vegetable varieties;

[0036] The fruit and vegetable category prediction model is trained by using the fruit and vegetable feature training set, and the fruit and vegetable category prediction model is constructed by combining the spatial dimension of the input layer, the number of color channels, and the number of neurons in the output layer.

[0037] In another possible implementation of the first aspect, the step of constructing the fruit and vegetable feature training set includes:

[0038] Using the image acquisition module, collect pictures of fruits and vegetables with different light intensities, pictures of fruits and vegetables in panoramic scenes, pictures of fruits and vegetables in local scenes, and pictures of fruits and vegetables in different placement methods;

[0039] The fruit and vegetable pictures with different light intensities, the panoramic scene fruit and vegetable pictures, the local scene fruit and vegetable pictures and the fruit and vegetable pictures with different placement methods are respectively used to construct a light intensity sample set, a panoramic scene sample set, a local scene sample set and a placement method sample set;

[0040] Performing image space geometric transformation on each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement mode sample set to obtain a transformed illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement mode sample set;

[0041] Performing image contrast adjustment on each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement method sample set to obtain illumination intensity sample sets, the panoramic scene sample set, the local scene sample set, and the placement method sample set at different contrasts;

[0042] The illumination intensity sample sets under different contrasts and after the transformation, the panoramic scene sample sets, the local scene sample sets, and the placement method sample sets are used to construct a fruit and vegetable feature training set.

[0043] In another possible implementation of the first aspect, the fruit and vegetable type prediction model includes a low-level feature extraction layer and a high-level feature extraction layer, and inputting the fruit and vegetable image into a preset fruit and vegetable type prediction model to determine the fruit and vegetable type of each fruit and vegetable in the fruit and vegetable placement area includes:

[0044] Extracting a feature map of basic features of the fruit and vegetable image through the low-level feature extraction layer, wherein the basic features include image edge features, image color features, and image texture features;

[0045] Extracting a feature map of semantic information of the fruit and vegetable image through the high-level feature extraction layer, wherein the semantic information includes object semantic information, attribute semantic information and scene global semantic information in the fruit and vegetable image;

[0046] Transmitting the feature map of the basic feature to the high-level feature extraction layer using a preset feature channel;

[0047] By horizontally connecting the feature map of the basic feature and the feature map of the semantic information, feature fusion is performed to obtain the final feature of the fruit and vegetable image;

[0048] The final feature is matched with the feature corresponding to each type of fruit and vegetable in the third preset database to determine the type of each fruit and vegetable in the fruit and vegetable placement area.

[0049] In a second aspect, the present application provides an electronic device, including:

[0050] a memory configured to store instructions; and

[0051] The processor is configured to call the instructions from the memory and implement the above-mentioned machine vision-based intelligent settlement method for fruits and vegetables when executing the instructions.

[0052] In a third aspect, the present application provides a fruit and vegetable intelligent settlement system, comprising:

[0053] The above electronic equipment;

[0054] A weighing module connected to the electronic device;

[0055] An image acquisition module connected to the electronic device;

[0056] A touch screen is connected to the electronic device, the weighing module and the image acquisition module.

[0057] Through the above technical solution, at least one type of fruit and vegetable in the fruit and vegetable placement area is determined through a preset fruit and vegetable type prediction model, and the types of fruit and vegetable in the fruit and vegetable placement area are identified with the help of advanced machine vision technology and deep learning algorithms, so that multiple types of fruits and vegetables can be determined, reducing manual intervention, and significantly improving the efficiency of fruit and vegetable settlement. The first weight value of each type of fruit and vegetable is determined by the fruit and vegetable image, and the second weight value of each type of fruit and vegetable is determined by the fruit and vegetable density, and the first weight value coefficient and the second weight value coefficient are determined according to the first weight value and the second weight value, and the comprehensive weight value of each type of fruit and vegetable is calculated. The settlement weight of each fruit and vegetable is reasonably allocated through the comprehensive weight value, further improving the accuracy of settlement, considering the dual factors of image recognition and actual weight, and this dual verification mechanism reduces human errors and improves the accuracy of settlement. The fruit and vegetable intelligent settlement system through machine vision can not only settle multiple types of fruits and vegetables at one time, improve settlement efficiency and accuracy, but also enhance user experience, optimize inventory management, and solve the problem of low efficiency of manual settlement.

[0058] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a method for intelligent settlement of fruits and vegetables based on machine vision provided in an embodiment of the present application;

[0060] Figure 2 This is a schematic diagram of the structure of a fruit and vegetable intelligent settlement system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0062] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0063] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0064] Figure 1 The following schematically shows a flow chart of a method for intelligent settlement of fruits and vegetables based on machine vision according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a method for intelligent settlement of fruits and vegetables based on machine vision, which may include the following steps.

[0065] S110, in response to a settlement instruction from a user through the touch screen, acquiring images of fruits and vegetables in the fruit and vegetable placement area through an image acquisition module;

[0066] S120, obtaining the actual weight of all fruits and vegetables in the fruit and vegetable placement area through a weighing module;

[0067] S130, inputting the fruit and vegetable image into a preset fruit and vegetable category prediction model to determine at least one fruit and vegetable category in the fruit and vegetable placement area, wherein each fruit and vegetable category corresponds to at least one fruit and vegetable;

[0068] S140, determining a first weight value for each type of fruit and vegetable based on the fruit and vegetable image;

[0069] S150, determining the density of fruits and vegetables corresponding to each type of fruits and vegetables in the first preset database, and determining a second weight value for each type of fruits and vegetables based on the density of fruits and vegetables;

[0070] S160, determining a comprehensive weight value for each type of fruit and vegetable according to the first weight value and the second weight value;

[0071] S170, allocating a settlement weight for each type of fruit and vegetable according to the comprehensive weight value and actual weight of each type of fruit and vegetable;

[0072] S180, determining the unit price amount corresponding to each type of fruit and vegetable in the second preset database;

[0073] S190: Output the sum of the products of the settlement weights of all types of fruits and vegetables and the corresponding unit prices as the settlement price.

[0074] In response to the settlement instruction of the user through the touch screen, the image acquisition module acquires the images of fruits and vegetables in the fruit and vegetable placement area. That is, the user issues a settlement instruction through the touch screen, and the system first recognizes this instruction and prepares for subsequent image acquisition. Subsequently, in response to the user's settlement instruction, the image acquisition module is activated. In this embodiment, the image acquisition module is a high-resolution camera that can capture high-quality images. The image acquisition module is aimed at the fruit and vegetable placement area to capture the images of fruits and vegetables in the area. The acquired images of fruits and vegetables will then be further processed and recognized.

[0075] Next, the actual weight of all fruits and vegetables in the fruit and vegetable placement area is obtained through the weighing module, that is, after the user issues a settlement instruction through the touch screen and the image acquisition module acquires the fruit and vegetable image, the weighing module will be used to measure the actual weight of all fruits and vegetables in the fruit and vegetable placement area. In this embodiment, the weighing module is usually a weight sensor, which is a device that converts a mass signal into a measurable electrical signal output. When the image acquisition module has completed the capture of the fruit and vegetable image, the weighing module will automatically start and enter the weighing state, and measure the total weight of all fruits and vegetables in the fruit and vegetable placement area through the built-in sensor system.

[0076] Subsequently, the fruit and vegetable images are input into a preset fruit and vegetable category prediction model to determine at least one fruit and vegetable category in the fruit and vegetable placement area. In this embodiment, each fruit and vegetable category corresponds to at least one fruit and vegetable, and the preset fruit and vegetable category prediction model is an image classification model based on a convolutional neural network model. Specifically, first, the image of the fruit and vegetable placement area is input into a preset fruit and vegetable category prediction model. The preset fruit and vegetable category prediction model uses a convolutional layer to perform feature extraction on the input image, and extracts the fruit and vegetable features in the image. Subsequently, the extracted features are mapped to specific fruit and vegetable categories through a fully connected layer, and the fruit and vegetable category is output as a recognition result.

[0077] After determining the types of fruits and vegetables in the fruit and vegetable placement area, the first weight value of each type of fruits and vegetables is determined based on the fruit and vegetable images. In this embodiment, the first weight value is used to evaluate the influence of area on the distribution weight. Specifically, first, the contour area of ​​each fruit and vegetable in the fruit and vegetable placement area is extracted through image recognition technology, and all contour areas are added to obtain the contour area of ​​each type of fruits and vegetables. In this embodiment, the image recognition technology is an image recognition technology based on deep learning. The image recognition technology is to process, analyze and understand images through computers to identify targets and objects of various different modes. Subsequently, the ratio between the contour area corresponding to each type of fruits and vegetables and the face value of all fruit and vegetable contours is used as the first weight value of each type of fruits and vegetables.

[0078] After determining the first weight value, the density of fruits and vegetables corresponding to each type of fruits and vegetables is determined in the first preset database, and the second weight value of each type of fruits and vegetables is determined based on the density of fruits and vegetables. In this embodiment, the second weight value refers to the degree of influence of the density of fruits and vegetables on the distribution weight. Specifically, first, the density of fruits and vegetables corresponding to each type of fruits and vegetables is searched in the first preset database. The density of fruits and vegetables refers to the mass or weight of fruits and vegetables per unit volume, which reflects the physical properties of fruits and vegetables, such as compactness, porosity, etc. Next, based on the obtained density value of fruits and vegetables, the second weight value of each type of fruits and vegetables is determined. That is, according to the size of the density value, a corresponding weight value is assigned to each type of fruits and vegetables. For example, fruits and vegetables with higher density may be assigned higher weight values, and vice versa.

[0079] According to the first weight value and the second weight value, the comprehensive weight value of each type of fruit and vegetable is determined. Specifically, first, the first weight value and the second weight value are standardized to ensure that they are compared on the same dimension. Standardization can be achieved by Z-score standardization, which is the process of converting raw data into deviations from the mean in units of standard deviation. Subsequently, the first weight coefficient and the second weight coefficient are determined for the first weight value and the second weight value, respectively. The correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data can be calculated, and this correlation coefficient can be used as the first weight coefficient; the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data can be calculated, and this correlation coefficient can be used as the second weight coefficient. Next, the first weight value and the second weight value are weighted and summed using the determined weight coefficient to obtain the comprehensive weight value of each type of fruit and vegetable.

[0080] According to the comprehensive weight value and actual weight of each type of fruit and vegetable, the settlement weight of each type of fruit and vegetable is allocated. That is, the comprehensive weight value of each type of fruit and vegetable is multiplied by its actual weight to calculate a settlement weight, thereby allocating the settlement weight of each type of fruit and vegetable.

[0081] The unit price amount corresponding to each type of fruit and vegetable is determined in the second preset database. Specifically, it is necessary to access the second preset database storing information on the types of fruit and vegetables and their unit prices. In the database, information on all types of fruit and vegetables can be retrieved by executing an operation to query the types of fruit and vegetables. For the information on each type of fruit and vegetable retrieved, it is necessary to further search for the corresponding unit price information in all the information, so as to obtain the unit price amount corresponding to each type of fruit and vegetable.

[0082] Subsequently, the sum of the products of the settlement weights of all types of fruits and vegetables and the corresponding unit prices is taken as the settlement price and output. That is to say, when the settlement weights and corresponding unit prices of each type of fruits and vegetables are available, the settlement weights of each type of fruits and vegetables and the corresponding unit prices are multiplied to obtain the corresponding settlement prices of each type of fruits and vegetables, and the settlement amounts of all types of fruits and vegetables are added together to obtain the total settlement price.

[0083] Through the preset fruit and vegetable type prediction model, at least one fruit and vegetable type in the fruit and vegetable placement area is determined. With the help of advanced machine vision technology and deep learning algorithms, the types of fruits and vegetables in the fruit and vegetable placement area can be identified, and multiple types of fruits and vegetables can be determined, reducing manual intervention, so that the efficiency of fruit and vegetable settlement is significantly improved. The first weight value of each fruit and vegetable type is determined by the fruit and vegetable image, and the second weight value of each fruit and vegetable type is determined by the fruit and vegetable density. According to the first weight value and the second weight value, the first weight value coefficient and the second weight value coefficient are determined, and the comprehensive weight value of each fruit and vegetable type is calculated. The settlement weight of each fruit and vegetable is reasonably allocated through the comprehensive weight value, which further improves the accuracy of settlement. Considering the dual factors of image recognition and actual weight, this dual verification mechanism reduces human errors and improves the accuracy of settlement. The fruit and vegetable intelligent settlement system through machine vision can not only settle multiple types of fruits and vegetables at one time, improve settlement efficiency and accuracy, but also enhance user experience, optimize inventory management, and solve the problem of low efficiency of manual settlement.

[0084] In one implementation of this embodiment, determining a first weight value for each type of fruit and vegetable based on the fruit and vegetable image includes:

[0085] S210, for each type of fruit and vegetable, extract the contour area of ​​each fruit and vegetable through the fruit and vegetable image, and summarize the contour areas of all fruits and vegetables to obtain the contour area of ​​fruits and vegetables corresponding to each type of fruit and vegetable;

[0086] S220, summing up the contour areas of fruits and vegetables corresponding to all types of fruits and vegetables to obtain a total contour area;

[0087] S230: Determine a first weight value for each type of fruit and vegetable based on the contour area of ​​the fruit and vegetable corresponding to each type of fruit and vegetable and the total contour area.

[0088] Based on the fruit and vegetable images, the first weight value of each type of fruit and vegetable is determined. First, for each type of fruit and vegetable, the contour area of ​​each fruit and vegetable is extracted through the fruit and vegetable images, and the contour areas of all fruits and vegetables are summarized to obtain the contour area of ​​each type of fruit and vegetable. Specifically, the high-definition images of fruits and vegetables in the fruit and vegetable placement area are taken by a camera, and the high-definition images of fruits and vegetables are preprocessed. Subsequently, the edge detection algorithm is used to detect the edges in the image for the preprocessed image, which can be achieved by the Canny edge detection algorithm. The Canny edge detection algorithm is a multi-level edge detection algorithm. On the basis of edge detection, a contour extraction algorithm can also be used to extract the contour of each fruit and vegetable, such as using the findContours function in OpenCV. OpenCV is an open source computer vision and image processing library. In OpenCV, the findContours function is used to detect contours in an image, and contours can be extracted from an image. After obtaining the contour of each fruit and vegetable, the contour areas of fruits and vegetables corresponding to all types of fruits and vegetables are summarized to obtain the sum of the contour areas, that is, the contour areas of all fruits and vegetables are added together to obtain the total contour area of ​​the fruit and vegetable type.

[0089] Secondly, based on the contour area of ​​each type of fruit and vegetable and the total contour area, the first weight value of each type of fruit and vegetable is determined. That is, according to the contour area of ​​each type of fruit and vegetable, its ratio to the total contour area is calculated as the first weight value. The calculation formula is:

[0090] The first weight value (i) = the contour area of ​​the i-th fruit and vegetable / the sum of the contour areas of all fruits and vegetables

[0091] The first weight value of each type of fruit and vegetable is determined through a calculation formula.

[0092] In another implementation of this embodiment, when the outlines of fruits and vegetables overlap in the fruit and vegetable images, and cameras with multiple angles are provided in the fruit and vegetable intelligent settlement system, first, the fruit and vegetable images at multiple angles are obtained through the cameras, that is, the images of the fruit and vegetable placement area are photographed at multiple angles through the cameras, and the images can be taken from the front, side, top and bottom, etc., to obtain the fruit and vegetable images at multiple angles. In this implementation, there are multiple cameras, and they are provided at different positions to photograph the fruit and vegetable images at multiple angles.

[0093] The combined features of the multi-angle images are determined by using the images of fruits and vegetables at multiple angles. Specifically, first, the convolution layer of the preset convolutional neural network is used to extract the two-dimensional fruit and vegetable contour image features of the fruit and vegetable images at multiple angles. That is to say, the fruit and vegetable images taken at multiple angles are input into the preset convolutional neural network, and the two-dimensional fruit and vegetable contour image features of the fruit and vegetable images at multiple angles are extracted through the convolution layer. The convolution layer is a basic component in the convolutional neural network, which can automatically learn and extract local features in the image, such as edges, textures, and shapes. Through the convolution operation, the spatial structure information in the image can be captured, thereby obtaining the two-dimensional fruit and vegetable contour image features of the fruit and vegetable images.

[0094] Secondly, multiple two-dimensional fruit and vegetable contour image features are input into a preset aggregation network model to obtain multi-angle image combination features. Specifically, a convolutional neural network is used to extract two-dimensional fruit and vegetable contour image features from fruit and vegetable contour images at multiple angles, and the two-dimensional fruit and vegetable contour image features are input into a preset aggregation network model. The preset aggregation network model can be determined according to specific application scenarios and requirements. It can be a simple fully connected network for mapping the spliced ​​or fused features to a new feature space, or it can be a more complex network structure, such as a deep neural network, a convolutional neural network, or a recurrent neural network to capture more advanced features. Next, these features extracted from different angles are aggregated together through a preset aggregation network model to form a unified multi-angle image combination feature. For example, the feature vectors of each angle are spliced ​​together to form a longer feature vector. The features of different angles can also be fused together using fusion strategies such as weighted average, maximum pooling, and minimum pooling.

[0095] The multi-angle image combination features are used to construct a sample feature database, and the initial convolutional neural network is trained to obtain a trained convolutional neural network. Specifically, a large number of fruit and vegetable images at multiple angles are obtained. For each image, a preset convolutional neural network is used to extract the two-dimensional fruit and vegetable contour image features, and the features extracted from different angles are combined to form multi-angle image combination features. The multi-angle combination features of all images are organized into a sample feature database, and the initial convolutional neural network is trained through the sample feature database to obtain a trained convolutional neural network.

[0096] Subsequently, the camera is used to capture images of the current fruit and vegetable placement area at multiple angles to obtain the current fruit and vegetable images. The multiple current fruit and vegetable images are input into the trained convolutional neural network to obtain multiple fruit and vegetable contour features at different angles. Specifically, the camera is used to capture the fruits and vegetables placed in a specific area at multiple angles to obtain the current fruit and vegetable images. Next, these multi-angle fruit and vegetable images will be input into the trained convolutional neural network. In the convolutional neural network, the images will be processed by multiple convolutional layers, pooling layers, and fully connected layers. Each convolutional layer uses a series of convolution kernels to scan the image and extract local features in the image, such as edges, textures, and shapes. Since the input is fruit and vegetable images at multiple angles, the convolutional neural network will process the images at each angle separately and extract the features of each angle to obtain multiple fruit and vegetable contour features at different angles.

[0097] Input multiple fruit and vegetable contour features into a preset point cloud data prediction model to obtain a three-dimensional point cloud prediction model of the fruit and vegetable image. In this embodiment, the three-dimensional point cloud prediction model can be a model based on deep learning. The point cloud data prediction model is a model that can predict the corresponding three-dimensional point cloud data according to the input two-dimensional image features. Input the multiple fruit and vegetable contour features extracted previously into the preset point cloud data prediction model. The model will predict the three-dimensional point cloud data of the fruit and vegetable according to these features.

[0098] The outline of each fruit and vegetable is determined through the three-dimensional point cloud prediction model. That is to say, after being processed by the prediction model, a three-dimensional point cloud prediction model of the fruit and vegetable image is obtained. The model may contain information such as the position, shape, size, etc. of the fruit and vegetable in the three-dimensional space, thereby determining the outline of each fruit and vegetable.

[0099] By processing fruit and vegetable images, extracting the contour areas of fruits and vegetables and determining the first weight value of each type of fruit and vegetable, the contour areas of fruits and vegetables can be accurately measured, the influence of the contour areas of fruits and vegetables on weight distribution can be better determined, and the accuracy of fruit and vegetable weight distribution can be ensured.

[0100] In one implementation of this embodiment, determining the second weight value of each type of fruit and vegetable based on the fruit and vegetable density includes:

[0101] S310, summing up the fruit and vegetable densities corresponding to all fruit and vegetable types to obtain a total density;

[0102] S320: Determine a second weight value for each type of fruit and vegetable based on the fruit and vegetable density corresponding to each type of fruit and vegetable and the total density.

[0103] The second weight value of each type of fruit and vegetable is determined based on the density of fruits and vegetables. First, the fruit and vegetable densities corresponding to all types of fruit and vegetables are summarized to obtain the total density. That is, the fruit and vegetable density corresponding to each type of fruit and vegetable is determined in the first preset database, and the densities of all types of fruit and vegetables are added to obtain the total density. For example, there are apples and bananas in the fruit and vegetable placement area, the density of apples is about 0.95 g / cubic centimeter, and the density of bananas is about 0.9 g / cubic centimeter. Therefore, the total density is 1.85 g / cubic centimeter.

[0104] Secondly, based on the fruit and vegetable density corresponding to each type of fruit and vegetable and the total density, determine the second weight value of each type of fruit and vegetable. That is, after adding the density values ​​corresponding to all types of fruit and vegetable in the fruit and vegetable placement area, the total density is divided by the fruit and vegetable density corresponding to each type of fruit and vegetable to determine the second weight value of each type of fruit and vegetable. For example, there are apples and bananas in the fruit and vegetable placement area. The density of apples is about 0.95 g / cm3, the density of bananas is about 0.9 g / cm3, and the total density is 1.85 g / cm3. The second weight value of apples is determined by the ratio of 0.95 g / cm3 to the total density of 1.85 g / cm3 to obtain the second weight value of apples; the second weight value of bananas is determined by the ratio of 0.9 g / cm3 to 1.85 g / cm3 to obtain the second weight value of bananas.

[0105] By determining the second weight value for each type of fruit and vegetable, the influence of the density of the fruit and vegetable on the weight distribution can be better determined, thereby ensuring the accuracy of the weight distribution of the fruit and vegetable.

[0106] In one implementation of this embodiment, the comprehensive weight value of each type of fruit and vegetable is determined according to the first weight value and the second weight value, including:

[0107] S410, obtaining fruit and vegetable area data, fruit and vegetable density data, and fruit and vegetable quality data corresponding to the fruit and vegetable types in the first preset database;

[0108] S420, using a preset first correlation coefficient formula, calculating the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data to obtain a first weight coefficient of a first weight value;

[0109] S430, using a preset second correlation coefficient formula, calculating the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data to obtain a second weight coefficient of a second weight value;

[0110] S440: Calculate the comprehensive weight of each type of fruit and vegetable based on the first weight value, the first weight coefficient, the second weight value, and the second weight coefficient.

[0111] The fruit and vegetable area data, fruit and vegetable density data and fruit and vegetable quality data corresponding to the fruit and vegetable types are obtained in the first preset database, that is, the fruit and vegetable area data, fruit and vegetable density data and fruit and vegetable quality data corresponding to different fruit and vegetable types are determined in the first preset database.

[0112] After determining the fruit and vegetable area data, the fruit and vegetable density data, and the fruit and vegetable quality data, the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data is calculated using a preset first correlation coefficient formula to obtain a first weight coefficient of a first weight value. Specifically, first, the fruit and vegetable area data and the fruit and vegetable quality data are extracted from a first preset database, wherein each fruit and vegetable type has a corresponding area value and quality value. The fruit and vegetable area data and the fruit and vegetable quality data are calculated using a preset correlation coefficient formula. In this embodiment, the preset first correlation coefficient formula may be a Pearson correlation coefficient. The formula of the Pearson correlation coefficient is as follows:

[0113]

[0114] Among them, x i and i are the i-th observation values ​​of fruit and vegetable area data and fruit and vegetable quality data respectively, and are the means of area and mass, respectively, n is the number of observations, and r represents the Pearson correlation coefficient.

[0115] Substitute the extracted fruit and vegetable area data and fruit and vegetable quality data into the correlation coefficient formula, calculate the correlation coefficient r between the fruit and vegetable area data and the fruit and vegetable quality data, and use this correlation coefficient as the first weight coefficient of the first weight value.

[0116] Next, the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data is calculated using the preset second correlation coefficient formula to obtain the second weight coefficient of the second weight value, and the fruit and vegetable density data and the fruit and vegetable quality data are extracted from the first preset database, wherein each fruit and vegetable type has a corresponding density value and quality value. Similarly, the fruit and vegetable density data and the fruit and vegetable quality data are calculated using the preset correlation coefficient formula. In this embodiment, the preset second correlation coefficient formula can be the Pearson correlation coefficient, which is a statistic that measures the degree of linear correlation between two variables. Substitute the extracted fruit and vegetable area data and the fruit and vegetable quality data into the correlation coefficient formula to calculate the correlation coefficient r between the fruit and vegetable density data and the fruit and vegetable quality data, and use this correlation coefficient as the second weight coefficient of the second weight value.

[0117] Based on the first weight value, the first weight coefficient, the second weight value and the second weight coefficient, the comprehensive weight of each type of fruit and vegetable is calculated. In order to comprehensively consider the two factors of area and density, it is necessary to determine their weight distribution coefficients. For example, if the area has a greater impact on the comprehensive weight, the value of the first weight coefficient will be larger. The comprehensive weight of each type of fruit and vegetable is calculated using the area weight base value and the density weight base value, as well as their weight distribution coefficients. The calculation formula is:

[0118] Comprehensive weight (i) = area weight base value (i) × area weight coefficient + density weight base value (i) × density weight coefficient

[0119] Through the comprehensive weight, the comprehensive weight of each type of fruit and vegetable is obtained.

[0120] Calculating the comprehensive weight value through the area, density and quality data of fruits and vegetables can more comprehensively reflect the overall condition of each type of fruit and vegetable, and using the correlation coefficient formula to calculate the weight coefficient can make the distribution of weights more scientific and objective, thereby improving the scientificity and accuracy of decision-making.

[0121] In one implementation manner of this embodiment, after acquiring the images of fruits and vegetables in the fruit and vegetable placement area through the image acquisition module, the method includes:

[0122] S510, extracting edge features and color features of the fruit and vegetable image;

[0123] S520: When the color feature is a preset color and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, a reminder message is issued and displayed on the touch screen, wherein the reminder message is used to prompt the user to remove the packaging bag or box of the fruit and vegetable.

[0124] After obtaining the fruit and vegetable images in the fruit and vegetable placement area through the image acquisition module, the edge features and color features of the fruit and vegetable images are extracted, that is, the edge information and color information of the image are identified and extracted from the fruit and vegetable images. The edge features of the fruit and vegetable images can be extracted by the Sobel operator, which is a discrete differential operator used for edge detection. The Sobel operator detects the edge by calculating the gradient of the image gray value. The color features can be extracted by color histogram, color moment and other methods. The color histogram counts the number of pixels in each color channel in the image, and the color moment uses the first few moments of the color distribution (such as mean, variance and slope) to describe the color features.

[0125] Subsequently, when the color feature is a preset color and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, a reminder message is issued and displayed on the touch screen. In this embodiment, the reminder message is used to prompt the user to remove the packaging bag or packaging box of the fruit and vegetable. Specifically, first, the fruit and vegetable intelligent settlement system extracts the color feature from the image and compares it with the preset color feature. In this embodiment, the preset color feature can be white, transparent, or the color of the packaging bag and packaging box in the sales place. And the fruit and vegetable intelligent settlement system extracts the edge feature from the image and matches it with the edge feature in the third preset database. If the color feature in the image successfully matches the preset color feature, and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, that is, there is no fruit and vegetable edge feature in the current image, indicating that the packaging bag or packaging box will block the fruit and vegetable to be settled, and it is impossible to identify and settle, then a reminder message will be issued and displayed on the touch screen, reminding the user to remove the packaging bag or packaging box of the fruit and vegetable, and the fruit and vegetable types can be smoothly identified and settled.

[0126] By identifying edge features and color features, it is determined whether there is a packaging bag or box in the current fruit and vegetable placement area, and the user is prompted to remove the packaging bag or box of the fruit and vegetable. These interference factors can be removed through user intervention, which can significantly improve the recognition accuracy and enhance the practicality and generalization ability of the system.

[0127] In one implementation of this embodiment, the steps of constructing the fruit and vegetable category prediction model include:

[0128] S610, determining the image size in the fruit and vegetable feature training set;

[0129] S620, determining the spatial dimension of the input layer of the fruit and vegetable type prediction model by using the image size of the fruit and vegetable feature training set;

[0130] S630, determining the number of color channels of the input layer of the fruit and vegetable variety prediction model according to the fruit and vegetable color features;

[0131] S640, determining the number of neurons in the output layer of the fruit and vegetable variety prediction model through all fruit and vegetable varieties;

[0132] S650, training a fruit and vegetable type prediction model using a fruit and vegetable feature training set, and constructing a fruit and vegetable type prediction model by combining the spatial dimension and the number of color channels of the input layer and the number of neurons of the output layer.

[0133] First, determine the image size in the fruit and vegetable feature training set. In this embodiment, the fruit and vegetable feature training set can be determined according to actual conditions. That is, when constructing the fruit and vegetable feature training set, the images need to be unified into a specific aspect ratio to facilitate subsequent feature extraction and model training. For example, the aspect ratio of the fruit and vegetable feature training set can be unified to help ensure that the images in the training set are consistent. Therefore, the image size can be directly obtained in the preset fruit and vegetable feature training set.

[0134] Subsequently, the spatial dimension of the input layer of the fruit and vegetable type prediction model is determined by the image size in the fruit and vegetable feature training set. That is to say, in deep learning, the convolutional neural network model is one of the most commonly used models for image processing, and the input layer of the convolutional neural network model usually expects the image to have a specific spatial size. For example, if all images in the image dataset are preprocessed to 224x224 pixels, then the model input layer should also be set to accept 224x224 pixel images. Since the convolutional layer, pooling layer, etc. in the convolutional neural network model need to determine the spatial size of the input data to calculate the moving step size of the convolution kernel and the size of the pooling window, it is necessary to determine the spatial dimension of the input layer of the fruit and vegetable type prediction model by the image size in the fruit and vegetable feature training set. Adjust the size of the model input layer according to the size of the image data. For example, if the image is preprocessed to 224x224 pixels, the size of the input layer should be set to 224x224x3.

[0135] After determining the spatial dimension of the input layer of the fruit and vegetable type prediction model, the number of color channels of the input layer of the fruit and vegetable type prediction model is determined by the fruit and vegetable color features. That is, the input layer should have the same number of color channels as the image data. For example, for a color RGB image, there are three color channels: red, green, and blue. Therefore, if the image is an RGB image, the model input layer should be set to receive data of three channels.

[0136] Next, the number of neurons in the output layer of the fruit and vegetable prediction model is determined through all fruit and vegetable categories. Specifically, first, all the fruit and vegetable categories currently in the sales venue need to be determined, and they can be adjusted according to the fruit and vegetable categories sold in real time in the sales venue. The output layer of the model is configured through all fruit and vegetable categories. In the implementation of the neural network, ensure that the number of neurons in the output layer is set to the number of fruit and vegetable categories. Each neuron corresponds to a fruit and vegetable category, and the output value usually represents the probability that the input data belongs to the corresponding category.

[0137] The fruit and vegetable feature training set is used to train the fruit and vegetable category prediction model, and the fruit and vegetable category prediction model is constructed by combining the spatial dimension and number of color channels of the input layer and the number of neurons of the output layer. Specifically, first, the fruit and vegetable feature training set is collected and preprocessed to ensure that the images have a uniform spatial dimension and number of color channels, and the correct label is assigned to each image. Then, a deep learning model is designed, and the input layer can accept the preprocessed image data, that is, the spatial dimension and number of color channels of the input layer are consistent with the images in the fruit and vegetable feature training set. Subsequently, the output layer is set, and its number of neurons is equal to the number of fruit and vegetable categories, and the predicted probability is output using the softmax activation function. The softmax function can convert a vector containing any real number into a vector containing positive numbers and the sum is 1, that is, a probability distribution. In this way, the output of each neuron can be interpreted as the probability that the input image belongs to the corresponding category. Finally, the model is trained using the training set data, and the model parameters are adjusted through the back propagation algorithm to minimize the prediction error. The predicted value is calculated using the training set data through the forward propagation algorithm, and the difference between the predicted value and the actual label is measured by the loss function (such as the cross entropy loss). In order to minimize this difference, i.e. the prediction error, the back propagation algorithm can be used to adjust the parameters of the model. The back propagation algorithm is an important algorithm for training artificial neural networks. It adjusts the weights and bias parameters in the network to minimize a loss function that measures the difference between the network's output and the actual label.

[0138] By designing the input layer and output layer of the model and training the model, the model can be able to efficiently and accurately learn the task of fruit and vegetable variety prediction, and show good prediction performance and generalization ability in practical applications.

[0139] In one implementation of this embodiment, the step of constructing the fruit and vegetable feature training set includes:

[0140] S710, using an image acquisition module to acquire pictures of fruits and vegetables with different light intensities, pictures of fruits and vegetables in panoramic scenes, pictures of fruits and vegetables in local scenes, and pictures of fruits and vegetables in different placement methods;

[0141] S720, constructing a light intensity sample set, a panoramic scene sample set, a local scene sample set, and a placement method sample set from the fruit and vegetable pictures with different light intensities, the fruit and vegetable pictures with panoramic scenes, the fruit and vegetable pictures with local scenes, and the fruit and vegetable pictures with different placement methods, respectively;

[0142] S730, performing image space geometric transformation on each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement mode sample set to obtain a transformed illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement mode sample set;

[0143] S740, adjusting the image contrast of each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement method sample set to obtain illumination intensity sample sets, panoramic scene sample sets, local scene sample sets, and placement method sample sets at different contrasts;

[0144] S750, constructing a fruit and vegetable feature training set by using the transformed illumination intensity sample set under different contrasts, the panoramic scene sample set, the local scene sample set, and the placement method sample set.

[0145] First, use the image acquisition module to collect pictures of fruits and vegetables with different light intensities, panoramic scenes, local scenes, and different placement methods. That is, pictures can be collected under natural light, artificial light, and other environments. For example, pictures can be taken at different time periods such as sunny days, cloudy days, and dusk to obtain pictures under different natural lighting conditions. Use lighting equipment to adjust different light intensities, from weak light to strong light, to simulate various indoor and outdoor lighting environments. Take panoramic scene fruit and vegetable pictures, that is, select multiple angles (such as looking down, looking straight, and looking up) to take panoramic pictures of fruits and vegetables to obtain features from different perspectives. At the same time, adjust the distance between the camera and the fruits and vegetables to obtain panoramic pictures of different scales. The local scene fruit and vegetable picture park captures details, focuses on specific parts of fruits and vegetables, and takes high-resolution local pictures. Or take multiple angles of the same local part to obtain more comprehensive feature information. Pictures of fruits and vegetables in different placement methods refer to stacking multiple fruits and vegetables together to simulate the actual storage or display situation and taking pictures, or scattering fruits and vegetables on the table or on the ground to capture the distribution and appearance of fruits and vegetables in their natural state. Fruits and vegetables can also be hung for photography, such as hanging them with ropes or hooks, to obtain different perspectives and characteristics.

[0146] Subsequently, the fruit and vegetable pictures with different light intensities, panoramic scenes, local scenes and different placement methods are used to construct a light intensity sample set, a panoramic scene sample set, a local scene sample set and a placement method sample set, respectively. In other words, the pictures are classified and organized into different sample sets according to the specific features or scenes shown in the pictures.

[0147] Image contrast adjustment is performed on each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set and the placement method sample set to obtain illumination intensity sample sets, the panoramic scene sample set, the local scene sample set and the placement method sample set under different contrasts. That is, contrast enhancement or reduction techniques are applied to the images in each set to generate a series of image sets with different contrast levels, which can enhance the details in the image, improve the image quality and provide more data diversity for subsequent image analysis, processing and recognition tasks.

[0148] The sample sets of light intensity under different contrasts and after transformation, the sample sets of panoramic scenes, the sample sets of local scenes, and the sample sets of placement methods are used to construct the feature training set of fruits and vegetables. Specifically, a feature extraction algorithm or a deep learning model is used to extract useful features from the preprocessed images. For the deep learning model, a convolutional neural network can be used. The extracted features are combined with the corresponding annotation information to construct a training set. Each training sample usually includes a feature vector and one or more labels. Subsequently, the training set is divided into a training subset, a validation subset, and a test subset. The training subset is used to train the model, the validation subset is used to adjust the model parameters and select the best model, and the test subset is used to evaluate the final performance of the model. The training set can be divided according to the actual situation to obtain the feature training set of fruits and vegetables.

[0149] By constructing a training set of fruit and vegetable features, the generalization ability of the model can be improved, which helps the model learn the feature representation of fruits and vegetables under different conditions, thereby improving the generalization ability of the model on unseen data. It can also improve the recognition performance of the model and improve the accuracy of the model in identifying fruits and vegetables.

[0150] In one implementation of this embodiment, the fruit and vegetable type prediction model includes a low-level feature extraction layer and a high-level feature extraction layer. The fruit and vegetable image is input into a preset fruit and vegetable type prediction model to determine the fruit and vegetable type of each fruit and vegetable in the fruit and vegetable placement area, including:

[0151] S810, extracting a feature map of basic features of the fruit and vegetable image through a low-level feature extraction layer, wherein the basic features include image edge features, image color features, and image texture features;

[0152] S820, extracting a feature map of semantic information of the fruit and vegetable image through a high-level feature extraction layer, wherein the semantic information includes object semantic information, attribute semantic information, and scene global semantic information in the fruit and vegetable image;

[0153] S830, transmitting the feature map of the basic feature to the high-level feature extraction layer using a preset feature channel;

[0154] S840, fusing the feature map of the basic feature and the feature map of the semantic information by horizontal connection to obtain the final feature of the fruit and vegetable image;

[0155] S850: Match the final feature with the feature corresponding to each type of fruit and vegetable in the third preset database to determine the type of each fruit and vegetable in the fruit and vegetable placement area.

[0156] The feature map of the basic features of the fruit and vegetable image is extracted through the low-level feature extraction layer, wherein the basic features include image edge features, image color features and image texture features. In this embodiment, the low-level feature extraction layer refers to the layer close to the input layer. The low-level feature extraction layer mainly extracts shallow features. The shallow features retain the original form of the input data, so that geometric information such as contours and edges can be clearly displayed. The low-level features are very sensitive to the local structure and details of the image. In other words, the low-level features such as color, shape, and texture are extracted through the low-level feature extraction layer. The color features can be extracted by color histograms, and the shape, texture and other features can be extracted by edge detection and other methods. After the low-level features such as color, shape and texture are extracted, these features can be represented in the form of feature maps. The feature map is a two-dimensional array or matrix, in which each element corresponds to a pixel or region in the image and contains the feature value of the pixel or region. The color feature map can be generated according to the color histogram or color moment. It maps each pixel or region in the image to the corresponding color feature value to obtain a color feature map. The shape feature map can be generated according to the results of edge detection or contour extraction. It represents the edge points or contours in the image in a specific way to obtain a shape feature map. The texture feature map can be generated based on the gray level co-occurrence matrix or LBP features. It maps each pixel point or area in the image to the corresponding texture feature value to obtain a texture feature map.

[0157] Subsequently, a feature map of the semantic information of the fruit and vegetable image is extracted through a high-level feature extraction layer, wherein the semantic information includes object semantic information, attribute semantic information, and scene global semantic information in the fruit and vegetable image. In this embodiment, the high-level feature extraction layer refers to a layer close to the output layer, and the high-level feature extraction layer mainly extracts high-level semantic information. In the high-level feature extraction layer, the model has learned the basic features of the image (such as edges, textures, etc.) through the low-level feature extraction layer, and further abstracts and integrates these features on this basis to extract higher-level semantic information. For example, in fruit and vegetable image recognition, the high-level feature extraction layer may be able to identify the specific types of fruits and vegetables in the image (such as apples, bananas, etc.), and judge their attributes such as maturity and color. In addition, if the image contains the environment or scene information of the fruits and vegetables, the high-level feature extraction layer may also be able to capture these global semantic information, thereby more comprehensively understanding the image content. Object semantic information mainly includes the types and individuals of fruits and vegetables in the image; attribute semantic information focuses on the color, size, maturity and other attributes of fruits and vegetables; scene global semantic information focuses on the environment and background of fruits and vegetables; after extracting object semantic information, attribute semantic information and scene global semantic information, the high-level feature extraction layer will represent this information in the form of feature maps.

[0158] Next, the feature map of the basic features is transmitted to the high-level feature extraction layer using the preset feature channel. That is to say, the feature map of the basic features extracted by the low-level network layer is transmitted to the high-level feature extraction layer through the predefined feature channel. The predefined feature channel can be determined according to the model design and task requirements.

[0159] Through lateral connection, the feature map of basic features and the feature map of semantic information are fused to obtain the final features of fruit and vegetable images. Horizontal connection is an operation in the feature pyramid network to fuse the upsampled high-level feature map with the low-level feature map. It is usually achieved by concatenating or adding the two feature maps element by element in the channel dimension. In other words, in order to build an effective network structure model, the low-level and high-level features need to be fused. This is achieved through skip connections, feature pyramid networks, or attention mechanisms. The fused feature map contains both geometric information and rich semantic information. Specifically, in the feature pyramid network, the low-level features are fused with the high-level features through upsampling to form a multi-scale feature map with rich semantic information. Upsampling is a basic operation in signal processing and image processing, which aims to increase the sampling rate of the signal or the resolution of the image. Upsampling can increase the sampling rate of the signal or the resolution of the image by filling the new pixel position by copying the value of the nearest pixel through nearest neighbor interpolation.

[0160] Subsequently, the final feature is matched with the feature corresponding to each type of fruit and vegetable in the third preset database to determine the type of fruit and vegetable of each fruit and vegetable in the fruit and vegetable placement area. Specifically, the feature vector is obtained after processing the fruit and vegetable image through the neural network model. This feature vector may contain its key feature information. In the third preset database, the final feature is compared with each fruit and vegetable feature in the database, and the similarity or distance between the two feature vectors can be calculated. The similarity can be calculated using methods such as Euclidean distance. According to the result of feature matching, the most likely type of fruit and vegetable can be determined for each fruit in the fruit and vegetable image, that is, the type of fruit and vegetable of each fruit and vegetable in the fruit and vegetable placement area is determined.

[0161] In another implementation of this embodiment, determining the fruit and vegetable characteristics of each fruit and vegetable type includes the following steps:

[0162] All fruit and vegetable images are segmented into target cells, and the first similarity between each target cell and the adjacent target cells is calculated. Specifically, first, the fruit and vegetable images need to be segmented into multiple target cells, and image segmentation techniques such as contour detection or segmentation based on coordinate points can be used. In each segmented target cell, the features of the fruit and vegetable objects are extracted, which may include color, texture, shape, etc. The similarity between each target cell and the adjacent target cells is calculated, and a similarity calculation method such as Euclidean distance, cosine similarity, etc. can be used to obtain the first similarity.

[0163] It is determined whether the first similarity is greater than a first preset similarity threshold. In this embodiment, the first preset similarity threshold can be determined according to actual conditions, that is, the calculated first similarity is compared with the first preset similarity threshold. If the first similarity is greater than the threshold, it indicates that the similarity between the two cells is high and they may belong to the same category of association.

[0164] If the similarity is greater than the first preset similarity threshold, each target cell is region-merged with the adjacent target cells to obtain a primary candidate region. In this embodiment, the primary candidate region is a primary candidate region for image recognition. That is, if the similarity is greater than the first preset similarity threshold, it means that the adjacent target cells have a high similarity and may belong to the same fruit and vegetable object or the same category. In this case, region merging can be performed to obtain a larger and more complete primary candidate region.

[0165] After obtaining the primary candidate regions, the second similarity between each primary candidate region and the adjacent primary candidate regions is calculated. Specifically, the features of each primary candidate region are extracted, which may be color, texture, shape or any other information that can characterize the regional characteristics. The similarity between each primary candidate region and its adjacent regions is calculated using cosine similarity. Finally, the calculated similarity result is output to obtain the second similarity.

[0166] If the second similarity is greater than the second preset similarity threshold, each primary candidate region is merged with an adjacent primary candidate region to obtain a final candidate region. In this embodiment, the second preset similarity threshold can be determined according to actual conditions. Specifically, when the second similarity between each primary candidate region and its adjacent region is greater than the second preset similarity threshold, the adjacent primary candidate regions with similarities greater than the threshold are merged. The merging method can be to simply merge the boundaries of the two regions to form a larger region to obtain a final candidate region. In this embodiment, the final candidate region is the final region of image recognition.

[0167] Each final candidate region is input into the pre-trained convolutional neural network model to obtain the regional features of each final candidate region. In other words, each candidate region is used as input and forward propagated through the pre-trained convolutional neural network model. During the forward propagation process, the candidate region will pass through multiple convolutional layers, pooling layers, and fully connected layers to gradually extract clearer features. In the pre-trained convolutional neural network model, features extracted through fully connected layers or convolutional layers can be high-dimensional vectors that contain the semantic information and spatial information of the candidate region, thereby obtaining the regional features of each final candidate region.

[0168] The support vector machine algorithm is used to classify the regional features of all the final candidate areas and determine the fruit and vegetable features of each fruit and vegetable type. Support vector machine is a supervised learning algorithm that is often used for classification and regression analysis. In other words, the trained support vector machine algorithm is used to classify the features of the final candidate areas to obtain the fruit and vegetable types to which each candidate area belongs. The classification results are analyzed to determine the typical features of each fruit and vegetable type. Based on the typical features of each fruit and vegetable type, the fruit and vegetable features of each fruit and vegetable type in the fruit and vegetable placement area are determined.

[0169] By combining multiple key steps such as low-level feature extraction, high-level semantic information extraction, feature fusion, and database matching, automatic fruit and vegetable classification and identification can be achieved, thereby improving the efficiency and accuracy of fruit and vegetable classification and identification.

[0170] The present application provides an electronic device, including:

[0171] a memory configured to store instructions; and

[0172] The processor is configured to call instructions from the memory and implement the above-mentioned machine vision-based intelligent settlement method for fruits and vegetables when executing the instructions.

[0173] The present application also provides a fruit and vegetable intelligent settlement system 10, such as Figure 2 As shown, including:

[0174] The above electronic equipment;

[0175] A weighing module 20 is connected to the electronic equipment;

[0176] An image acquisition module 30 connected to the electronic device;

[0177] The touch screen 40 is connected to the electronic device, the weighing module and the image acquisition module.

[0178] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0182] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0183] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0184] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0185] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0186] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for intelligent settlement of fruits and vegetables based on machine vision, characterized in that: Applied to a fruit and vegetable intelligent settlement system, the fruit and vegetable intelligent settlement system includes a touch screen, a weighing module and an image acquisition module, and the method includes: In response to a settlement instruction from a user through the touch screen, the image acquisition module acquires images of fruits and vegetables in the fruit and vegetable placement area; Obtaining the actual weight of all fruits and vegetables in the fruit and vegetable placement area through the weighing module; Inputting the fruit and vegetable image into a preset fruit and vegetable category prediction model to determine at least one fruit and vegetable category in the fruit and vegetable placement area, wherein each of the fruit and vegetable categories corresponds to at least one fruit and vegetable; Determine a first weight value for each type of the fruit and vegetable based on the fruit and vegetable image; Determine the density of fruits and vegetables corresponding to each of the fruit and vegetable types in the first preset database, and determine a second weight value for each of the fruit and vegetable types based on the density of fruits and vegetables; Determining a comprehensive weight value of each type of fruit and vegetable according to the first weight value and the second weight value; Allocate the settlement weight of each type of fruits and vegetables according to the comprehensive weight value and the actual weight of each type of fruits and vegetables; Determine the unit price amount corresponding to each type of fruit and vegetable in the second preset database; The sum of the products of the settlement weights of all the fruit and vegetable types and the corresponding unit prices is output as the settlement price.

2. The method according to claim 1, characterized in that: The determining a first weight value of each type of fruit and vegetable based on the fruit and vegetable image includes: For each type of fruit and vegetable, extract the contour area of ​​each fruit and vegetable through the fruit and vegetable image, and summarize the contour areas of all the fruits and vegetables to obtain the contour area of ​​the fruits and vegetables corresponding to each type of fruit and vegetable; Summarizing the contour areas of the fruits and vegetables corresponding to all the fruit and vegetable types to obtain a total contour area; Based on the fruit and vegetable contour area corresponding to each of the fruit and vegetable types and the sum of the contour areas, a first weight value of each of the fruit and vegetable types is determined.

3. The method according to claim 1, characterized in that The determining of the second weight value of each type of fruit and vegetable based on the fruit and vegetable density includes: Summarizing the density of the fruits and vegetables corresponding to all the types of fruits and vegetables to obtain a total density; Based on the fruit and vegetable density corresponding to each of the fruit and vegetable types and the total density, a second weight value of each of the fruit and vegetable types is determined.

4. The method according to claim 1, characterized in that Determining the comprehensive weight value of each type of fruit and vegetable according to the first weight value and the second weight value includes: Acquire fruit and vegetable area data, fruit and vegetable density data, and fruit and vegetable quality data corresponding to the fruit and vegetable types in the first preset database; Using a preset first correlation coefficient formula, the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data is calculated to obtain a first weight coefficient of the first weight value; Using a preset second correlation coefficient formula, the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data is calculated to obtain a second weight coefficient of the second weight value; Based on the first weight value, the first weight coefficient, the second weight value and the second weight coefficient, a comprehensive weight of each type of fruits and vegetables is calculated.

5. The method according to claim 1, characterized in that After acquiring the images of fruits and vegetables in the fruit and vegetable placement area through the image acquisition module, the method includes: Extracting edge features and color features of the fruit and vegetable images; When the color feature is a preset color and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, a reminder message is issued and displayed on the touch screen, wherein the reminder message is used to prompt the user to remove the packaging bag or box of the fruits and vegetables.

6. The method according to claim 1, characterized in that The steps of constructing the fruit and vegetable category prediction model include: Determine the image size in the fruit and vegetable feature training set; Determining the spatial dimension of the input layer of the fruit and vegetable variety prediction model by using the image size in the fruit and vegetable feature training set; Determining the number of color channels of the input layer of the fruit and vegetable variety prediction model according to the fruit and vegetable color features; Determine the number of neurons in the output layer of the fruit and vegetable variety prediction model through all the fruit and vegetable varieties; The fruit and vegetable category prediction model is trained by using the fruit and vegetable feature training set, and the fruit and vegetable category prediction model is constructed by combining the spatial dimension of the input layer, the number of color channels, and the number of neurons in the output layer.

7. The method according to claim 6, characterized in that The steps of constructing the fruit and vegetable feature training set include: Using the image acquisition module, collect pictures of fruits and vegetables with different light intensities, pictures of fruits and vegetables in panoramic scenes, pictures of fruits and vegetables in local scenes, and pictures of fruits and vegetables in different placement methods; The fruit and vegetable pictures with different light intensities, the panoramic scene fruit and vegetable pictures, the local scene fruit and vegetable pictures and the fruit and vegetable pictures with different placement methods are respectively used to construct a light intensity sample set, a panoramic scene sample set, a local scene sample set and a placement method sample set; Performing image space geometric transformation on each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement mode sample set to obtain a transformed illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement mode sample set; Performing image contrast adjustment on each sample image in the illumination intensity sample set, the panoramic scene sample set, the local scene sample set, and the placement method sample set to obtain illumination intensity sample sets, the panoramic scene sample set, the local scene sample set, and the placement method sample set at different contrasts; The illumination intensity sample sets under different contrasts and after the transformation, the panoramic scene sample sets, the local scene sample sets, and the placement method sample sets are used to construct a fruit and vegetable feature training set.

8. The method according to claim 6, characterized in that The fruit and vegetable type prediction model includes a low-level feature extraction layer and a high-level feature extraction layer. The fruit and vegetable image is input into a preset fruit and vegetable type prediction model to determine the fruit and vegetable type of each fruit and vegetable in the fruit and vegetable placement area, including: Extracting a feature map of basic features of the fruit and vegetable image through the low-level feature extraction layer, wherein the basic features include image edge features, image color features, and image texture features; Extracting a feature map of the semantic information of the fruit and vegetable image through the high-level feature extraction layer, wherein the semantic information includes object semantic information, attribute semantic information and scene global semantic information in the fruit and vegetable image; Transmitting the feature map of the basic feature to the high-level feature extraction layer using a preset feature channel; By horizontally connecting the feature map of the basic feature and the feature map of the semantic information, feature fusion is performed to obtain the final feature of the fruit and vegetable image; The final feature is matched with the feature corresponding to each type of fruit and vegetable in the third preset database to determine the type of each fruit and vegetable in the fruit and vegetable placement area.

9. An electronic device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instruction from the memory and implement the machine vision-based intelligent settlement method for fruits and vegetables according to any one of claims 1 to 8 when executing the instruction.

10. An intelligent settlement system for fruits and vegetables, characterized in that: include: The electronic device according to claim 9; A weighing module connected to the electronic device; An image acquisition module connected to the electronic device; A touch screen is connected to the electronic device, the weighing module and the image acquisition module.

Images