Method and system for intelligent settlement of fruits and vegetables based on machine vision

Abstract

The application discloses a fruit and vegetable intelligent settlement method and system based on machine vision, and relates to the technical field of machine vision. The method comprises the following steps: acquiring fruit and vegetable images in a fruit and vegetable placement area; acquiring actual weights of all fruits and vegetables; determining fruit and vegetable types in the fruit and vegetable placement area through a preset fruit and vegetable type prediction model; determining first weight values of each fruit and vegetable type based on the fruit and vegetable images; determining second weight values of each fruit and vegetable type based on fruit and vegetable densities; determining comprehensive weight values of each fruit and vegetable type according to the first weight values and the second weight values; distributing settlement weights of each fruit and vegetable type according to the comprehensive weight values and the actual weights; and taking a sum of products of the settlement weights of all fruit and vegetable types and corresponding unit prices as a settlement price and outputting the settlement price. The application can effectively solve the problem of low artificial settlement efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of machine vision, in particular to a fruit and vegetable intelligent settlement method and system based on machine vision. BACKGROUND

[0002] In fresh supermarket, farmers market and other retail places, the settlement process of fruits and vegetables usually involves weighing, identifying commodity types, calculating prices and other links, which requires special staff to weigh, settle and other operations, with high labor cost. And in peak period, there is a demand for customers to weigh multiple fruits and vegetables, but because the staff needs to weigh, pack and code each kind of fruit and vegetable separately, the weighing and settlement processing speed is particularly slow, which cannot meet the needs of such customers for quick settlement. At the same time, the staff may need to serve multiple customers and goods at the same time, resulting in low efficiency of manual settlement, thereby affecting the operation efficiency of the retail industry.

[0003] Therefore, there is an urgent need for a solution to solve the above problems. SUMMARY

[0004] Embodiments of the present application provide a fruit and vegetable intelligent settlement method and system based on machine vision, which is used to solve the problem of low efficiency of manual settlement.

[0005] To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a fruit and vegetable intelligent settlement method based on machine vision is provided, applied to a fruit and vegetable intelligent settlement system, the fruit and vegetable intelligent settlement system comprising a touch screen, a weighing module and an image acquisition module, the method comprising:

[0007] In response to a settlement instruction of a user through the touch screen, acquiring fruit and vegetable images in a fruit and vegetable placement area through the image acquisition module;

[0008] Acquiring actual weights of all fruits and vegetables in the fruit and vegetable placement area through the weighing module;

[0009] Inputting the fruit and vegetable images into a preset fruit and vegetable type prediction model to determine at least one fruit and vegetable type in the fruit and vegetable placement area, wherein each fruit and vegetable type corresponds to at least one fruit and vegetable;

[0010] Determining a first weight value of each fruit and vegetable type based on the fruit and vegetable images;

[0011] Determining a fruit and vegetable density corresponding to each fruit and vegetable type in a first preset database, and determining a second weight value of each fruit and vegetable type based on the fruit and vegetable density;

[0012] determine a comprehensive weight value of each of the fruit and vegetable categories according to the first weight value and the second weight value;

[0013] allocate a settlement weight of each of the fruit and vegetable categories according to the comprehensive weight value of each of the fruit and vegetable categories and the actual weight;

[0014] determine a unit price amount corresponding to each of the fruit and vegetable categories in a second preset database;

[0015] take a sum of products of the settlement weight of all the fruit and vegetable categories and the corresponding unit price amount as a settlement price and output the settlement price.

[0016] In a possible implementation form of the first aspect, the determining of the first weight value of each of the fruit and vegetable categories based on the fruit and vegetable image comprises:

[0017] for each of the fruit and vegetable categories, extracting a contour area of each fruit and vegetable from the fruit and vegetable image, and summing up the contour areas of all the fruit and vegetables to obtain a fruit and vegetable contour area corresponding to each of the fruit and vegetable categories;

[0018] summing up the fruit and vegetable contour areas corresponding to all the fruit and vegetable categories to obtain a contour area sum;

[0019] determining the first weight value of each of the fruit and vegetable categories based on the fruit and vegetable contour area corresponding to each of the fruit and vegetable categories and the contour area sum.

[0020] In a possible implementation form of the first aspect, the determining of the second weight value of each of the fruit and vegetable categories based on the fruit and vegetable density comprises:

[0021] summing up the fruit and vegetable densities corresponding to all the fruit and vegetable categories to obtain a density sum;

[0022] determining the second weight value of each of the fruit and vegetable categories based on the fruit and vegetable density corresponding to each of the fruit and vegetable categories and the density sum.

[0023] In a possible implementation form of the first aspect, the determining of the comprehensive weight value of each of the fruit and vegetable categories according to the first weight value and the second weight value comprises:

[0024] obtaining fruit and vegetable area data, fruit and vegetable density data and fruit and vegetable quality data corresponding to the fruit and vegetable categories in a first preset database;

[0025] calculating a correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data by using a preset first correlation coefficient formula to obtain a first weight coefficient of the first weight value;

[0026] The preset second correlation coefficient formula is used to calculate a correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data, to obtain a second weight coefficient of the second weight value;

[0027] Based on the first weight value, the first weight coefficient, the second weight value, and the second weight coefficient, a comprehensive weight of each fruit and vegetable category is calculated.

[0028] In a possible implementation of the first aspect, the method further includes:

[0029] The edge feature and the color feature of the fruit and vegetable image are extracted;

[0030] In a case where the color feature is a preset color and there is no fruit and vegetable edge feature matching the edge feature in a third preset database, a prompt information is sent and displayed on the touch screen, where the prompt information is used to prompt the user to remove the packaging bag or packaging box of the fruit and vegetable.

[0031] In a possible implementation of the first aspect, the constructing of the fruit and vegetable category prediction model includes:

[0032] The picture size in the fruit and vegetable feature training set is determined;

[0033] The spatial dimension of the input layer of the fruit and vegetable category prediction model is determined according to the picture size in the fruit and vegetable feature training set;

[0034] The color channel number of the input layer of the fruit and vegetable category prediction model is determined according to the fruit and vegetable color feature;

[0035] The number of neurons of the output layer of the fruit and vegetable category prediction model is determined according to all the fruit and vegetable categories;

[0036] The fruit and vegetable category prediction model is trained according to the fruit and vegetable feature training set, and the fruit and vegetable category prediction model is constructed according to the spatial dimension of the input layer, the color channel number, and the number of neurons of the output layer.

[0037] In a possible implementation of the first aspect, the constructing of the fruit and vegetable feature training set includes:

[0038] Different light intensity fruit and vegetable pictures, panoramic scene fruit and vegetable pictures, local scene fruit and vegetable pictures, and different placement mode fruit and vegetable pictures are collected by using the image collection module;

[0039] The different light intensity fruit and vegetable pictures, the panoramic scene fruit and vegetable pictures, the local scene fruit and vegetable pictures and the different placement mode fruit and vegetable pictures are respectively constructed into a light intensity sample set, a panoramic scene sample set, a local scene sample set and a placement mode sample set;

[0040] Each sample image in the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set is subjected to image space geometric transformation, to obtain a transformed light intensity sample set, a transformed panoramic scene sample set, a transformed local scene sample set and a transformed placement mode sample set;

[0041] Each sample image in the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set is subjected to image contrast adjustment, to obtain a light intensity sample set, a panoramic scene sample set, a local scene sample set and a placement mode sample set under different contrasts;

[0042] The light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set under different contrasts and the transformed light intensity sample set, the transformed panoramic scene sample set, the transformed local scene sample set and the transformed placement mode sample set are constructed into a fruit and vegetable feature training set.

[0043] In a possible implementation manner of the first aspect, the fruit and vegetable category prediction model comprises a low-level feature extraction layer and a high-level feature extraction layer, and the inputting the fruit and vegetable image into the preset fruit and vegetable category prediction model to determine the fruit and vegetable category of each fruit and vegetable in the fruit and vegetable placement region comprises:

[0044] The low-level feature extraction layer is used to extract a feature map of basic features of the fruit and vegetable image, wherein the basic features comprise image edge features, image color features and image texture features;

[0045] The high-level feature extraction layer is used to extract a feature map of semantic information of the fruit and vegetable image, wherein the semantic information comprises object semantic information, attribute semantic information and scene global semantic information in the fruit and vegetable image;

[0046] The feature map of the basic features is transmitted to the high-level feature extraction layer through a preset feature channel;

[0047] The feature map of the basic features and the feature map of the semantic information are fused through a transverse connection mode, to obtain a final feature of the fruit and vegetable image;

[0048] The final feature is matched with features corresponding to each fruit and vegetable category in a third preset database, to determine the fruit and vegetable category of each fruit and vegetable in the fruit and vegetable placement region.

[0049] In a second aspect, the present application provides an electronic device, comprising:

[0050] a memory configured to store instructions; and

[0051] a processor configured to call the instructions from the memory and enable the machine vision-based fruit and vegetable intelligent settlement method described above when the instructions are executed.

[0052] In a third aspect, the present application provides a fruit and vegetable intelligent settlement system, comprising:

[0053] the electronic device described above;

[0054] a weighing module connected to the electronic device;

[0055] an image acquisition module connected to the electronic device;

[0056] a touch screen connected to the electronic device, the weighing module, and the image acquisition module.

[0057] Through the above technical solution, at least one fruit and vegetable species in the fruit and vegetable placement area is determined through a preset fruit and vegetable species prediction model, and the fruit and vegetable species in the fruit and vegetable placement area is identified with the aid of advanced machine vision technology and deep learning algorithm. The species of multiple fruits and vegetables can be determined, human intervention is reduced, and the fruit and vegetable settlement efficiency is significantly improved. The first weight value of each fruit and vegetable species is determined through a fruit and vegetable image, and the second weight value of each fruit and vegetable species is determined through a fruit and vegetable density. According to the first weight value and the second weight value, the first weight value coefficient and the second weight value coefficient are determined, the comprehensive weight value of each fruit and vegetable species is calculated, the settlement weight of each fruit and vegetable is reasonably allocated through the comprehensive weight value, and the settlement accuracy is further improved. Considering the dual factors of image recognition and actual weight, this double verification mechanism reduces human error and improves the accuracy of settlement. The fruit and vegetable intelligent settlement system based on machine vision can not only settle multiple types of fruits and vegetables at a time, but also improve the settlement efficiency and accuracy, enhance the user experience, optimize the inventory management, and solve the problem of low efficiency of manual settlement.

[0058] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific embodiments part. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 A flowchart of a machine vision-based fruit and vegetable intelligent settlement method provided by the embodiments of the present application;

[0060] Figure 2 A structural diagram of a fruit and vegetable intelligent settlement system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the specific implementation described herein is only used to explain and illustrate the embodiments of the present application, and is not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0062] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0063] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0064] Figure 1 A flowchart of a fruit and vegetable intelligent settlement method based on machine vision according to an embodiment of the present application is schematically shown. As shown in Figure 1 The present application provides a fruit and vegetable intelligent settlement method based on machine vision, which can include the following steps.

[0065] S110, in response to a settlement instruction of a user through a touch screen, acquiring a fruit and vegetable image in a fruit and vegetable placement area through an image acquisition module;

[0066] S120, acquiring actual weights of all fruits and vegetables in the fruit and vegetable placement area through a weighing module;

[0067] S130, inputting the fruit and vegetable image into a preset fruit and vegetable type prediction model to determine at least one fruit and vegetable type in the fruit and vegetable placement area, wherein each fruit and vegetable type corresponds to at least one fruit and vegetable;

[0068] S140, determining a first weight value of each fruit and vegetable type based on the fruit and vegetable image;

[0069] S150, determining a fruit and vegetable density corresponding to each fruit and vegetable category in the first preset database, and determining a second weight value of each fruit and vegetable category based on the fruit and vegetable density;

[0070] S160, determining a comprehensive weight value of each fruit and vegetable category according to the first weight value and the second weight value;

[0071] S170, distributing a settlement weight of each fruit and vegetable category according to the comprehensive weight value of each fruit and vegetable category and the actual weight;

[0072] S180, determining a unit price amount corresponding to each fruit and vegetable category in the second preset database;

[0073] S190, taking the sum of the products of the settlement weights of all fruit and vegetable categories and the corresponding unit price amounts as a settlement price and outputting.

[0074] In response to the user's settlement instruction through the touch screen, the fruit and vegetable image in the fruit and vegetable placement area is acquired through the image acquisition module, that is, the user issues a settlement instruction through the touch screen, and the system first identifies this instruction and prepares for subsequent image acquisition work. Subsequently, in response to the user's settlement instruction, the image acquisition module is activated, which is a high-resolution camera in this embodiment and can capture high-quality images. The image acquisition module is aimed at the fruit and vegetable placement area to capture the fruit and vegetable image in the area, and the acquired fruit and vegetable image will be further processed and recognized.

[0075] Next, the actual weight of all fruits and vegetables in the fruit and vegetable placement area is acquired through the weighing module, that is, after the user issues a settlement instruction through the touch screen and the fruit and vegetable image is acquired by the image acquisition module, the actual weight of all fruits and vegetables in the fruit and vegetable placement area is measured by the weighing module. In this embodiment, the weighing module is usually a weight sensor, which is a device that converts a mass signal into a measurable electrical signal output. When the image acquisition module has completed the shooting of the fruit and vegetable image, the weighing module will automatically start and enter the weighing state, and through the built-in sensor system, the total weight of all fruits and vegetables in the fruit and vegetable placement area is measured.

[0076] Subsequently, the fruit and vegetable image is input into a preset fruit and vegetable category prediction model to determine at least one fruit and vegetable category in the fruit and vegetable placement area, and in this embodiment, each fruit and vegetable category corresponds to at least one fruit and vegetable. The preset fruit and vegetable category prediction model is an image classification model based on a convolutional neural network model. Specifically, first, the image of the fruit and vegetable placement area is input into the preset fruit and vegetable category prediction model, and the preset fruit and vegetable category prediction model uses a convolutional layer to extract features of the input image, that is, fruit and vegetable features in the image. Subsequently, the extracted features are mapped to specific fruit and vegetable categories through a fully connected layer, and the fruit and vegetable categories are output as recognition results.

[0077] After determining the fruit and vegetable species in the fruit and vegetable placement area, a first weight value of each fruit and vegetable species is determined based on the fruit and vegetable image. In this embodiment, the first weight value is used to evaluate the degree of influence of the area on the allocated weight. Specifically, first, the contour area of each fruit and vegetable in the fruit and vegetable placement area is extracted through image recognition technology, and all contour areas are added to obtain the fruit and vegetable contour area corresponding to each fruit and vegetable species. In this embodiment, the image recognition technology is a deep learning-based image recognition technology. Image recognition technology is to process, analyze and understand images by computer to identify various different patterns of targets and objects. Subsequently, the ratio between the contour area corresponding to each fruit and vegetable species and all fruit and vegetable contour area values is taken as the first weight value of each fruit and vegetable species.

[0078] After determining the first weight value, the fruit and vegetable density corresponding to each fruit and vegetable species is determined in the first preset database, and the second weight value of each fruit and vegetable species is determined based on the fruit and vegetable density. In this embodiment, the second weight value refers to the degree of influence of the fruit and vegetable density on the allocated weight. Specifically, first, the fruit and vegetable density corresponding to each fruit and vegetable species is found in the first preset database. Fruit and vegetable density refers to the mass or weight of fruit and vegetables per unit volume, which reflects the physical properties of fruit and vegetables, such as tightness, void ratio, etc. Next, based on the obtained fruit and vegetable density value, the second weight value of each fruit and vegetable species is determined, that is, according to the size of the density value, a corresponding weight value is allocated to each fruit and vegetable, for example, the fruit and vegetable with higher density may be given a higher weight value, and vice versa.

[0079] According to the first weight value and the second weight value, the comprehensive weight value of each fruit and vegetable species is determined. Specifically, first, the first weight value and the second weight value are standardized to ensure that they are compared on the same dimension. Standardization can be achieved by Z-score standardization, which is a process of converting original data into deviation from mean in standard deviation units. Subsequently, the first weight coefficient and the second weight coefficient are determined for the first weight value and the second weight value, respectively. The correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data can be calculated, and this correlation coefficient is taken as the first weight coefficient; the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data is calculated, and this correlation coefficient is taken as the second weight coefficient. Next, the first weight value and the second weight value are weighted and summed using the determined weight coefficients to obtain the comprehensive weight value of each fruit and vegetable species.

[0080] According to the comprehensive weight value of each fruit and vegetable species and the actual weight, the settlement weight of each fruit and vegetable species is allocated, that is, the comprehensive weight value of each fruit and vegetable species is multiplied by its actual weight to calculate a settlement weight, thereby achieving the allocation of the settlement weight of each fruit and vegetable species.

[0081] In the second preset database, the unit price amount corresponding to each fruit and vegetable category is determined. Specifically, the second preset database storing the fruit and vegetable categories and their unit price information needs to be accessed. By executing the operation of querying the fruit and vegetable category, the information of all fruit and vegetable categories can be retrieved. For the retrieved information of each fruit and vegetable category, the corresponding unit price information needs to be further searched in all the information, so as to obtain the unit price amount corresponding to each fruit and vegetable category.

[0082] Subsequently, the sum of the products of the settlement weight of all fruit and vegetable categories and the corresponding unit price amount is taken as the settlement price and output. That is, after the settlement weight and the corresponding unit price amount of each fruit and vegetable category have been obtained, the settlement weight and the corresponding unit price amount of each fruit and vegetable category are multiplied to obtain the settlement price corresponding to each fruit and vegetable. The settlement amounts of all fruit and vegetable categories are added to obtain the total settlement price.

[0083] By means of the preset fruit and vegetable category prediction model, at least one fruit and vegetable category in the fruit and vegetable placement area is determined. By means of advanced machine vision technology and deep learning algorithm, the fruit and vegetable category of the fruit and vegetable placement area can be identified, and the categories of multiple fruits and vegetables can be determined, reducing manual intervention and significantly improving the fruit and vegetable settlement efficiency. The first weight value of each fruit and vegetable category is determined by means of the fruit and vegetable image, and the second weight value of each fruit and vegetable category is determined by means of the fruit and vegetable density. According to the first weight value and the second weight value, the first weight value coefficient and the second weight value coefficient are determined, the comprehensive weight value of each fruit and vegetable category is calculated, and the settlement weight of each fruit and vegetable is reasonably allocated by means of the comprehensive weight value, further improving the accuracy of settlement. Considering the dual factors of image recognition and actual weight, this dual verification mechanism reduces human error and improves the accuracy of settlement. The fruit and vegetable intelligent settlement system based on machine vision can not only settle multiple types of fruits and vegetables at one time, but also improve the settlement efficiency and accuracy, enhance the user experience, optimize the inventory management, and solve the problem of low efficiency of manual settlement.

[0084] In one of the embodiments of the present embodiment, the first weight value of each fruit and vegetable category is determined based on the fruit and vegetable image, which includes:

[0085] S210, for each fruit and vegetable category, the contour area of each fruit and vegetable is extracted from the fruit and vegetable image, and the contour areas of all fruits and vegetables are summarized to obtain the fruit and vegetable contour area corresponding to each fruit and vegetable category;

[0086] S220, the fruit and vegetable contour areas corresponding to all fruit and vegetable categories are summarized to obtain the total contour area;

[0087] S230, based on the fruit and vegetable contour area corresponding to each fruit and vegetable category and the total contour area, the first weight value of each fruit and vegetable category is determined.

[0088] The first weight value of each fruit and vegetable category is determined based on the fruit and vegetable image. First, for each fruit and vegetable category, the contour area of each fruit and vegetable is extracted from the fruit and vegetable image, and the contour areas of all fruits and vegetables are summarized to obtain the fruit and vegetable contour area corresponding to each fruit and vegetable category. Specifically, a high-definition image of the fruits and vegetables in the fruit and vegetable placement area is captured by a camera, and the high-definition image of the fruits and vegetables is preprocessed. Subsequently, an edge detection algorithm is used to detect the edges in the preprocessed image, which can be achieved by a Canny edge detection algorithm. The Canny edge detection algorithm is a multi-stage edge detection algorithm. Based on edge detection, a contour extraction algorithm can also be used to extract the contour of each fruit and vegetable, such as the findContours function in OpenCV, which is an open-source computer vision and image processing library. In OpenCV, the findContours function is used to detect contours in an image and can extract contours from an image. After obtaining the contour of each fruit and vegetable, the fruit and vegetable contour areas corresponding to all fruit and vegetable categories are summarized to obtain the total contour area, that is, the total contour area of the fruit and vegetable category is obtained by adding the contour areas of all fruits and vegetables.

[0089] Secondly, based on the fruit and vegetable contour area corresponding to each fruit and vegetable category and the total contour area, the first weight value of each fruit and vegetable category is determined, that is, the proportion of the contour area of each fruit and vegetable relative to the total contour area is calculated as the first weight value. The calculation formula is:

[0090] The first weight value (i) = the contour area of the ith fruit and vegetable / the sum of the contour areas of all fruits and vegetables

[0091] The first weight value of each fruit and vegetable category is determined by the calculation formula.

[0092] In another embodiment of the present embodiment, when the contours of the fruits and vegetables in the fruit and vegetable image overlap and multiple angle cameras are provided in the fruit and vegetable intelligent settlement system, first, the fruit and vegetable images under multiple angles are obtained by the cameras, that is, the images of the fruit and vegetable placement area are captured by the cameras under multiple angles, which can be captured from the front, side, top and bottom, etc. to obtain the fruit and vegetable images under multiple angles. In this embodiment, the cameras are multiple and are arranged at different positions to capture the fruit and vegetable images under multiple angles.

[0093] The multi-angle image combination features are determined through the fruit and vegetable images under multiple angles. Specifically, first, the two-dimensional fruit and vegetable contour image features of the fruit and vegetable images under multiple angles are extracted by using the convolution layer of the preset convolutional neural network. That is, the fruit and vegetable images under multiple angles are input into the preset convolutional neural network, and the two-dimensional fruit and vegetable contour image features of the fruit and vegetable images under multiple angles are extracted through the convolution layer. The convolution layer is a basic component in the convolutional neural network and can automatically learn and extract local features in an image, such as edges, textures, and shapes. Through convolution operation, the spatial structure information in the image can be captured, so that the two-dimensional fruit and vegetable contour image features of the fruit and vegetable images are obtained.

[0094] Secondly, the multiple two-dimensional fruit and vegetable contour image features are input into the preset aggregation network model to obtain the multi-angle image combination features. Specifically, the two-dimensional fruit and vegetable contour image features extracted from the fruit and vegetable contour images under multiple angles are input into the preset aggregation network model. The preset aggregation network model can be determined according to specific application scenarios and requirements. It can be a simple fully connected network for mapping the spliced or fused features to a new feature space, or a more complex network structure such as a deep neural network, a convolutional neural network, or a recurrent neural network to capture higher-level features. Next, the features extracted from different angles are aggregated together by the preset aggregation network model to form a unified multi-angle image combination feature. For example, the feature vectors of different angles are spliced together to form a longer feature vector. Weighted averaging, maximum pooling, minimum pooling, and other fusion strategies can also be used to fuse the features of different angles.

[0095] The multi-angle image combination features are constructed into a sample feature database, and an initial convolutional neural network is trained to obtain a trained convolutional neural network. Specifically, a large number of fruit and vegetable images under multiple angles are obtained. For each image, the two-dimensional fruit and vegetable contour image features are extracted by using the preset convolutional neural network, and the features extracted from different angles are combined to form multi-angle image combination features. The multi-angle combination features of all images are sorted into a sample feature database, and the initial convolutional neural network is trained through the sample feature database to obtain the trained convolutional neural network.

[0096] Subsequently, the current fruit and vegetable placement area is photographed from multiple angles by the camera to obtain current fruit and vegetable images, and the multiple current fruit and vegetable images are input into the trained convolutional neural network to obtain multiple fruit and vegetable contour features at different angles. Specifically, the camera is used to take multiple-angle photographs of the fruit and vegetables placed in the specific area to obtain the current fruit and vegetable images. Next, these multi-angle fruit and vegetable images are input into the trained convolutional neural network. In the convolutional neural network, the images are processed through multiple convolutional layers, pooling layers, and fully connected layers. Each convolutional layer uses a series of convolutional kernels to scan the image and extract local features such as edges, textures, and shapes from the image. Since multiple-angle fruit and vegetable images are input, the convolutional neural network processes each angle of the image and extracts features at each angle to obtain multiple fruit and vegetable contour features at different angles.

[0097] The multiple fruit and vegetable contour features are input into a preset point cloud data prediction model to obtain a three-dimensional point cloud prediction model of the fruit and vegetable image. In this embodiment, the three-dimensional point cloud prediction model can be a model based on deep learning. The point cloud data prediction model is a model that can predict corresponding three-dimensional point cloud data based on input two-dimensional image features. The multiple fruit and vegetable contour features extracted earlier are input into the preset point cloud data prediction model, and the model predicts the three-dimensional point cloud data of the fruit and vegetables based on these features.

[0098] The three-dimensional point cloud prediction model determines the contour of each fruit and vegetable. That is, after processing by the prediction model, the three-dimensional point cloud prediction model of the fruit and vegetable image is obtained. The model may contain information such as the position, shape, and size of the fruit and vegetables in three-dimensional space, thereby determining the contour of each fruit and vegetable.

[0099] By processing the fruit and vegetable image, the contour area of the fruit and vegetable is extracted and the first weight value of each fruit and vegetable species is determined. This can accurately measure the contour area of the fruit and vegetable and better determine the degree of influence of the contour area of the fruit and vegetable on weight distribution, thereby ensuring the accuracy of fruit and vegetable weight distribution.

[0100] In one embodiment of the present embodiment, the second weight value of each fruit and vegetable species is determined based on the fruit and vegetable density, comprising:

[0101] S310, aggregating the fruit and vegetable densities corresponding to all fruit and vegetable species to obtain a density sum;

[0102] S320, determining the second weight value of each fruit and vegetable species based on the fruit and vegetable density corresponding to each fruit and vegetable species and the density sum.

[0103] The second weight value of each fruit and vegetable type is determined based on the fruit and vegetable density. First, the fruit and vegetable densities corresponding to all fruit and vegetable types are summed up to obtain a total density, that is, the fruit and vegetable densities corresponding to each fruit and vegetable type in the first preset database are determined, and the densities of all fruit and vegetable types are added to obtain the total density. For example, there are apples and bananas in the fruit and vegetable placement area, the density of apples is about 0.95 g / cm3, the density of bananas is about 0.9 g / cm3, and the total density is 1.85 g / cm3.

[0104] Secondly, the second weight value of each fruit and vegetable type is determined based on the fruit and vegetable density corresponding to each fruit and vegetable type and the total density. That is, the total density obtained by adding the density values corresponding to all fruit and vegetable types in the fruit and vegetable placement area is divided by the fruit and vegetable density corresponding to each fruit and vegetable type to determine the second weight value of each fruit and vegetable type. For example, there are apples and bananas in the fruit and vegetable placement area, the density of apples is about 0.95 g / cm3, the density of bananas is about 0.9 g / cm3, and the total density is 1.85 g / cm3. The second weight value of apples is obtained by the ratio of 0.95 g / cm3 to the total density of 1.85 g / cm3; the second weight value of bananas is obtained by the ratio of 0.9 g / cm3 to 1.85 g / cm3;

[0105] By determining the second weight value of each fruit and vegetable type, the influence of the density of the fruit and vegetable on the weight distribution can be better determined, and the accuracy of the weight distribution of the fruit and vegetable can be ensured.

[0106] In one embodiment of the present embodiment, the comprehensive weight value of each fruit and vegetable type is determined according to the first weight value and the second weight value, including:

[0107] S410, obtaining fruit and vegetable area data, fruit and vegetable density data and fruit and vegetable quality data corresponding to the fruit and vegetable type in the first preset database;

[0108] S420, calculating the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data by using a preset first correlation coefficient formula to obtain a first weight coefficient of the first weight value;

[0109] S430, calculating the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data by using a preset second correlation coefficient formula to obtain a second weight coefficient of the second weight value;

[0110] S440, calculating the comprehensive weight of each fruit and vegetable type based on the first weight value, the first weight coefficient, the second weight value and the second weight coefficient.

[0111] The fruit and vegetable area data, the fruit and vegetable density data, and the fruit and vegetable quality data corresponding to the fruit and vegetable categories are obtained in the first preset database, that is, the fruit and vegetable area data, the fruit and vegetable density data, and the fruit and vegetable quality data corresponding to different fruit and vegetable categories are determined in the first preset database.

[0112] After the fruit and vegetable area data, the fruit and vegetable density data, and the fruit and vegetable quality data are determined, a preset first correlation coefficient formula is used to calculate the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data, to obtain a first weight coefficient of a first weight value. Specifically, first, the fruit and vegetable area data and the fruit and vegetable quality data are extracted from the first preset database, wherein each fruit and vegetable category has a corresponding area value and a quality value. The fruit and vegetable area data and the fruit and vegetable quality data are used to calculate the correlation coefficient by using a preset correlation coefficient formula. In this embodiment, the preset first correlation coefficient formula can be a Pearson correlation coefficient. The formula of the Pearson correlation coefficient is as follows:

[0113]

[0114] wherein x i and y i are the ith observation values of the fruit and vegetable area data and the fruit and vegetable quality data respectively, and are the average values of the area and the quality respectively, n is the number of observation values, and r represents the Pearson correlation coefficient.

[0115] The extracted fruit and vegetable area data and the fruit and vegetable quality data are substituted into the correlation coefficient formula to calculate the correlation coefficient r between the fruit and vegetable area data and the fruit and vegetable quality data. This correlation coefficient is taken as the first weight coefficient of the first weight value.

[0116] Next, a preset second correlation coefficient formula is used to calculate the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data, to obtain a second weight coefficient of a second weight value. The fruit and vegetable density data and the fruit and vegetable quality data are extracted from the first preset database, wherein each fruit and vegetable category has a corresponding density value and a quality value. Similarly, the fruit and vegetable density data and the fruit and vegetable quality data are used to calculate the correlation coefficient by using a preset correlation coefficient formula. In this embodiment, the preset second correlation coefficient formula can be a Pearson correlation coefficient. The Pearson correlation coefficient is a statistical quantity for measuring the degree of linear correlation between two variables. The extracted fruit and vegetable area data and the fruit and vegetable quality data are substituted into the correlation coefficient formula to calculate the correlation coefficient r between the fruit and vegetable density data and the fruit and vegetable quality data. This correlation coefficient is taken as the second weight coefficient of the second weight value.

[0117] Based on the first weight value, the first weight coefficient, the second weight value and the second weight coefficient, the comprehensive weight of each fruit and vegetable category is calculated. In order to comprehensively consider the two factors of area and density, the weight distribution coefficient needs to be determined. For example, if the area has a greater impact on the comprehensive weight, the value of the first weight coefficient will be larger. The comprehensive weight of each fruit and vegetable is calculated by using the area weight base value and the density weight base value, and their weight distribution coefficients. The calculation formula is:

[0118] Comprehensive weight (i) = area weight base value (i) x area weight coefficient + density weight base value (i) x density weight coefficient

[0119] Through the comprehensive weight, the comprehensive weight of each fruit and vegetable category is obtained.

[0120] By calculating the comprehensive weight value based on the area, density and quality data of the fruit and vegetable, the overall condition of each fruit and vegetable category can be more comprehensively reflected. By using the correlation coefficient formula to calculate the weight coefficient, the distribution of the weight can be more scientific and objective, and the scientificity and accuracy of the decision-making can be improved.

[0121] In one of the embodiments of the present embodiment, after obtaining the fruit and vegetable image in the fruit and vegetable placement area through the image acquisition module, the following steps are included:

[0122] S510, extracting the edge feature and color feature of the fruit and vegetable image;

[0123] S520, in the case that the color feature is a preset color and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, issuing a prompt information and displaying it on the touch screen, wherein the prompt information is used to prompt the user to remove the packaging bag or packaging box of the fruit and vegetable.

[0124] After obtaining the fruit and vegetable image in the fruit and vegetable placement area through the image acquisition module, the edge feature and color feature of the fruit and vegetable image are extracted, that is, the edge information and color information of the image are identified and extracted from the fruit and vegetable image. The edge feature of the fruit and vegetable image can be extracted by Sobel operator, which is a discrete differential operator for edge detection. The edge is detected by calculating the gradient of the image gray value. The color feature can be extracted by color histogram, color moment, etc. The color histogram counts the number of pixels in each color channel of the image, and the color moment describes the color feature using the first few moments of color distribution (such as mean, variance and skewness).

[0125] Subsequently, in the case that the color feature is the preset color and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, a prompt information is sent and displayed on the touch screen, in the embodiment, the prompt information is used to prompt the user to remove the packaging bag or packaging box of the fruit and vegetable. Specifically, first, the fruit and vegetable intelligent settlement system extracts the color feature from the image and compares it with the preset color feature, in the embodiment, the preset color feature can be white, transparent or the color of the packaging bag and packaging box in the sales place. And the fruit and vegetable intelligent settlement system extracts the edge feature from the image and matches it with the edge feature in the third preset database. If the color feature in the image matches the preset color feature successfully, and there is no fruit and vegetable edge feature matching the edge feature in the third preset database, that is, there is no fruit and vegetable edge feature in the current image, indicating that the packaging bag or packaging box blocks the fruit and vegetable to be settled, and cannot be recognized and settled, at this time, a prompt information is sent and displayed on the touch screen, prompting the user to remove the packaging bag or packaging box of the fruit and vegetable, so that the identification and settlement of the fruit and vegetable species can be carried out smoothly.

[0126] By identifying the edge feature and the color feature, it is determined whether there is a packaging bag or a packaging box in the current fruit and vegetable placement area, and the user is prompted to remove the packaging bag or the packaging box of the fruit and vegetable. These interference factors can be removed through user intervention, which can significantly improve the accuracy of identification and improve the practicality and generalization ability of the system.

[0127] In one of the embodiments of the embodiment, the construction step of the fruit and vegetable species prediction model comprises:

[0128] S610, determine the picture size in the fruit and vegetable feature training set;

[0129] S620, determine the spatial dimension of the input layer of the fruit and vegetable species prediction model through the picture size in the fruit and vegetable feature training set;

[0130] S630, determine the color channel number of the input layer of the fruit and vegetable species prediction model through the fruit and vegetable color feature;

[0131] S640, determine the number of neurons of the output layer of the fruit and vegetable species prediction model through all fruit and vegetable species;

[0132] S650, train the fruit and vegetable species prediction model through the fruit and vegetable feature training set, and combine the spatial dimension and color channel number of the input layer and the number of neurons of the output layer to construct the fruit and vegetable species prediction model.

[0133] Firstly, the picture size in the fruit and vegetable feature training set is determined. In the embodiment, the fruit and vegetable feature training set can be determined according to the actual situation, that is, when the fruit and vegetable feature training set is constructed, the pictures need to be unified to a specific width-height ratio, so as to facilitate subsequent feature extraction and model training. For example, the fruit and vegetable feature training set can be unified to a width-height ratio, which helps to ensure the consistency of the pictures in the training set. Therefore, the picture size can be directly obtained in the preset fruit and vegetable feature training set.

[0134] Subsequently, the spatial dimension of the input layer of the fruit and vegetable category prediction model is determined by the picture size in the fruit and vegetable feature training set, that is, in deep learning, the convolutional neural network model is one of the most commonly used models for image processing, and the input layer of the convolutional neural network model usually expects the image to have a specific spatial size. For example, if all images in the image data set are preprocessed to 224x224 pixels, the model input layer should also be set to accept 224x224 pixel images. Since the convolutional layer, pooling layer and the like in the convolutional neural network model need to determine the spatial size of the input data to calculate the moving step of the convolution kernel and the size of the pooling window, the spatial dimension of the input layer of the fruit and vegetable category prediction model needs to be determined by the picture size in the fruit and vegetable feature training set. The size of the model input layer is adjusted according to the size of the picture data. For example, if the picture is preprocessed to 224x224 pixels, the size of the input layer should be set to 224x224x3.

[0135] After determining the spatial dimension of the input layer of the fruit and vegetable category prediction model, the number of color channels of the input layer of the fruit and vegetable category prediction model is determined by the fruit and vegetable color feature, that is, the input layer should have the same number of color channels as the image data. For example, for a color RGB image, there are three color channels: red, green and blue. Therefore, if the image is an RGB image, the model input layer should be set to receive three-channel data.

[0136] Next, the number of neurons of the output layer of the fruit and vegetable category prediction model is determined by all fruit and vegetable categories. Specifically, first, all fruit and vegetable categories in the sales place need to be determined, and can be adjusted according to the real-time sales of fruit and vegetable categories in the sales place. The output layer of the model is configured by all fruit and vegetable categories. In the implementation of neural network, the number of neurons of the output layer is set to the number of fruit and vegetable categories. Each neuron corresponds to a fruit and vegetable category, and the output value usually represents the probability that the input data belongs to the corresponding category.

[0137] The fruit and vegetable feature training set is used to train the fruit and vegetable category prediction model, and the spatial dimension and color channel number of the input layer and the number of neurons of the output layer are combined to construct the fruit and vegetable category prediction model. Specifically, first, the fruit and vegetable feature training set is collected and preprocessed to ensure that the images have uniform spatial dimensions and color channel numbers, and each image is assigned a correct label. Then, a deep learning model is designed, and the input layer can accept preprocessed image data, that is, the spatial dimension and color channel number of the input layer are consistent with the images in the fruit and vegetable feature training set. Subsequently, the output layer is set, and the number of neurons is equal to the number of fruit and vegetable categories, and the softmax activation function is used to output the prediction probability. The softmax function can convert a vector containing any real number into a vector containing positive numbers and a sum of 1, that is, a probability distribution. In this way, the output of each neuron can be interpreted as the probability that the input image belongs to the corresponding category. Finally, the model is trained using the training set data, and the model parameters are adjusted using the backpropagation algorithm to minimize the prediction error. The prediction value is calculated using the forward propagation algorithm, and the difference between the prediction value and the actual label is measured by the loss function (such as cross-entropy loss). In order to minimize this difference, that is, the prediction error, the backpropagation algorithm can be used to adjust the parameters of the model. Backpropagation algorithm is an important algorithm for training artificial neural networks, which adjusts the weights and bias parameters in the network to minimize a loss function that measures the difference between the network's output and the actual label.

[0138] By designing the input layer and output layer of the model and training the model, the model can efficiently and accurately learn for the fruit and vegetable category prediction task, and exhibit good prediction performance and generalization ability in practical applications.

[0139] In one embodiment of the present embodiment, the construction step of the fruit and vegetable feature training set comprises:

[0140] S710, using an image acquisition module to acquire different light intensity fruit and vegetable pictures, panoramic scene fruit and vegetable pictures, local scene fruit and vegetable pictures and different placement mode fruit and vegetable pictures;

[0141] S720, constructing light intensity sample set, panoramic scene sample set, local scene sample set and placement mode sample set from different light intensity fruit and vegetable pictures, panoramic scene fruit and vegetable pictures, local scene fruit and vegetable pictures and different placement mode fruit and vegetable pictures, respectively;

[0142] S730, performing image space geometric transformation on each sample image in the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set to obtain the transformed light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set;

[0143] S740, image contrast adjustment is performed on each sample image in the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set to obtain the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set under different contrasts.

[0144] S750, the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set under different contrasts and after transformation are constructed into a fruit and vegetable feature training set.

[0145] Firstly, the image acquisition module is used to acquire different light intensity fruit and vegetable pictures, panoramic scene fruit and vegetable pictures, local scene fruit and vegetable pictures and different placement mode fruit and vegetable pictures, that is, pictures under natural light, artificial light and other environments can be acquired, for example, pictures under different natural light conditions are acquired by shooting at different time periods such as sunny days, cloudy days and dusk. The light equipment is used to adjust different light intensities from weak light to strong light to simulate various indoor and outdoor light environments. The panoramic scene fruit and vegetable pictures are shot, that is, panoramic pictures of fruits and vegetables are shot from multiple angles (such as top view, eye level and overhead view) to acquire features under different viewing angles, and at the same time, the distance between the camera and the fruit and vegetable is adjusted to acquire panoramic pictures of different scales. The local scene fruit and vegetable pictures focus on capturing details and focus on specific parts of the fruit and vegetable to shoot high-resolution local pictures. Or multiple-angle shooting is performed on the same local part to acquire more comprehensive feature information. The different placement mode fruit and vegetable pictures refer to shooting pictures of multiple fruits and vegetables stacked together to simulate the actual storage or display situation, or shooting pictures of fruits and vegetables scattered on a table or the ground to capture the distribution and appearance of fruits and vegetables in a natural state, and fruits and vegetables can also be hung up for shooting, such as hanging by a rope or a hook to acquire different viewing angles and features.

[0146] Subsequently, the different light intensity fruit and vegetable pictures, the panoramic scene fruit and vegetable pictures, the local scene fruit and vegetable pictures and the different placement mode fruit and vegetable pictures are respectively constructed into a light intensity sample set, a panoramic scene sample set, a local scene sample set and a placement mode sample set, that is, the pictures are classified and organized into different sample sets according to the specific features or scenes exhibited by the pictures.

[0147] The image contrast of each sample image in the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set is adjusted to obtain the light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set under different contrasts, that is, the contrast enhancement or weakening technology is applied to the images in each set to generate a series of image sets with different contrast levels, which can enhance the details in the images, improve the image quality, and provide more data diversity for subsequent image analysis, processing and recognition tasks.

[0148] The light intensity sample set, the panoramic scene sample set, the local scene sample set and the placement mode sample set under different contrasts and after transformation are used to construct a fruit and vegetable feature training set. Specifically, useful features are extracted from the preprocessed images using a feature extraction algorithm or a deep learning model. For the deep learning model, a convolutional neural network or the like can be used. The extracted features are combined with the corresponding label information to construct a training set. Each training sample usually includes a feature vector and one or more labels. Subsequently, the training set is divided into a training subset, a validation subset and a test subset. The training subset is used to train the model, the validation subset is used to adjust the model parameters and select the best model, and the test subset is used to evaluate the final performance of the model. The training set can be divided according to the actual situation to obtain the fruit and vegetable feature training set.

[0149] By constructing the fruit and vegetable feature training set, the generalization ability of the model can be improved, which helps the model to learn the feature representation of fruits and vegetables under different conditions, thereby improving the generalization ability of the model on unseen data. The recognition performance of the model can also be improved, and the accuracy of the model in identifying fruits and vegetables can be improved.

[0150] In one of the embodiments of the present embodiment, the fruit and vegetable category prediction model includes a low-level feature extraction layer and a high-level feature extraction layer. The fruit and vegetable image is input into a preset fruit and vegetable category prediction model to determine the fruit and vegetable category of each fruit and vegetable in the fruit and vegetable placement region, including:

[0151] S810, extracting a feature map of basic features of the fruit and vegetable image through the low-level feature extraction layer, wherein the basic features include image edge features, image color features and image texture features;

[0152] S820, extracting a feature map of semantic information of the fruit and vegetable image through the high-level feature extraction layer, wherein the semantic information includes object semantic information, attribute semantic information and scene global semantic information in the fruit and vegetable image;

[0153] S830, transmitting the feature map of the basic features to the high-level feature extraction layer through a preset feature channel;

[0154] S840, the feature maps of the basic features and the feature maps of the semantic information are fused by a transverse connection manner to obtain final features of the fruit and vegetable image;

[0155] S850, the final features are matched with features corresponding to each fruit and vegetable type in the third preset database to determine the fruit and vegetable type of each fruit and vegetable in the fruit and vegetable placement area.

[0156] The basic feature of the fruit and vegetable image is extracted through the low-level feature extraction layer, wherein the basic feature includes image edge feature, image color feature and image texture feature. In this embodiment, the low-level feature extraction layer refers to the layer close to the input layer. The low-level feature extraction layer mainly extracts shallow features. The shallow features retain the original form of the input data, so they can clearly show the contour, edge and other geometric information. The low-level features are very sensitive to the local structure and details of the image. That is, the low-level features such as color, shape and texture are extracted through the low-level feature extraction layer. The color feature can be extracted by color histogram, and the shape and texture features can be extracted by edge detection and other methods. After the low-level features such as color, shape and texture are extracted, these features can be represented in the form of feature maps. The feature map is a two-dimensional array or matrix, where each element corresponds to a pixel point or region in the image and contains the feature value of the pixel point or region. The color feature map can be generated according to the color histogram or color matrix. It maps each pixel point or region in the image to the corresponding color feature value, thereby obtaining the color feature map. The shape feature map can be generated according to the edge detection or contour extraction result. It represents the edge points or contour lines in the image in a specific way, thereby obtaining the shape feature map. The texture feature map can be generated according to the gray level co-occurrence matrix or LBP feature. It maps each pixel point or region in the image to the corresponding texture feature value, thereby obtaining the texture feature map.

[0157] Subsequently, the feature map of the semantic information of the fruit and vegetable image is extracted through a high-level feature extraction layer, wherein the semantic information includes object semantic information, attribute semantic information and scene global semantic information in the fruit and vegetable image. In this embodiment, the high-level feature extraction layer refers to the layer close to the output layer, and the high-level feature extraction layer mainly extracts high-level semantic information. In the high-level feature extraction layer, the model has learned the basic features of the image (such as edges, textures, etc.) through the low-level feature extraction layer, and further abstracts and integrates these features to extract higher-level semantic information. For example, in fruit and vegetable image recognition, the high-level feature extraction layer can identify the specific fruit and vegetable species (such as apples, bananas, etc.) in the image and judge their maturity, color and other attributes. In addition, if the image contains the environment or scene information where the fruit and vegetable is located, the high-level feature extraction layer can also capture these global semantic information, so as to more comprehensively understand the image content. The object semantic information mainly includes the species and individuals of the fruit and vegetable in the image; the attribute semantic information focuses on the color, size, maturity and other attributes of the fruit and vegetable; the scene global semantic information focuses on the environment, background and other information where the fruit and vegetable is located; after the object semantic information, the attribute semantic information and the scene global semantic information are extracted, the high-level feature extraction layer will represent these information in the form of a feature map.

[0158] Next, the feature map of the basic feature is transmitted to the high-level feature extraction layer through a preset feature channel, that is, the feature map of the basic feature extracted by the low-level network layer is transmitted to the high-level feature extraction layer through a predefined feature channel, and the predefined feature channel can be determined according to the model design and task requirements.

[0159] The feature map of the basic feature and the feature map of the semantic information are fused through a transverse connection mode to obtain the final feature of the fruit and vegetable image. The transverse connection is an operation in the feature pyramid network for fusing the up-sampled high-level feature map with the low-level feature map, which is usually realized by concatenating or element-wise adding the two feature maps in the channel dimension, that is, in order to construct an effective network structure model, the low-level and high-level features need to be fused. Through a skip connection, a feature pyramid network or an attention mechanism. The fused feature map contains both geometric information and rich semantic information. Specifically, in the feature pyramid network, the low-level feature is fused with the high-level feature through up-sampling to form a multi-scale feature map with rich semantic information. Up-sampling is a basic operation in signal processing and image processing, which aims to increase the sampling rate of the signal or the resolution of the image. Up-sampling can increase the sampling rate of the signal or the resolution of the image by copying the value of the nearest pixel to fill the new pixel position through nearest neighbor interpolation.

[0160] Subsequently, the final feature is matched with the feature corresponding to each fruit and vegetable category in the third preset database to determine the fruit and vegetable category of each fruit and vegetable in the fruit and vegetable placement area. Specifically, the feature vector obtained after the fruit and vegetable image is processed by the neural network model. This feature vector can contain key feature information. In the third preset database, the final feature is compared with the feature of each fruit and vegetable in the database. The similarity or distance between the two feature vectors can be calculated. The similarity can be calculated using the Euclidean distance method. According to the feature matching result, a most possible fruit and vegetable category can be determined for each fruit in the fruit and vegetable image, that is, the fruit and vegetable category of each fruit in the fruit and vegetable placement area is determined.

[0161] In another embodiment of the present embodiment, the fruit and vegetable feature of each fruit and vegetable category is determined, including the following steps:

[0162] All fruit and vegetable pictures are segmented into target cells, and the first similarity between each target cell and the adjacent target cell is calculated. Specifically, first, the fruit and vegetable picture needs to be segmented and divided into multiple target cells. Image segmentation techniques such as contour detection or coordinate point-based segmentation can be used. In each segmented target cell, the features of the fruit and vegetable object are extracted, which can include color, texture, shape, etc. The similarity between each target cell and the adjacent target cell is calculated, and the first similarity is obtained using similarity calculation methods such as Euclidean distance, cosine similarity, etc.

[0163] It is judged whether the first similarity is greater than a first preset similarity threshold. In the present embodiment, the first preset similarity threshold can be determined according to the actual situation, that is, the calculated first similarity is compared with the first preset similarity threshold. If the first similarity is greater than the threshold, it indicates that the similarity between the two cells is high, and they may belong to the same category.

[0164] If the similarity is greater than the first preset similarity threshold, the region merging of each target cell and the adjacent target cell is performed to obtain a primary candidate region. In the present embodiment, the primary candidate region is the primary candidate region of image recognition, that is, if the similarity is greater than the first preset similarity threshold, it indicates that the adjacent target cells have high similarity and may belong to the same fruit and vegetable object or the same category. In this case, region merging can be performed to obtain a larger and more complete primary candidate region.

[0165] After obtaining the primary candidate regions, a second similarity between each primary candidate region and a neighboring primary candidate region is calculated. Specifically, a feature of each primary candidate region is extracted, which can be color, texture, shape, or any other information capable of representing the characteristics of the region. The cosine similarity is used to calculate the similarity between each primary candidate region and its neighboring region. Finally, the calculated similarity result is output to obtain the second similarity.

[0166] If the second similarity is greater than a second preset similarity threshold, each primary candidate region and a neighboring primary candidate region are merged to obtain a final candidate region. In this embodiment, the second preset similarity threshold can be determined according to actual conditions. Specifically, when the second similarity between each primary candidate region and its neighboring region is greater than the second preset similarity threshold, the neighboring primary candidate regions with a similarity greater than the threshold are merged. The merging manner can be to simply merge the boundaries of the two regions to form a larger region to obtain the final candidate region. In this embodiment, the final candidate region is the final region for image recognition.

[0167] Each final candidate region is input into a pre-trained convolutional neural network model to obtain the region feature of each final candidate region. That is, each candidate region is input as an input, and forward propagation is performed through the pre-trained convolutional neural network model. In the forward propagation process, the candidate region will pass through multiple convolutional layers, pooling layers, and fully connected layers to gradually extract clearer features. In the pre-trained convolutional neural network model, the features extracted through the fully connected layer or the convolutional layer can be a high-dimensional vector containing semantic information and spatial information of the candidate region, thereby obtaining the region feature of each final candidate region.

[0168] The region features of all final candidate regions are classified using a support vector machine algorithm to determine the fruit and vegetable features of each fruit and vegetable category. The support vector machine is a supervised learning algorithm commonly used for classification and regression analysis. That is, the trained support vector machine algorithm is used to classify the features of the final candidate regions to obtain the fruit and vegetable category to which each candidate region belongs. The classification results are analyzed to determine the typical features of each fruit and vegetable category. Based on the typical features of each fruit and vegetable category, the fruit and vegetable features of each fruit and vegetable category in the fruit and vegetable placement region are determined.

[0169] By combining multiple key steps such as low-level feature extraction, high-level semantic information extraction, feature fusion, and database matching, automatic fruit and vegetable classification and recognition can be achieved, improving the work efficiency and accuracy of fruit and vegetable classification and recognition.

[0170] The present application provides an electronic device, comprising:

[0171] a memory configured to store instructions; and

[0172] a processor configured to call the instructions from the memory and implement the above-mentioned machine vision-based fruit and vegetable intelligent settlement method when the instructions are executed.

[0173] The present application also provides a fruit and vegetable intelligent settlement system 10, as shown in the accompanying drawings, comprising: Figure 2

[0174] the above-mentioned electronic device;

[0175] a weighing module 20 connected with the electronic device;

[0176] an image acquisition module 30 connected with the electronic device;

[0177] a touch screen 40 connected with the electronic device, the weighing module and the image acquisition module.

[0178] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be realized by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 an apparatus for realizing the functions specified in one or more flows and / or blocks. Figure 1 an apparatus for realizing the functions specified in one or more flows and / or blocks.

[0180] These computer program instructions can also be stored in a computer-readable memory capable of guiding the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction apparatus, which realizes the functions specified in the flowcharts and / or block diagrams. Figure 1 an apparatus for realizing the functions specified in one or more flows and / or blocks. Figure 1 an apparatus for realizing the functions specified in one or more flows and / or blocks.

[0181] ​These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 The flowchart blocks or blocks in the multiple flows and / or blocks

[0182] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0183] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), flash memory, or a combination of non-volatile memories in different types. The memory is an example of computer readable media.

[0184] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0185] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0186] ​The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A machine vision-based intelligent settlement method for fruits and vegetables, characterized in that, Applied to a smart settlement system for fruits and vegetables, the smart settlement system for fruits and vegetables includes a touch screen, a weighing module, and an image acquisition module, the method includes: In response to the user's payment instruction via the touch screen, the image acquisition module acquires images of fruits and vegetables within the fruit and vegetable placement area. The actual weight of all fruits and vegetables in the fruit and vegetable placement area is obtained through the weighing module. The fruit and vegetable images are input into a preset fruit and vegetable type prediction model to determine at least one type of fruit and vegetable in the fruit and vegetable placement area, wherein each type of fruit and vegetable corresponds to at least one fruit and vegetable. A first weight value is determined for each type of fruit and vegetable based on the fruit and vegetable images; The density of fruits and vegetables corresponding to each type of fruit and vegetable is determined in a first preset database, and a second weight value for each type of fruit and vegetable is determined based on the density of fruits and vegetables; Based on the first weight value and the second weight value, a comprehensive weight value is determined for each type of fruit and vegetable; The settlement weight for each type of fruit and vegetable is allocated based on the comprehensive weight value and the actual weight of each type of fruit and vegetable. Determine the unit price for each type of fruit and vegetable in the second preset database; The sum of the products of the settlement weight of all the fruits and vegetables and the corresponding unit price is used as the settlement price and then output.

2. The method according to claim 1, characterized in that, Determining a first weight value for each type of fruit and vegetable based on the fruit and vegetable image includes: For each type of fruit and vegetable, the outline area of ​​each fruit and vegetable is extracted from the fruit and vegetable image, and the outline areas of all the fruits and vegetables are summarized to obtain the outline area of ​​the fruit and vegetable corresponding to each type of fruit and vegetable. Sum the outline areas of all the fruits and vegetables mentioned above to obtain the total outline area; Based on the outline area of ​​each fruit and vegetable type and the sum of the outline areas, a first weight value is determined for each fruit and vegetable type.

3. The method according to claim 1, characterized in that, The determination of a second weight value for each type of fruit and vegetable based on the fruit and vegetable density includes: Sum the densities of all the aforementioned fruit and vegetable types to obtain the total density. A second weight value is determined for each type of fruit and vegetable based on the fruit and vegetable density corresponding to each type of fruit and vegetable and the sum of the densities.

4. The method according to claim 1, characterized in that, The step of determining the comprehensive weight value for each type of fruit and vegetable based on the first weight value and the second weight value includes: Obtain fruit and vegetable area data, fruit and vegetable density data, and fruit and vegetable quality data corresponding to the fruit and vegetable types from the first preset database; Using a preset first correlation coefficient formula, the correlation coefficient between the fruit and vegetable area data and the fruit and vegetable quality data is calculated to obtain the first weight coefficient of the first weight value; Using a preset second correlation coefficient formula, the correlation coefficient between the fruit and vegetable density data and the fruit and vegetable quality data is calculated to obtain the second weight coefficient of the second weight value; Based on the first weight value, the first weight coefficient, the second weight value, and the second weight coefficient, calculate the comprehensive weight for each type of fruit and vegetable.

5. The method according to claim 1, characterized in that, After acquiring images of fruits and vegetables within the fruit and vegetable placement area via the image acquisition module, the process includes: Extract the edge and color features of the fruit and vegetable images; If the color feature is a preset color and there is no matching fruit and vegetable edge feature in the third preset database, a reminder message is issued and displayed on the touch screen. The reminder message is used to prompt the user to remove the packaging bag or box of the fruit and vegetable.

6. The method according to claim 5, characterized in that, The steps for constructing the fruit and vegetable type prediction model include: Determine the image size in the fruit and vegetable feature training set; The spatial dimension of the input layer of the fruit and vegetable type prediction model is determined by using the image size in the fruit and vegetable feature training set. The number of color channels in the input layer of the fruit and vegetable type prediction model is determined based on the color features of the fruits and vegetables. The number of neurons in the output layer of the fruit and vegetable type prediction model is determined by using all the aforementioned fruit and vegetable types. The fruit and vegetable type prediction model is trained using the fruit and vegetable feature training set, and then constructed by combining the spatial dimension of the input layer, the number of color channels, and the number of neurons in the output layer.

7. The method according to claim 6, characterized in that, The steps for constructing the fruit and vegetable feature training set include: Using the image acquisition module, images of fruits and vegetables under different light intensities, panoramic scenes of fruits and vegetables, partial scenes of fruits and vegetables, and images of fruits and vegetables placed in different ways are acquired. The fruit and vegetable images with different light intensities, the panoramic scene fruit and vegetable images, the local scene fruit and vegetable images, and the fruit and vegetable images with different placement methods are respectively used to construct light intensity sample sets, panoramic scene sample sets, local scene sample sets, and placement method sample sets; Image space geometric transformation is performed on each sample image in the light intensity sample set, panoramic scene sample set, local scene sample set, and placement method sample set to obtain the transformed light intensity sample set, panoramic scene sample set, local scene sample set, and placement method sample set. Image contrast adjustment is performed on each sample image in the light intensity sample set, panoramic scene sample set, local scene sample set, and placement method sample set to obtain light intensity sample set, panoramic scene sample set, local scene sample set, and placement method sample set under different contrast. The fruit and vegetable feature training set is constructed by using the sample sets of light intensity under different contrasts and the transformed light intensity, the panoramic scene sample set, the local scene sample set, and the placement method sample set.

8. The method according to claim 6, characterized in that, The fruit and vegetable type prediction model includes a low-level feature extraction layer and a high-level feature extraction layer. The step of inputting the fruit and vegetable image into the preset fruit and vegetable type prediction model to determine the type of each fruit and vegetable in the fruit and vegetable placement area includes: The low-level feature extraction layer extracts feature maps of the basic features of the fruit and vegetable image, wherein the basic features include image edge features, image color features, and image texture features. The high-level feature extraction layer extracts feature maps of semantic information from the fruit and vegetable images, wherein the semantic information includes object semantic information, attribute semantic information, and scene global semantic information in the fruit and vegetable images. The feature map of the basic features is transmitted to the high-level feature extraction layer using a preset feature channel; By using a horizontal connection method, the feature maps of the basic features and the feature maps of the semantic information are fused to obtain the final features of the fruit and vegetable image; The final feature is matched with the feature corresponding to each type of fruit and vegetable in the third preset database to determine the type of fruit and vegetable in the fruit and vegetable placement area.

9. An electronic device, characterized in that, include: The memory is configured to store instructions; as well as The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the machine vision-based intelligent settlement method for fruits and vegetables according to any one of claims 1 to 8.

10. A smart settlement system for fruits and vegetables, characterized in that, include: The electronic device according to claim 9; The weighing module is connected to the electronic device; The image acquisition module is connected to the electronic device; The touch screen is connected to the electronic device, the weighing module, and the image acquisition module.

Images