Fruit counting methods, apparatus, equipment and storage media
By using the VGG-16 network and local feature extraction technology, combined with the Hungarian algorithm to optimize the counting model, the problem of low counting accuracy in plant fruit images was solved, and higher accuracy fruit counting was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-27
- Publication Date
- 2026-04-03
AI Technical Summary
In existing technologies, directly using crowd counting methods to count plant fruit images results in low fruit counting accuracy.
The VGG-16 network is used for multiple convolutions and pooling to extract feature information from fruit images. The counting model is then optimized by combining local feature extraction and contrast feature transformation with the Hungarian algorithm and loss function to improve the accuracy of fruit counting.
Through meticulous feature extraction and precise counting models, the accuracy and precision of fruit counting have been significantly improved.
Smart Images
Figure CN115424256B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image counting technology, and in particular to a fruit counting method, apparatus, device, and storage medium. Background Technology
[0002] Current methods for counting plant fruits use crowd counting methods. However, when applying crowd counting methods to count plant fruits, the feature information of plant fruit images is naturally different from that of crowd images. Directly using crowd counting methods to count the fruits in plant fruit images will result in low counting accuracy. Summary of the Invention
[0003] The main objective of this invention is to provide a fruit counting method, apparatus, device, and storage medium, aiming to solve the technical problem of low fruit counting accuracy caused by directly using crowd counting methods to count fruits in plant fruit images in the prior art.
[0004] To achieve the above objectives, the present invention provides a fruit counting method, the method comprising the following steps:
[0005] Obtain the image of the first fruit;
[0006] The first fruit image is input into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information;
[0007] Local feature extraction is performed on the first feature information to obtain several blocks of local feature information;
[0008] The local feature information of the several blocks is transformed and cross-referenced according to the first feature information to obtain several contrast feature information, and the second feature information is obtained according to the several contrast features and the first feature information.
[0009] The second feature information is input into the classification network of the first counting model to obtain the number of fruits in the first fruit image.
[0010] Optionally, the VGG-16 network includes a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, a max pooling layer, and a fourth convolutional layer connected in sequence.
[0011] The first fruit image is input into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information, including:
[0012] The first fruit image is sequentially input into the first convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the first preset number of channels is output.
[0013] The feature information of the first preset number of channels is sequentially input into the second convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the second preset number of channels is output, wherein the second preset number is greater than the first preset number;
[0014] The feature information of the second preset number of channels is sequentially input into the third convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the third preset number of channels is output, wherein the third preset number is greater than the second preset number;
[0015] The feature information of the third preset number of channels is sequentially input into the fourth convolutional layer of the VGG-16 network, and the feature information of the fourth preset number of channels is output. The feature information of the fourth preset number of channels is the first feature information, and the fourth preset number is greater than the third preset number.
[0016] Optionally, the step of extracting local features from the first feature information to obtain several blocks of local feature information includes:
[0017] The first feature information is sequentially input into the first average pooling layer and the target convolutional layer to obtain the first block of local feature information, wherein the target convolutional layer is a convolutional layer with a kernel of 1;
[0018] The first feature information is sequentially input into the second average pooling layer and the target convolutional layer to obtain the second block of local feature information;
[0019] The first feature information is sequentially input into the third average pooling layer and the target convolutional layer to obtain the third block of local feature information.
[0020] Optionally, the step of transforming and crossing the plurality of block local feature information according to the first feature information to obtain a plurality of contrast features, and obtaining second feature information according to the plurality of contrast feature information and the first feature information, includes:
[0021] The first block of local feature information, the second block of local feature information, and the third block of local feature information are respectively input into the upsampling layer to convert the first block of local feature information, the second block of local feature information, and the third block of local feature information into the size of the first feature information, and obtain new first block of local feature information, new second block of local feature information, and new third block of local feature information.
[0022] Based on the differences between the new first block of local feature information, the new second block of local feature information, and the new third block of local feature information and the first feature information, respectively, a first difference feature, a second difference feature, and a third difference feature are obtained;
[0023] The first contrast feature, the second contrast feature, and the third contrast feature are obtained by determining the product of the first local feature information and the first difference feature, the product of the second local feature information and the second difference feature, and the product of the third local feature information and the third difference feature.
[0024] The first feature information, the first contrast feature, the second contrast feature, and the third contrast feature are superimposed to obtain the second feature information.
[0025] Optionally, before inputting the first fruit image into the VGG-16 network of the first counting model for multiple convolutions and pooling, the method further includes:
[0026] Obtain a second fruit image, wherein the true center point of each fruit is marked in the second fruit image;
[0027] The second fruit image is input into the VGG-16 network of the second counting model to obtain the first target feature information;
[0028] Based on the contextual method, the first feature information is processed to obtain the second target feature information;
[0029] The second target feature information is input into the classification network of the second counting model to obtain the predicted center point in the second fruit image;
[0030] The predicted center points are paired with the true center points based on the Hungarian algorithm, and a loss function is established.
[0031] The second counting model is updated based on the loss function to obtain the first counting model.
[0032] Optionally, the step of pairing the predicted center points with the true center points based on the Hungarian algorithm and establishing a loss function includes:
[0033] The predicted center point and the true center point are paired based on the Hungarian algorithm to obtain the predicted center point and the true center point of each fruit.
[0034] The Euclidean distance difference of each fruit in the second fruit image is determined based on the predicted center point and the true center point.
[0035] Based on the Euclidean distance difference, a loss function is established.
[0036] Optionally, the step of inputting the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image includes:
[0037] The second feature information is input into the classification network of the first counting model to determine the probability value of each pixel in the first fruit image;
[0038] The number of fruits in the first fruit image is determined based on the probability value.
[0039] Furthermore, to achieve the above objectives, the present invention also proposes a fruit counting device, the fruit counting device comprising:
[0040] The acquisition module is used to acquire the image of the first fruit;
[0041] The acquisition module is used to input the first fruit image into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information;
[0042] The acquisition module is used to extract local features from the first feature information to obtain several blocks of local feature information;
[0043] The acquisition module is used to transform and cross the local feature information of the plurality of blocks according to the first feature information to obtain a plurality of contrast feature information, and to obtain second feature information according to the plurality of contrast features and the first feature information.
[0044] The counting module is used to input the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image.
[0045] In addition, to achieve the above objectives, the present invention also proposes a fruit counting device, which includes: a memory, a processor, and a fruit counting program stored in the memory and executable on the processor, the fruit counting program being configured to implement the steps of the fruit counting method described above.
[0046] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a fruit counting program, which, when executed by a processor, implements the steps of the fruit counting method described above.
[0047] The fruit counting method proposed in this invention involves: acquiring a first fruit image; inputting the first fruit image into a VGG-16 network of a first counting model for multiple convolutions and pooling to obtain first feature information; extracting local features from the first feature information to obtain several block local feature information; transforming and cross-referencing the several block local feature information based on the first feature information to obtain several contrast feature information; and obtaining second feature information based on the several contrast features and the first feature information; and inputting the second feature information into a classification network of the first counting model to obtain the number of fruits in the first fruit image. Through this method, the first feature information of the fruit image is initially extracted using the VGG-16 network, and then more image feature information is extracted from the first feature information through perception at different scales, thereby obtaining more detailed fruit features. These more detailed fruit features are then input into a classification network to count the number of fruits in the fruit image, thus improving the final fruit counting accuracy. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the structure of the fruit counting device in the hardware operating environment involved in the embodiments of the present invention;
[0049] Figure 2 This is a flowchart illustrating the first embodiment of the fruit counting method of the present invention;
[0050] Figure 3 This is a schematic diagram of the VGG-16 network structure in the fruit counting method of the present invention;
[0051] Figure 4 This refers to the first feature information extracted by the VGG-16 network in the fruit counting method of the present invention;
[0052] Figure 5 This is a schematic diagram of the context-aware method in the fruit counting method of the present invention;
[0053] Figure 6 This refers to the second feature information extracted using a context-aware method in the fruit counting method of this invention.
[0054] Figure 7 This is a flowchart illustrating the fruit counting method of the present invention, which calculates the number of fruits using a classification network.
[0055] Figure 8 This is a flowchart illustrating the second embodiment of the fruit counting method of the present invention;
[0056] Figure 9 This is a structural block diagram of the first embodiment of the fruit counting device of the present invention.
[0057] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0058] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0059] In the description of this invention, it should be understood that the designations themselves, such as "first" and "second", are only used to distinguish the objects being described and have no technical meaning, and should not be construed as specifying or implying the importance of the objects being described.
[0060] Reference Figure 1 , Figure 1 This is a schematic diagram of the fruit counting device structure in the hardware operating environment of the embodiment of the present invention.
[0061] like Figure 1 As shown, the fruit counting device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.
[0062] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the fruit counting device and may include more or fewer parts than shown, or combine certain parts, or have different arrangements of parts.
[0063] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a fruit counting program.
[0064] exist Figure 1In the fruit counting device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the fruit counting device of the present invention can be set in the fruit counting device. The fruit counting device calls the fruit counting program stored in the memory 1005 through the processor 1001 and executes the fruit counting method provided in the embodiment of the present invention.
[0065] This invention provides a method for counting fruits, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a fruit counting method according to the present invention.
[0066] In this embodiment, the fruit counting method includes the following steps:
[0067] Step S10: Obtain the first fruit image.
[0068] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, or personal computer, or an electronic device or fruit counting device capable of performing the above functions. The following description uses the fruit counting device as an example to illustrate this embodiment and the subsequent embodiments.
[0069] It should be noted that the first fruit image can be collected by an individual or directly obtained from a publicly available plant fruit dataset. For example, using apples as an example, apple fruit images can be directly obtained from the ACFR APPLE dataset. If the first fruit image is collected by an individual, it can be taken during the fruit's ripening period. This ensures that most fruits in the collected images are ripe, preventing an excessive number of immature fruits from affecting the counting accuracy of ripe fruits. Specifically, image acquisition equipment can be used to collect fruit images of the orchard under test. This equipment can be any of a depth camera, hyperspectral camera, thermal infrared camera, or remote sensing camera. Fruit image collection can be conducted during periods of good lighting, or supplemental lighting can be applied to the image acquisition equipment to improve the quality of the collected orchard images, thereby improving the counting accuracy. The resolution of the fruit image can be 308×208.
[0070] It should be noted that the first fruit image can be understood as a fruit image in which the true center point of each fruit in the image is not labeled.
[0071] Step S20: Input the first fruit image into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information.
[0072] It should be noted that the first counting model was obtained by training the initial counting model multiple times using fruit images labeled with the true center point of each fruit.
[0073] It should be noted that performing multiple convolutions on the first fruit image can extract the feature information in the fruit image, that is, store the features in the fruit image in the feature matrix; performing multiple pooling operations on the first fruit image can effectively remove redundant information in the feature information.
[0074] In one embodiment, the VGG-16 network includes a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, a max pooling layer, and a fourth convolutional layer connected in sequence.
[0075] The first fruit image is input into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information, including:
[0076] The first fruit image is sequentially input into the first convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the first preset number of channels is output.
[0077] The feature information of the first preset number of channels is sequentially input into the second convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the second preset number of channels is output, wherein the second preset number is greater than the first preset number;
[0078] The feature information of the second preset number of channels is sequentially input into the third convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the third preset number of channels is output, wherein the third preset number is greater than the second preset number;
[0079] The feature information of the third preset number of channels is sequentially input into the fourth convolutional layer of the VGG-16 network, and the feature information of the fourth preset number of channels is output. The feature information of the fourth preset number of channels is the first feature information, and the fourth preset number is greater than the third preset number.
[0080] Specifically, such as Figure 3 As shown, the VGG-16 network includes a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, a max pooling layer, and a fourth convolutional layer connected in sequence.
[0081] It should be noted that the feature information of the preset number of channels can be understood as a feature matrix of the preset number of dimensions. For example, the feature information of 64 channels can be understood as a feature matrix of 64 dimensions.
[0082] It should be noted that the first preset number is related to the number of convolutional kernels in the first convolutional layer, the second preset number is related to the number of convolutional kernels in the second convolutional layer, the third preset number is related to the number of convolutional kernels in the third convolutional layer, and the fourth preset number is related to the number of convolutional kernels in the fourth convolutional layer. The number of convolutional kernels in the first, second, third, and fourth convolutional layers can be set in advance according to the actual situation. For example, if the kernel size of each convolutional layer is 3, the first preset number can be 64, the second preset number can be 128, the third preset number can be 256, and the fourth preset number can be 512.
[0083] Specifically, Figure 4 The image shows the first feature information extracted through the VGG-16 network.
[0084] In this embodiment, by adding a max pooling layer between each part of the convolutional layer, redundant information in the feature information can be effectively removed, thereby effectively improving the fruit counting accuracy of the first counting model.
[0085] Step S30: Extract local features from the first feature information to obtain several blocks of local feature information.
[0086] It should be noted that a contextual approach can be used to extract more feature information from fruit images by perceiving at different scales. Specifically, several local feature information blocks can be extracted from the first feature information by using average pooling layers with different output sizes. A schematic diagram of the context-aware method is shown below. Figure 5 As shown, the specific implementation is as follows:
[0087] In one embodiment, the step of extracting local features from the first feature information to obtain several blocks of local feature information includes:
[0088] The first feature information is sequentially input into the first average pooling layer and the target convolutional layer to obtain the first block of local feature information, wherein the target convolutional layer is a convolutional layer with a kernel of 1;
[0089] The first feature information is sequentially input into the second average pooling layer and the target convolutional layer to obtain the second block of local feature information;
[0090] The first feature information is sequentially input into the third average pooling layer and the target convolutional layer to obtain the third block of local feature information.
[0091] It should be noted that the average pooling layer is used to calculate the average value of the feature information of each channel. The main difference between the first average pooling layer, the second average pooling layer, and the third average pooling layer is that they are at different output scales. Inputting the first feature information into the first average pooling layer, the second average pooling layer, and the third average pooling layer respectively can yield local feature information of different sizes.
[0092] Step S40: The local feature information of the several blocks is transformed and cross-referenced according to the first feature information to obtain several contrast feature information, and second feature information is obtained according to the several contrast features and the first feature information.
[0093] It should be noted that by transforming the aforementioned block local feature information, block local feature information of the same size as the first feature information can be obtained. Crossover can be understood as performing a product operation.
[0094] In one embodiment, the step of transforming and crossing the plurality of block local feature information according to the first feature information to obtain a plurality of contrast features, and obtaining second feature information according to the plurality of contrast feature information and the first feature information, includes:
[0095] The first block of local feature information, the second block of local feature information, and the third block of local feature information are respectively input into the upsampling layer to convert the first block of local feature information, the second block of local feature information, and the third block of local feature information into the size of the first feature information, and obtain new first block of local feature information, new second block of local feature information, and new third block of local feature information.
[0096] Based on the differences between the new first block of local feature information, the new second block of local feature information, and the new third block of local feature information and the first feature information, respectively, a first difference feature, a second difference feature, and a third difference feature are obtained;
[0097] The product of the first local feature information and the first difference feature, the product of the second local feature information and the first difference feature, and the product of the third local feature information and the first difference feature are determined to obtain the first contrast feature, the second contrast feature, and the third contrast feature.
[0098] The first feature information, the first contrast feature, the second contrast feature, and the third contrast feature are superimposed to obtain the second feature information.
[0099] It is understandable that the sizes of the new first local feature information, the new second local feature information, and the new third local feature information are all consistent with the size of the first feature information.
[0100] It should be noted that the difference feature is obtained by removing the new local feature information from the first feature information, that is, by subtracting the first feature information from the new local feature information.
[0101] It should be noted that the first contrast feature, the second contrast feature, and the third contrast feature were obtained using average pooling layers with different output scales.
[0102] It is understandable that superimposing the first contrast feature, the second contrast feature, and the third contrast feature onto the first feature information can yield more detailed fruit image feature information, namely the second feature information. The second feature information is more detailed than the first feature information, such as... Figure 6 As shown, Figure 6 This refers to the second feature information extracted based on the context-aware method.
[0103] Step S50: Input the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image.
[0104] In one embodiment, the step of inputting the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image includes:
[0105] The second feature information is input into the classification network of the first counting model to determine the probability value of each pixel in the first fruit image;
[0106] The number of fruits in the first fruit image is determined based on the probability value.
[0107] It should be noted that for the second feature information, classification and regression processing can be implemented using the point matching method proposed in the P2PNET network to achieve the statistical analysis of the number of fruits in the fruit image. The classification network includes a classification layer and a regression layer. The classification processing is mainly performed by the classification layer, which determines the probability value of each pixel in the fruit image as the center point of the fruit. The regression processing is mainly performed by the regression layer, which identifies pixels in the fruit image with a probability value greater than a preset probability value and uses these pixels as the center points of the fruit.
[0108] In specific implementations, such as Figure 7 As shown, the second feature information is input into the classification network in the first counting model. First, the classification layer of the classification network determines the probability value of each pixel in the second feature information as the center of the fruit. Then, the regression layer of the classification network uses the pixels with a probability value greater than the preset probability value as the center of the fruit. Thus, the number of fruits in the fruit image can be determined based on the number of fruit centers in the fruit image.
[0109] This embodiment acquires a first fruit image; inputs the first fruit image into a VGG-16 network of a first counting model for multiple convolutions and pooling to obtain first feature information; extracts local features from the first feature information to obtain several block local feature information; transforms and crosses the several block local feature information according to the first feature information to obtain several contrast feature information, and obtains second feature information based on the several contrast features and the first feature information; inputs the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image. Through the above method, the first feature information of the fruit image is initially extracted using the VGG-16 network, and then more image feature information is extracted from the first feature information through perception at different scales, thereby obtaining more detailed fruit features. Finally, the more detailed fruit features are input into the classification network to count the number of fruits in the fruit image, thus improving the final fruit counting accuracy.
[0110] refer to Figure 8 , Figure 8 This is a flowchart illustrating a second embodiment of a fruit counting method according to the present invention.
[0111] Based on the first embodiment described above, the fruit counting method of this embodiment further includes, before step S20:
[0112] Step S101: Obtain a second fruit image, wherein the true center point of each fruit is marked in the second fruit image.
[0113] It should be noted that the second fruit image can be collected by an individual and then the true center point of the fruit is labeled, or it can be directly obtained from a publicly available plant fruit dataset and then the true center point of the fruit is labeled. For example, taking apple fruit as an example, apple fruit images can be directly obtained from the ACFR APPLE dataset. If the second fruit image is collected by an individual, it can be collected when the fruit is ripe. This can ensure that most of the fruits in the collected fruit image are ripe and prevent the counting accuracy of ripe fruits from being affected by the large number of immature fruits in the fruit image.
[0114] Step S102: Input the second fruit image into the VGG-16 network of the second counting model to obtain the first target feature information.
[0115] It should be noted that the second counting model can be understood as an untrained network model or a poorly trained network model.
[0116] Step S103: Based on the Contextual method, process the first target feature information to obtain the second target feature information.
[0117] It should be noted that the Contextual method originates from the Context-Aware Network crowd counting network structure and is a method to extract more image feature information through perception at different scales. That is, the method mentioned in steps S30-S40 of the first embodiment can further process the first target feature information to obtain more detailed plant and fruit features, i.e., the second target feature information.
[0118] Step S104: Input the second target feature information into the classification network of the second counting model to obtain the predicted center point in the second fruit image.
[0119] It should be noted that for the second target feature information, classification and regression processing can be achieved using the point matching method proposed in the P2PNET network to count the number of fruits in the fruit image. The classification network includes a classification layer and a regression layer. Classification processing is mainly performed by the classification layer, which determines the predicted probability value of each pixel in the fruit image as the center point of the fruit. Regression processing is mainly performed by the regression layer, which identifies pixels in the fruit image whose predicted probability value is greater than a preset probability value and uses these pixels as the predicted center points of the fruit.
[0120] Step S105: Pair the predicted center point with the real center point based on the Hungarian algorithm and establish a loss function.
[0121] It is understandable that since there are multiple fruits in a fruit image, there are multiple predicted center points and true center points in each second fruit image. Therefore, it is necessary to match the predicted center points with the true center points in each second fruit image. Specifically, the Hungarian algorithm can be used to match the predicted center points with the true center points, forming multiple center point pairs. Each center point corresponds to one fruit in the fruit image, and each fruit includes a predicted center point and a true center point.
[0122] In one embodiment, the step of pairing the predicted center points with the true center points based on the Hungarian algorithm and establishing a loss function includes:
[0123] The predicted center point and the true center point are paired based on the Hungarian algorithm to obtain the predicted center point and the true center point of each fruit.
[0124] The Euclidean distance difference of each fruit in the second fruit image is determined based on the predicted center point and the true center point.
[0125] Based on the Euclidean distance difference, a loss function is established.
[0126] In practical implementation, the specific formula for the loss function is as follows:
[0127]
[0128] In the formula, N represents the number of fruits in the fruit image, i represents the i-th fruit, and P i Let represent the predicted center point coordinates of the i-th fruit, and let Qi represent the actual center point coordinates of the i-th fruit.
[0129] Understandably, the loss function is designed as the Euclidean distance difference between the predicted center point and the true center point to ensure that the second counting model can converge to appropriate parameters during training, so that the difference between the predicted center point and the true center point is minimized.
[0130] Step S106: Update the second counting model based on the loss function to obtain the first counting model.
[0131] It should be noted that updating the second counting model based on the loss function can be understood as using the loss function as a constraint to supervise the training process of the second counting model, thereby updating the parameters in the second counting model, and thus obtaining the trained counting model, i.e., the first counting model.
[0132] Understandably, the training process of the second counting model is obtained by training the initial counting model multiple times using fruit images labeled with the true center point of each fruit. In other words, multiple second fruit images are needed to complete the training of the second counting model.
[0133] In a practical implementation, the probability values of the predicted center points obtained through the classification layer and the loss function can also be used as constraints to jointly supervise the training process of the second counting model, thereby updating the parameters in the second counting model and obtaining the trained counting model, i.e., the first counting model.
[0134] This embodiment acquires a second fruit image, in which the true center point of each fruit is marked; inputs the second fruit image into the VGG-16 network of a second counting model to obtain first target feature information; processes the first feature information based on the contextual method to obtain second target feature information; inputs the second target feature information into the classification network of the second counting model to obtain predicted center points in the second fruit image; pairs the predicted center points with the true center points based on the Hungarian algorithm and establishes a loss function; updates the second counting model based on the loss function to obtain a first counting model. By using the Hungarian algorithm to pair the predicted center points and true center points in the fruit image, the predicted number of fruits in the fruit image can be as close as possible to the actual number of fruits, further improving the accuracy of fruit counting.
[0135] Furthermore, this embodiment of the invention also proposes a storage medium storing a fruit counting program, which, when executed by a processor, implements the steps of the fruit counting method described above.
[0136] Reference Figure 9 , Figure 9 This is a structural block diagram of the first embodiment of the fruit counting device of the present invention.
[0137] like Figure 9 As shown, the fruit counting device proposed in this embodiment of the invention includes:
[0138] The acquisition module 10 is used to acquire the first fruit image.
[0139] The acquisition module 10 is used to input the first fruit image into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information.
[0140] The acquisition module 10 is used to extract local features from the first feature information to obtain several blocks of local feature information.
[0141] The acquisition module 10 is used to convert and cross the local feature information of the plurality of blocks according to the first feature information to obtain a plurality of contrast feature information, and to obtain second feature information according to the plurality of contrast features and the first feature information.
[0142] The counting module 20 is used to input the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image.
[0143] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.
[0144] This embodiment acquires a first fruit image; inputs the first fruit image into a VGG-16 network of a first counting model for multiple convolutions and pooling to obtain first feature information; extracts local features from the first feature information to obtain several block local feature information; transforms and crosses the several block local feature information according to the first feature information to obtain several contrast feature information, and obtains second feature information based on the several contrast features and the first feature information; inputs the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image. Through the above method, the first feature information of the fruit image is initially extracted using the VGG-16 network, and then more image feature information is extracted from the first feature information through perception at different scales, thereby obtaining more detailed fruit features. Finally, the more detailed fruit features are input into the classification network to count the number of fruits in the fruit image, thus improving the final fruit counting accuracy.
[0145] In one embodiment, the VGG-16 network includes a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, a max pooling layer, and a fourth convolutional layer connected in sequence.
[0146] The acquisition module 10 is further configured to:
[0147] The first fruit image is input into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information, including:
[0148] The first fruit image is sequentially input into the first convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the first preset number of channels is output.
[0149] The feature information of the first preset number of channels is sequentially input into the second convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the second preset number of channels is output, wherein the second preset number is greater than the first preset number;
[0150] The feature information of the second preset number of channels is sequentially input into the third convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the third preset number of channels is output, wherein the third preset number is greater than the second preset number;
[0151] The feature information of the third preset number of channels is sequentially input into the fourth convolutional layer of the VGG-16 network, and the feature information of the fourth preset number of channels is output. The feature information of the fourth preset number of channels is the first feature information, and the fourth preset number is greater than the third preset number.
[0152] In one embodiment, the acquisition module 10 is further configured to:
[0153] The step of extracting local features from the first feature information to obtain several blocks of local feature information includes:
[0154] The first feature information is sequentially input into the first average pooling layer and the target convolutional layer to obtain the first block of local feature information, wherein the target convolutional layer is a convolutional layer with a kernel of 1;
[0155] The first feature information is sequentially input into the second average pooling layer and the target convolutional layer to obtain the second block of local feature information;
[0156] The first feature information is sequentially input into the third average pooling layer and the target convolutional layer to obtain the third block of local feature information.
[0157] In one embodiment, the acquisition module 10 is further configured to:
[0158] The first block of local feature information, the second block of local feature information, and the third block of local feature information are respectively input into the upsampling layer to convert the first block of local feature information, the second block of local feature information, and the third block of local feature information into the size of the first feature information, and obtain new first block of local feature information, new second block of local feature information, and new third block of local feature information.
[0159] Based on the differences between the new first block of local feature information, the new second block of local feature information, and the new third block of local feature information and the first feature information, respectively, a first difference feature, a second difference feature, and a third difference feature are obtained;
[0160] The first contrast feature, the second contrast feature, and the third contrast feature are obtained by determining the product of the first local feature information and the first difference feature, the product of the second local feature information and the second difference feature, and the product of the third local feature information and the third difference feature.
[0161] The first feature information, the first contrast feature, the second contrast feature, and the third contrast feature are superimposed to obtain the second feature information.
[0162] In one embodiment, the acquisition module 10 is further configured to:
[0163] Obtain a second fruit image, wherein the true center point of each fruit is marked in the second fruit image;
[0164] The second fruit image is input into the VGG-16 network of the second counting model to obtain the first target feature information;
[0165] Based on the contextual method, the first target feature information is processed to obtain the second target feature information;
[0166] The second target feature information is input into the classification network of the second counting model to obtain the predicted center point in the second fruit image;
[0167] The predicted center points are paired with the true center points based on the Hungarian algorithm, and a loss function is established.
[0168] The second counting model is updated based on the loss function to obtain the first counting model.
[0169] In one embodiment, the acquisition module 10 is further configured to:
[0170] The predicted center point and the true center point are paired based on the Hungarian algorithm to obtain the predicted center point and the true center point of each fruit.
[0171] The Euclidean distance difference of each fruit in the second fruit image is determined based on the predicted center point and the true center point.
[0172] Based on the Euclidean distance difference, a loss function is established.
[0173] In one embodiment, the counting module 20 is further configured to:
[0174] The second feature information is input into the classification network of the first counting model to determine the probability value of each pixel in the first fruit image;
[0175] The number of fruits in the first fruit image is determined based on the probability value.
[0176] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.
[0177] In addition, for technical details not described in detail in this embodiment, please refer to the fruit counting method provided in any embodiment of the present invention, which will not be repeated here.
[0178] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0179] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0181] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method for counting fruits, characterized in that, The fruit counting method includes: Obtain the image of the first fruit; The first fruit image is input into the VGG-16 network of the first counting model, and multiple convolutions and pooling are performed to obtain the first feature information; Local feature extraction is performed on the first feature information to obtain several blocks of local feature information, wherein the several blocks of local feature information include a first block of local feature information, a second block of local feature information, and a third block of local feature information; The block local feature information is transformed and crossed according to the first feature information to obtain several contrast feature information, and second feature information is obtained according to the several contrast features and the first feature information. The transformation is used to obtain block local feature information of the same size as the first feature information, and the crossing refers to performing a product operation. The second feature information is input into the classification network of the first counting model to obtain the number of fruits in the first fruit image; The step of transforming and cross-referencing the several block local feature information according to the first feature information to obtain several contrast features, and obtaining second feature information according to the several contrast feature information and the first feature information, includes: The first block of local feature information, the second block of local feature information, and the third block of local feature information are respectively input into the upsampling layer to convert the first block of local feature information, the second block of local feature information, and the third block of local feature information into the size of the first feature information, and obtain new first block of local feature information, new second block of local feature information, and new third block of local feature information. Based on the differences between the new first block of local feature information, the new second block of local feature information, and the new third block of local feature information and the first feature information, respectively, a first difference feature, a second difference feature, and a third difference feature are obtained; The first contrast feature, the second contrast feature, and the third contrast feature are obtained by determining the product of the first local feature information and the first difference feature, the product of the second local feature information and the second difference feature, and the product of the third local feature information and the third difference feature. The first feature information, the first contrast feature, the second contrast feature, and the third contrast feature are superimposed to obtain the second feature information.
2. The method as described in claim 1, characterized in that, The VGG-16 network includes a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, a max pooling layer, and a fourth convolutional layer connected in sequence. The first fruit image is input into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information, including: The first fruit image is sequentially input into the first convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the first preset number of channels is output. The feature information of the first preset number of channels is sequentially input into the second convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the second preset number of channels is output, wherein the second preset number is greater than the first preset number; The feature information of the second preset number of channels is sequentially input into the third convolutional layer and the max pooling layer of the VGG-16 network, and the feature information of the third preset number of channels is output, wherein the third preset number is greater than the second preset number; The feature information of the third preset number of channels is sequentially input into the fourth convolutional layer of the VGG-16 network, and the feature information of the fourth preset number of channels is output. The feature information of the fourth preset number of channels is the first feature information, and the fourth preset number is greater than the third preset number.
3. The method as described in claim 1, characterized in that, The step of extracting local features from the first feature information to obtain several blocks of local feature information includes: The first feature information is sequentially input into the first average pooling layer and the target convolutional layer to obtain the first block of local feature information, wherein the target convolutional layer is a convolutional layer with a kernel of 1; The first feature information is sequentially input into the second average pooling layer and the target convolutional layer to obtain the second block of local feature information; The first feature information is sequentially input into the third average pooling layer and the target convolutional layer to obtain the third block of local feature information.
4. The method as described in claim 1, characterized in that, Before inputting the first fruit image into the VGG-16 network of the first counting model for multiple convolutions and pooling, the process further includes: Obtain a second fruit image, wherein the true center point of each fruit is marked in the second fruit image; The second fruit image is input into the VGG-16 network of the second counting model to obtain the first target feature information; Based on the contextual method, the first feature information is processed to obtain the second target feature information; The second target feature information is input into the classification network of the second counting model to obtain the predicted center point in the second fruit image; The predicted center points are paired with the true center points based on the Hungarian algorithm, and a loss function is established. The second counting model is updated based on the loss function to obtain the first counting model.
5. The method as described in claim 4, characterized in that, The process of pairing the predicted center points with the true center points based on the Hungarian algorithm and establishing a loss function includes: The predicted center point and the true center point are paired based on the Hungarian algorithm to obtain the predicted center point and the true center point of each fruit. The Euclidean distance difference of each fruit in the second fruit image is determined based on the predicted center point and the true center point. Based on the Euclidean distance difference, a loss function is established.
6. The method according to any one of claims 1 to 5, characterized in that, The step of inputting the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image includes: The second feature information is input into the classification network of the first counting model to determine the probability value of each pixel in the first fruit image; The number of fruits in the first fruit image is determined based on the probability value.
7. A fruit counting device, characterized in that, The fruit counting device includes: The acquisition module is used to acquire the image of the first fruit; The acquisition module is used to input the first fruit image into the VGG-16 network of the first counting model for multiple convolutions and pooling to obtain the first feature information; The acquisition module is used to extract local features from the first feature information to obtain several blocks of local feature information, wherein the several blocks of local feature information include a first block of local feature information, a second block of local feature information, and a third block of local feature information. The acquisition module is used to transform and cross the several block local feature information according to the first feature information to obtain several contrast feature information, and to obtain second feature information according to the several contrast features and the first feature information. The transformation is used to obtain block local feature information of the same size as the first feature information, and the cross refers to performing a product operation. The counting module is used to input the second feature information into the classification network of the first counting model to obtain the number of fruits in the first fruit image; The acquisition module is used for: The first block of local feature information, the second block of local feature information, and the third block of local feature information are respectively input into the upsampling layer to convert the first block of local feature information, the second block of local feature information, and the third block of local feature information into the size of the first feature information, and obtain new first block of local feature information, new second block of local feature information, and new third block of local feature information. Based on the differences between the new first block of local feature information, the new second block of local feature information, and the new third block of local feature information and the first feature information, respectively, a first difference feature, a second difference feature, and a third difference feature are obtained; The first contrast feature, the second contrast feature, and the third contrast feature are obtained by determining the product of the first local feature information and the first difference feature, the product of the second local feature information and the second difference feature, and the product of the third local feature information and the third difference feature. The first feature information, the first contrast feature, the second contrast feature, and the third contrast feature are superimposed to obtain the second feature information.
8. A fruit counting device, characterized in that, The device includes: a memory, a processor, and a fruit counting program stored in the memory and executable on the processor, the fruit counting program being configured to implement the steps of the fruit counting method as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium stores a fruit counting program, which, when executed by a processor, implements the steps of the fruit counting method as described in any one of claims 1 to 6.