Fruit counting method, device, equipment and medium

Abstract

The present application provides a fruit counting method, apparatus, device and medium, wherein the method comprises: obtaining a foreground image corresponding to a target fruit tree in an orchard captured by a camera; inputting the foreground image into a target detection model to obtain an object detection sequence; clustering all detection objects in the foreground image based on the scale information of each detection object to obtain a first cluster and a second cluster; determining a target cluster from the first cluster and the second cluster, and determining the number of detection objects included in the target cluster as the number of fruits on the target fruit tree. The present application obtains a target image and detects and clusters the scale information of the detection objects in the image, and performs depth estimation on the fruit objects in the foreground image based on the scale differences between different fruits, so as to eliminate the interference of the fruit trees in the back row on the fruit counting.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a fruit counting method, device, equipment, and medium. Background Art

[0002] In the field of agricultural production technology, accurate crop yield statistics are crucial. Counting fruit during the crop planting process, and subsequently predicting crop yields, is crucial for guiding orchard management and developing crop planting and marketing strategies.

[0003] Currently, there are three methods for counting crop fruit: the first uses remote sensing to analyze citrus yield over a large area using remote sensing spectra; the second uses a growth model for citrus trees, incorporating factors such as soil and climate, to indirectly predict fruit yield; and the third uses computer vision techniques to directly obtain images of fruit trees and directly determine the number of citrus fruits. Of these three methods, remote sensing is suitable for overall estimation of citrus yield over a large area, while fruit tree growth models require specialized agricultural knowledge to construct and predict. Neither method can accurately determine the number of fruits in an orchard.

[0004] However, computer vision counting often involves capturing the fruit from trees in the back row of trees in the frame due to the camera's field of view. This creates a background that interferes with the count of the target fruit in the foreground, making it difficult for the computer vision counting system to accurately count the fruit. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a fruit counting method, device, equipment and medium, which can solve the problem in the prior art that the number of fruits cannot be accurately counted due to the interference of the background image by detecting and clustering the scale information of the detection objects in the foreground image, and achieve the effect of accurately identifying the number of fruits on the target fruit tree.

[0006] In a first aspect, an embodiment of the present application provides a fruit counting method, the method comprising: obtaining a foreground image corresponding to a target fruit tree in an orchard captured by a camera, wherein a plurality of fruit trees arranged in multiple rows are planted in the orchard, and the foreground image includes the target fruit tree and other fruit trees, and the rows where the other fruit trees are located are different from the row where the target fruit tree is located; inputting the foreground image into a target detection model to obtain an object detection sequence, the object detection sequence comprising a plurality of detection items, each detection item corresponding to a detection object in the foreground image, and each detection item including scale information of the corresponding detection object; clustering all detection objects in the foreground image based on the scale information of each detection object to obtain a first cluster cluster and a second cluster cluster; determining a target cluster cluster from the first cluster cluster and the second cluster cluster, and determining the number of detection objects included in the target cluster cluster as the number of fruits on the target fruit tree.

[0007] Optionally, at least one track is arranged in the orchard, and the at least one track is arranged parallel to each row of fruit trees. A pulley is arranged on each track, and the pulley can move along the corresponding track. The camera is arranged on the pulley. The target fruit tree is a fruit tree in a row of fruit trees that is closest to the track where the camera is located within the shooting range of the camera, and the other fruit trees are fruit trees in other rows located behind the target fruit tree.

[0008] Optionally, the scale information includes an image size characteristic value occupied by the detection object in the foreground image, wherein, based on the scale information of each detection object, all detection objects in the foreground image are clustered, and the step of obtaining the first cluster cluster and the second cluster cluster includes: identifying the maximum image size characteristic value of all detection objects in the foreground image; for each detection object, calculating the ratio of the image size characteristic value corresponding to the detection object to the maximum image size characteristic value; for each detection object, mapping the ratio corresponding to the detection object to obtain the enhanced characteristic value of the detection object; clustering all detection objects according to the enhanced characteristic values ​​of all detection objects to obtain the first cluster cluster and the second cluster cluster.

[0009] Optionally, the step of determining a target cluster from the first cluster and the second cluster includes: determining a first eigenvalue of the cluster center of the first cluster based on the enhanced eigenvalue corresponding to the detection object in the first cluster; determining a second eigenvalue of the cluster center of the second cluster based on the enhanced eigenvalue corresponding to the detection object in the second cluster; determining the maximum value between the first eigenvalue and the second eigenvalue and determining the cluster corresponding to the maximum value as the target cluster.

[0010] Optionally, the target detection model includes a backbone network and a detection output network, and the backbone network includes at least three residual modules. The step of inputting the foreground image into the target detection model to obtain an object detection sequence includes: scaling the foreground image to obtain a first image that meets the standard image size; inputting the first image into the backbone network and processing it in turn through the multiple residual modules; performing standard data processing on the first output vector output by the last residual module to obtain a first feature map; upsampling the first feature map, and splicing the upsampled first feature map with the second output vector output by the second-to-last residual module to obtain a third output vector; performing standard data processing on the third output vector to obtain a second feature map; upsampling the second feature map, and splicing the upsampled second feature map with the fourth output vector output by the third-to-last residual module to obtain a fifth output vector; performing standard data processing on the fifth output vector to obtain a third feature map; and obtaining an object detection sequence based on the first feature map, the second feature map and the third feature map.

[0011] Optionally, the step of obtaining an object detection sequence based on the first feature map, the second feature map and the third feature map includes: for each feature map, performing standard grid division on the feature map to obtain multiple grid images; for each grid image, performing the following processing: calculating the predicted probability value of detecting the detection object in the grid image, if the predicted probability value corresponding to the grid image is greater than the probability threshold, then generating a detection frame at the corresponding position of the feature map corresponding to the grid image; performing non-maximum suppression processing on all detection frames in the three feature maps to remove duplicate detection frames; and obtaining an object detection sequence based on the position information of each detection frame after the non-maximum suppression processing and the scale information of the detection object in each detection frame.

[0012] Optionally, the target detection model is trained in the following manner: obtaining a training foreground image corresponding to a fruit tree of the same variety as the target fruit tree captured by a camera; marking multiple fruit positions of the fruit tree in the training foreground image; performing standard grid division on the training foreground image to obtain multiple standard grid images; for each standard grid image, inputting the standard grid image into the target detection model to obtain a first training output vector of the standard grid image; for each standard grid image, determining whether the fruit position corresponding to the standard grid image in the training foreground image is marked, and if marked, setting the prediction probability value corresponding to the standard grid image to 1 to obtain a second training output vector of the standard grid image; for each standard grid image, obtaining a loss function value of the standard grid image based on the first training output vector and the second training output vector of the standard grid image; according to the obtained loss function value, training the initial detection model until the loss function value of the initial detection model converges to a minimum value; the initial detection model when the loss function value converges to the minimum value is determined as the trained target detection model.

[0013] In a second aspect, an embodiment of the present application further provides a fruit counting device, which includes: a foreground image acquisition module for acquiring a foreground image corresponding to a target fruit tree in an orchard captured by a camera, wherein the orchard is planted with a plurality of fruit trees arranged in multiple rows, and the foreground image includes the target fruit tree and other fruit trees, and the row where the other fruit trees are located is different from the row where the target fruit tree is located; a detection sequence acquisition module for inputting the foreground image into a target detection model to obtain an object detection sequence, wherein the object detection sequence includes multiple detection items, each detection item corresponds to a detection object in the foreground image, and each detection item includes scale information of the corresponding detection object; a detection object clustering module for clustering all detection objects in the foreground image based on the scale information of each detection object to obtain a first cluster cluster and a second cluster cluster; a fruit number determination module for determining a target cluster cluster from the first cluster cluster and the second cluster cluster, and determining the number of detection objects included in the target cluster cluster as the number of fruits on the target fruit tree.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned fruit counting method are performed.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned fruit counting method are executed.

[0016] The fruit counting method, apparatus, device, and medium provided in the embodiments of this application calculate the number of fruits on a target fruit tree by detecting and clustering the scale information of detection objects in a foreground image. Compared with existing fruit counting methods, this method solves the problem of accurate fruit counting due to interference from background images.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a fruit counting method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the camera layout and fruit tree arrangement in an orchard provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of an object detection model provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a foreground image provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of a fruit counting device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0026] First, the application scenarios to which this application is applicable are introduced. This application can be applied to counting fruits on fruit trees.

[0027] Research has found that when counting fruit trees using computer vision, due to camera field of view issues, when the camera captures the target tree, the fruit from trees in the back row often ends up in the frame to ensure the entire tree is captured. This background interference interferes with the count of the target fruit in the foreground, making it difficult for the computer vision counting system to accurately count the fruit on the tree.

[0028] Based on this, the embodiments of the present application provide a fruit counting method, device, equipment and medium to achieve the effect of accurately identifying the number of fruits on the target fruit tree.

[0029] See also Figure 1 , Figure 1 This is a flow chart of a fruit counting method provided in an embodiment of the present application. Figure 1 As shown in , the fruit counting method provided in the embodiment of the present application includes:

[0030] S101: Acquire a foreground image corresponding to a target fruit tree in an orchard captured by a camera.

[0031] There are multiple fruit trees arranged in multiple rows in the orchard, and the foreground image includes the target fruit tree and other fruit trees, and the row where the other fruit trees are located is different from the row where the target fruit tree is located.

[0032] Specifically, such as Figure 2 As shown in FIG, a plurality of fruit trees 206 are planted in an orchard and arranged in multiple rows.

[0033] It should be noted that if Figure 2As shown in , at least one track 201 is arranged in the orchard, and at least one track 201 is arranged parallel to each row of fruit trees. A pulley 202 is arranged on each track, and the pulley 202 can move along the corresponding track. The camera 203 is arranged on the pulley.

[0034] Specifically, the camera 203 provided on the trolley 202 can be disassembled according to the need to collect images of fruit trees and installed on another trolley (not shown in the figure).

[0035] The target fruit tree 204 is a fruit tree in a row of fruit trees that is closest to the track 201 where the camera 203 is located within the shooting range 205 of the camera 203, and the other fruit trees 206 are fruit trees in other rows located behind the target fruit tree.

[0036] In this way, the camera 203 can capture a foreground image including the target fruit tree 204 and other fruit trees 206 .

[0037] For example, the foreground image captured by the camera is as follows: Figure 4 As shown in FIG, the foreground image includes a target fruit tree 401 and other fruit trees 402 in a row different from that of the target fruit tree 401.

[0038] S102: Input the foreground image into the target detection model to obtain an object detection sequence.

[0039] The object detection sequence includes a plurality of detection items, each detection item corresponds to a detection object in the foreground image, and each detection item includes scale information of the corresponding detection object.

[0040] Specifically, the scale information includes: a characteristic value of the image size occupied by the detection object in the foreground image.

[0041] Here, the scale information includes information that may indicate the area occupied by the detected object in the foreground image.

[0042] For example, the detection object may be a citrus fruit, which is round. The scale information in the detection item for the citrus fruit may include radius size information of the citrus fruit. For example, the object detection sequence for the citrus fruit may be:

[0043] C K =[(X0, Y0, R0), (X1, Y1, R1)…(X K , Y K , R K )]

[0044] Among them, (X K , Y K ) represents the coordinate position of the kth citrus in the foreground image, R Krepresents the radius size of the kth citrus, and k is the total number of citrus fruits included in the foreground image.

[0045] Specifically, the target detection model includes a backbone network and a detection output network, and the backbone network includes at least three residual modules.

[0046] The foreground image is input into the target detection model, and the steps of obtaining the object detection sequence include: scaling the foreground image to obtain a first image that meets the standard image size; inputting the first image into the backbone network and processing it in sequence through the multiple residual modules; performing standard data processing on the first output vector output by the last residual module to obtain a first feature map; upsampling the first feature map, and concatenating the upsampled first feature map with the second output vector output by the second-to-last residual module to obtain a third output vector; performing standard data processing on the third output vector to obtain a second feature map; upsampling the second feature map, and concatenating the upsampled second feature map with the fourth output vector output by the third-to-last residual module to obtain a fifth output vector; performing standard data processing on the fifth output vector to obtain a third feature map; and obtaining the object detection sequence based on the first feature map, the second feature map, and the third feature map.

[0047] The target detection model may perform a reduction or enlargement operation on the foreground image so as to adjust the size of the foreground image to a first image that meets the standard image size.

[0048] For example, Figure 3 As shown in , the backbone network may include five residual modules. After the foreground image is input into the target detection model, it is processed by the five residual modules in sequence after image processing 301. The target detection model performs standard data processing DBL on the first output vector output by the fifth residual module res5 to obtain a first feature map 304 with a dimension of 13×13×4.

[0049] The first feature map 304 is upsampled 302, and the first feature map 304 after the upsampling 302 is spliced ​​303 with the second output vector output by the fourth residual module res4 to obtain a third output vector. After the third output vector is subjected to standard data processing DBL, a second feature map 305 with a dimension of 26×26×4 is obtained.

[0050] The second feature map 305 is upsampled 302, and the upsampled second feature map 305 is concatenated 303 with the fourth output vector output by the third residual module res3 to obtain a fifth output vector. After standard data processing DBL is performed on the fifth output vector, a third feature map with a dimension of 52×52×4 is obtained.

[0051] The standard data processing (DBL) method includes convolution, batch normalization, and activation function (Leaky_Relu) processing on the output vector. Upsampling can amplify the feature map.

[0052] Specifically, the step of obtaining an object detection sequence based on the first feature map, the second feature map, and the third feature map includes: for each feature map, performing standard grid division on the feature map to obtain multiple grid images; for each grid image, performing the following processing: calculating a predicted probability value of detecting a detection object in the grid image, and if the predicted probability value corresponding to the grid image is greater than a probability threshold, generating a detection frame at a corresponding position of the feature map corresponding to the grid image; performing non-maximum suppression processing on all detection frames in the three feature maps to remove duplicate detection frames; and obtaining an object detection sequence based on the position information of each detection frame after the non-maximum suppression processing and the scale information of the detection object in each detection frame.

[0053] After the predicted probability value corresponding to the grid image is calculated, an output vector of the grid can be output.

[0054] For example, when the detected object is a circle, the output vector may be:

[0055] Among them, i∈s 2 is the grid number. For example, the network image is divided into 64 grids of 8×8. At this time, s=8, i∈[1,64], j∈[1,2,3] is the feature map number. The position x of the detection object relative to the center of the network in grid i, j i，j ,y i，j The prediction offset of The position x of the detection object relative to the center of the network in grid i, j i，j ,y i，j The prediction offset of The circular feature scale r of the detection object is the prefabricated circular feature scale relative to grid i, j i，j The prediction offset of is the predicted probability value of detecting the detection object in grid i, j.

[0056] When performing inference operations, when the feature vectors in the i-th and j-th grids If it is greater than 0.5, a circular detection frame is generated on the foreground image (such as Figure 4 403), the center of which is Its radius is

[0057] like Figure 4As shown in , if the three feature maps detect a total of n Then generate n circular detection frames on the foreground image (such as Figure 4 As shown in 403), for these n circular detection frames (as Figure 4 403 in the figure) to perform non-maximum suppression processing, and the repeated circular detection frame in the feature map (such as Figure 4 403) to obtain k circular detection frames (as shown in Figure 4 The object sequence shown in 403) may be:

[0058] C K =[(X0, Y0, R0), (X1, Y1, R1)…(X K , Y K , R K )]

[0059] In this way, the foreground image can be used to calculate the target detection graph to obtain the object detection sequence.

[0060] S103 : Clustering all the detected objects in the foreground image based on the scale information of each detected object to obtain a first cluster and a second cluster.

[0061] Specifically, the scale information includes: a characteristic value of the image size occupied by the detection object in the foreground image.

[0062] Here, the scale information includes information that may indicate the area occupied by the detected object in the foreground image.

[0063] For example, the detection object may be citrus, which is round in shape. The scale information in the detection item for citrus may include radius size information of the citrus.

[0064] Among them, based on the scale information of each detection object, all detection objects in the foreground image are clustered to obtain the first cluster cluster and the second cluster cluster, which includes: identifying the maximum image size eigenvalue of all detection objects in the foreground image; for each detection object, calculating the ratio of the image size eigenvalue corresponding to the detection object to the maximum image size eigenvalue; for each detection object, performing a mapping transformation on the ratio corresponding to the detection object to obtain the enhanced eigenvalue of the detection object; clustering all detection objects according to the enhanced eigenvalues ​​of all detection objects to obtain the first cluster cluster and the second cluster cluster.

[0065] For example, when the detection object is citrus, the R value corresponding to each detection frame can be obtained through the above object sequence, and the maximum image feature value R among all the detection objects can be identified. max , according to the image size feature R of each detected object K, calculate the ratio of the image size characteristic value to the maximum image size characteristic value in, The calculation formula can be: In obtaining the ratio of each detection object Then, the ratio corresponding to each detection object Perform mapping transformation to obtain the enhanced feature value of the detection object in, The calculation formula can be:

[0066] In this way, the difference between the enhanced feature values ​​obtained after enhancement is larger, and it is easier to distinguish them during clustering.

[0067] After obtaining the enhanced feature value corresponding to the detection object, all detection objects are clustered to obtain the first cluster cluster and the second cluster cluster. For example, clustering can be performed according to the enhanced feature value, and it can be set according to actual conditions that those greater than the clustering threshold belong to the first cluster cluster, and those less than or equal to the clustering threshold belong to the second cluster cluster, or the detection objects can be divided into two cluster clusters according to the segmentation of the enhanced feature value.

[0068] In this way, the detected fruits can be clustered according to the changes in fruit size between the target fruit tree in the foreground image and other fruit trees in the back row.

[0069] S104: Determine a target cluster from the first cluster and the second cluster, and determine the number of detection objects included in the target cluster as the number of fruits on the target fruit tree.

[0070] Specifically, the step of determining the target cluster from the first cluster and the second cluster includes: determining the first eigenvalue of the cluster center of the first cluster according to the enhanced eigenvalue corresponding to the detection object in the first cluster; determining the second eigenvalue of the cluster center of the second cluster according to the enhanced eigenvalue corresponding to the detection object in the second cluster; determining the maximum value between the first eigenvalue and the second eigenvalue; and determining the cluster corresponding to the maximum value as the target cluster.

[0071] For example, when the detection object is citrus, the first eigenvalue of the cluster center of the first cluster can be calculated by finding the average value of all enhanced eigenvalues ​​in the first cluster. The second eigenvalue of the cluster center of the second cluster is calculated by averaging all the enhanced eigenvalues ​​in the second cluster. Determine the first eigenvalue and the second eigenvalue For example, the maximum of the two eigenvalues ​​above is the second eigenvalue The second cluster is determined as the target cluster.

[0072] In this way, after the target cluster is determined, the number of fruits on the target fruit tree can be determined based on the number of samples in the target cluster.

[0073] Optionally, the target detection model can be trained in the following manner: obtaining a training foreground image corresponding to a fruit tree of the same variety as the target fruit tree captured by a camera; marking multiple fruit positions of the fruit tree in the training foreground image; performing standard grid division on the training foreground image to obtain multiple standard grid images; for each standard grid image, inputting the standard grid image into the target detection model to obtain a first training output vector of the standard grid image; for each standard grid image, determining whether the fruit position corresponding to the standard grid image in the training foreground image is marked, and if marked, setting the prediction probability value corresponding to the standard grid image to 1 to obtain a second training output vector of the standard grid image; for each standard grid image, obtaining the loss function value of the standard grid image based on the first training output vector and the second training output vector of the standard grid image; according to the obtained loss function value, training the initial detection model until the loss function value of the initial detection model converges to a minimum value; the initial detection model when the loss function value converges to the minimum value is determined as the trained target detection model.

[0074] For example, the loss function can be:

[0075] Loss = l center +l r +l obj

[0076]

[0077]

[0078]

[0079] Where Δx i，j , Δy i，j To detect the position x of the target in grid i, j relative to the center of the grid i，j ,y i，j The actual offset; Δr i，j The circular feature scale r of the detected target is the prefabricated circular feature scale relative to grid i, j i，j The actual offset of p i，j is the probability value of detecting the target in grid i, j. When the target is detected in grid i, j, p i，j =1, otherwise p i，j =0;l centerRepresents the center term of the loss function, which is used to calculate the predicted detection target center offset The loss value of r The r term of the loss function is used to calculate the target feature scale Δr for regression i，j The loss value of obj The obj item represents the loss function, which is used to calculate the probability value of the detection target in the regression calculation network Loss items; obj Represents the weight of the obj loss function, which is determined by the actual task; Indicates: 1 if there is a target in the grid number i, j, otherwise 0; λ center Represents: the weight coefficient of the center term of the loss function, which is determined according to the actual task.

[0080] Compared with the fruit counting methods in the prior art, the fruit counting method provided in the embodiments of the present application solves the problem in the prior art of being unable to accurately count the number of fruits due to interference from the background image by detecting and clustering the scale information of the detection objects in the foreground image, thereby achieving the effect of accurately identifying the number of fruits on the target fruit tree.

[0081] See also Figure 5 , Figure 5 This is a structural diagram of a fruit counting device provided in an embodiment of the present application. Figure 5 As shown in , the fruit counting device 500 includes:

[0082] The foreground image acquisition module 501 is used to obtain a foreground image corresponding to a target fruit tree in an orchard captured by a camera, wherein the orchard is planted with multiple fruit trees arranged in multiple rows, and the foreground image includes the target fruit tree and other fruit trees, and the row where the other fruit trees are located is different from the row where the target fruit tree is located.

[0083] The detection sequence acquisition module 502 is used to input the foreground image into the target detection model to obtain an object detection sequence, where the object detection sequence includes multiple detection items, each detection item corresponds to a detection object in the foreground image, and each detection item includes scale information of the corresponding detection object.

[0084] The detection object clustering module 503 is configured to cluster all detection objects in the foreground image based on scale information of each detection object to obtain a first cluster and a second cluster.

[0085] The fruit quantity determination module 504 is configured to determine a target cluster from the first cluster and the second cluster, and determine the quantity of detection objects included in the target cluster as the quantity of fruits on the target fruit tree.

[0086] The fruit counting device provided in the embodiment of the present application solves the problem in the prior art of being unable to accurately count the number of fruits due to interference from the background image by detecting and clustering the scale information of the detection objects in the foreground image, thereby achieving the effect of accurately identifying the number of fruits on the target fruit tree.

[0087] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in FIG, the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .

[0088] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 via the bus 630. When the machine-readable instructions are executed by the processor 610, the above-mentioned Figure 1 The specific implementation of the steps of the fruit counting method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0089] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the fruit counting method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0090] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0094] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0095] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A fruit counting method, characterized in that: The method comprises: Acquire a foreground image corresponding to a target fruit tree in an orchard captured by a camera, wherein the orchard includes a plurality of fruit trees arranged in multiple rows, the foreground image including the target fruit tree and other fruit trees, and the rows where the other fruit trees are located are different from the row where the target fruit tree is located; Inputting the foreground image into an object detection model to obtain an object detection sequence, the object detection sequence including a plurality of detection items, each detection item corresponding to a detection object in the foreground image, each detection item including scale information of the corresponding detection object, the scale information including an image size feature value of the detection object in the foreground image; Clustering all detected objects in the foreground image based on scale information of each detected object to obtain a first cluster and a second cluster; A target cluster is determined from the first cluster and the second cluster, and the number of detection objects included in the target cluster is determined as the number of fruits on the target fruit tree.

2. The method according to claim 1, characterized in that At least one track is arranged in the orchard, and the at least one track is arranged parallel to each row of fruit trees. A pulley is arranged on each track, and the pulley can move along the corresponding track. The camera is arranged on the pulley. The target fruit tree is a fruit tree in a row of fruit trees that is closest to the track where the camera is located within the shooting range of the camera, and the other fruit trees are fruit trees in other rows that are located behind the target fruit tree.

3. The method according to claim 1, characterized in that The step of clustering all the detected objects in the foreground image based on the scale information of each detected object to obtain a first cluster and a second cluster comprises: Identify the maximum image size feature value of all detection objects in the foreground image; For each detection object, calculating the ratio of the image size characteristic value corresponding to the detection object to the maximum image size characteristic value; For each detection object, a mapping transformation is performed on the ratio corresponding to the detection object to obtain an enhanced feature value of the detection object; All detection objects are clustered according to their enhanced feature values ​​to obtain a first cluster and a second cluster.

4. The method according to claim 3, characterized in that The step of determining a target cluster from the first cluster and the second cluster includes: Determining a first eigenvalue of a cluster center of the first cluster according to the enhanced eigenvalue corresponding to the detection object in the first cluster; Determining a second eigenvalue of the cluster center of the second cluster according to the enhanced eigenvalue corresponding to the detection object in the second cluster; determining the maximum of the first eigenvalue and the second eigenvalue; The cluster corresponding to the maximum value is determined as the target cluster.

5. The method according to claim 1, characterized in that The target detection model includes a backbone network and a detection output network, wherein the backbone network includes at least three residual modules. The step of inputting the foreground image into the target detection model to obtain the object detection sequence includes: Scaling the foreground image to obtain a first image that meets a standard image size; Inputting the first image into the backbone network and processing it sequentially through the multiple residual modules; Performing standard data processing on the first output vector output by the last residual module to obtain a first feature map; Upsampling the first feature map, and concatenating the upsampled first feature map with the second output vector output by the penultimate residual module to obtain a third output vector; Performing standard data processing on the third output vector to obtain a second feature map; Upsampling the second feature map, and concatenating the upsampled second feature map with the fourth output vector output by the third-to-last residual module to obtain a fifth output vector; Performing standard data processing on the fifth output vector to obtain a third feature map; An object detection sequence is obtained according to the first feature map, the second feature map, and the third feature map.

6. The method according to claim 5, characterized in that The step of obtaining an object detection sequence according to the first feature map, the second feature map, and the third feature map includes: For each feature map, perform standard grid division on the feature map to obtain multiple grid images; For each grid image, the following processing is performed: the predicted probability value of detecting the detection object in the grid image is calculated. If the predicted probability value corresponding to the grid image is greater than the probability threshold, a detection box is generated at the corresponding position of the feature map corresponding to the grid image; Perform non-maximum suppression on all detection frames in the three feature maps to remove duplicate detection frames; An object detection sequence is obtained based on the position information of each detection frame after non-maximum suppression processing and the scale information of the detection object in each detection frame.

7. The method according to claim 1, characterized in that The object detection model is trained in the following way: Obtaining a training foreground image corresponding to a fruit tree of the same variety as the target fruit tree, captured by a camera; Marking multiple fruit positions of the fruit tree in the training foreground image; Performing standard grid division on the training foreground image to obtain a plurality of standard grid images; For each standard grid image, input the standard grid image into the object detection model to obtain a first training output vector of the standard grid image; For each standard grid image, determining whether a fruit position corresponding to the standard grid image is marked in the training foreground image; if so, setting the prediction probability value corresponding to the standard grid image to 1 to obtain a second training output vector for the standard grid image; For each standard grid image, obtaining a loss function value of the standard grid image based on the first training output vector and the second training output vector of the standard grid image; Training the initial detection model according to the obtained loss function value until the loss function value of the initial detection model converges to a minimum value; The initial detection model when the loss function value converges to the minimum value is determined as the trained target detection model.

8. A fruit counting device, characterized in that: The device comprises: a foreground image acquisition module, configured to acquire a foreground image corresponding to a target fruit tree in an orchard captured by a camera, wherein the orchard includes a plurality of fruit trees arranged in a plurality of rows, the foreground image including the target fruit tree and other fruit trees, the rows where the other fruit trees are located being different from the row where the target fruit tree is located; a detection sequence acquisition module, configured to input the foreground image into an object detection model to obtain an object detection sequence, wherein the object detection sequence includes a plurality of detection items, each detection item corresponding to a detection object in the foreground image, and each detection item includes scale information of the corresponding detection object, wherein the scale information includes an image size feature value of the detection object in the foreground image; a detection object clustering module, configured to cluster all detection objects in the foreground image based on scale information of each detection object to obtain a first cluster and a second cluster; The fruit quantity determination module is used to determine a target cluster from the first cluster and the second cluster, and determine the quantity of detection objects included in the target cluster as the quantity of fruits on the target fruit tree.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of any one of the methods described in claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images