Automatic identification method for center flower and side flower of fruit tree, electronic equipment and storage medium

Abstract

The invention provides an automatic identification method for a central flower and an edge flower of a fruit tree, and the method comprises the steps: firstly constructing and training a target detection model to recognize an inflorescence and / or flower target in an image, and obtaining the coordinates of a flower detection frame and the coordinates of a central pixel; aiming at inflorescence affiliation division, mapping a flower center coordinate into a pre-detected inflorescence frame to realize association; or adopting a clustering algorithm to perform unsupervised clustering on the flower detection frames by taking the center coordinates as features, and dynamically dividing inflorescence categories; in a central flower determination link, two technical paths are provided: calculating a minimum boundary distance average value of each flower detection frame and other frames in the same inflorescence based on geometric features, and selecting a flower corresponding to the minimum value as a central flower; or a relation prediction model is constructed through a scene graph generation technology, and the center flower and the edge flower are determined based on position relation reasoning. Finally, the inflorescence, the central flower and the side flower are distinguished through visual labeling. The invention further provides corresponding electronic equipment and a storage medium. Therefore, target detection, spatial relation analysis and machine learning technologies are fused, and automatic and accurate identification of the inflorescence structure of the fruit tree is realized.

Description

Technical Field

[0001] The invention relates to the technical field of intelligent flower thinning for fruit trees, and in particular to an automatic identification method for central flowers and side flowers of fruit trees, electronic equipment and storage medium. Background Art

[0002] Flower thinning is a key agronomic measure. Reasonable flower thinning can effectively adjust the load of fruit trees, help maintain the health of the trees, ensure sufficient nutrient supply during subsequent fruit growth, and thus improve fruit quality.

[0003] Apple inflorescences generally contain one central flower and multiple side flowers. During flower thinning operations, the central flower in the inflorescence is usually retained first, and the side flowers are thinned out. Manual flower thinning is currently a common method used in my country's fruit-growing areas, but this method has a huge demand for labor and high labor costs; the effect of chemical flower thinning is difficult to predict and will cause pollution to environmental factors such as soil, water sources and air; mechanical flower thinning is currently usually non-selective and cannot distinguish between central flowers and side flowers in the inflorescence of fruit trees, and cannot achieve refined flower thinning. Therefore, in the context of artificial intelligence empowerment, flower thinning robots need to accurately and efficiently identify central flowers and side flowers in apple inflorescences to ensure that the robots complete targeted flower thinning tasks.

[0004] Among the existing related methods, Patent 202110969938.4 provides a method and system for identifying and locating central flowers based on regional attribute extraction, which realizes the identification of central flowers through a combination of a series of traditional image processing algorithms. However, traditional image processing algorithms are too dependent on factors such as color and texture, and are easily affected by ambient lighting. In addition, this method does not involve the identification of side flowers. Patent 202010564264.5 provides a method for identifying and obtaining the location of fruit tree flowers. The location and category of the central flowers and side flowers of a single inflorescence are marked in the data set, and a deep convolutional neural network is used to train a flower detection model. , to predict the central flower and side flowers in the inflorescence. However, the central flower and side flowers in the inflorescence sometimes present the same color, shape, texture and other features, and the shapes and relative positions of the central flower and side flowers presented at different visual angles are also different. If the flower detection model trained by the deep convolutional neural network is directly used to predict the central flower and side flowers, the flower detection model will only consider the characteristics of the central flower or side flower itself, and will not pay attention to the relative position relationship between the central flower and the side flowers. When the central flower and the side flowers present the same color, shape, texture and other features, the flower detection model's prediction ability for the central flower and the side flowers may fail.

[0005] Therefore, how to overcome the practical technical difficulties in identifying the above-mentioned central flowers and side flowers has become an important issue that needs to be urgently solved in the field of intelligent flower thinning of fruit trees. Summary of the invention

[0006] Aiming at the above-mentioned defects, the purpose of the present invention is to provide an automatic recognition method, an electronic device and a storage medium for the central flower and marginal flowers of fruit trees, so as to realize the automatic and accurate recognition of the inflorescence structure of fruit trees.

[0007] In order to achieve the above technical effects, in the first aspect, the present invention provides an automatic recognition method for the central flower and marginal flowers of fruit trees, including the steps of:

[0008] Construct and train a target detection model for inflorescences and flowers, which is used to detect both inflorescence targets and flower targets in an image simultaneously;

[0009] Detect the image to be measured through the target detection model, obtain the detection box coordinates of all inflorescences and flowers in the image to be measured, and calculate the central pixel coordinates of each flower;

[0010] Map the central pixel coordinates into the inflorescence detection box, and according to the inflorescence detection box where the central pixel coordinates are located, attribute the corresponding flower to the inflorescence corresponding to the inflorescence detection box;

[0011] For all the flowers in the same inflorescence, calculate the average value of the minimum boundary distances between each flower detection box and other flower detection boxes respectively, and determine the flower with the minimum average value as the central flower of the inflorescence, and the remaining flowers as marginal flowers;

[0012] Visually annotate the detection boxes corresponding to the inflorescences, the central flowers and the marginal flowers in the image to be measured, and display them separately with different identifiers.

[0013] Optionally, the constructing and training the target detection model for inflorescences and flowers includes:

[0014] Construct a data set, which includes a number of training images, and each training image includes inflorescences and flowers;

[0015] Mark bounding boxes for the inflorescences and flowers in the training images in the data set respectively; among them, each flower in the inflorescence is independently marked as the flower category, and the inflorescence as a whole is marked as the inflorescence category;

[0016] Use the marked data set to train a deep learning target detection model, and the target detection model is any one of Faster R-CNN, YOLO or SSD.

[0017] Optionally, the calculation formula for the central pixel coordinates of the flower is:

[0018] x c =(x 2 -x 1 ) / 2, y c =(y2 -y 1 ) / 2;

[0019] Wherein, [x 1 , y 1 is the upper left corner coordinate of the detection box, and [x 2 , y 2 is the lower right corner coordinate of the detection box.

[0020] Optionally, the detection box is a rectangular box; the calculation method for the minimum boundary distance between two different flower detection boxes within the same inflorescence includes:

[0021] Calculate the minimum distances dx and dy of the two different flower detection boxes in the x-axis and y-axis directions respectively; dx and dy are calculated based on the following formulas:

[0022] dx = max(0, max(x 1_1 , x 1_2 ) - min(x 2_1 , x 2_2 ));

[0023] dy = max(0, max(y 1_1 , y 1_2 ) - min(y 2_1 , y 2_2 ));

[0024] Calculate the minimum boundary distance between the two different flower detection boxes through the Pythagorean theorem formula:

[0025] ;

[0026] Wherein, the coordinates of the two different flower detection boxes are [x 1_1 , y 1_1 , x 2_1 , y 2_1 and [x 1_2 , y 1_2 , x 2_2 , y 2_2 .

[0027] Optionally, for all the flowers within the same inflorescence, calculate the average value of the minimum boundary distances between each flower detection box and other flower detection boxes respectively, and determine the flower with the minimum average value as the central flower of the inflorescence, and the remaining flowers as the marginal flowers, including:

[0028] For all the flowers within the same inflorescence, calculate the sum of the minimum bounding distances between each flower detection box and other flower detection boxes respectively, and divide the sum of the minimum bounding distances by the number of statistical times to obtain the average value of the corresponding minimum bounding distance; wherein, the number of statistical times is the number of flowers in the inflorescence minus one;

[0029] Loop through and compare the numerical magnitudes of the average values of the minimum bounding distances corresponding to two flower detection boxes within the same inflorescence, and retain the flower detection box with the smaller average value until the target flower detection box with the minimum average value is obtained;

[0030] Determine the flower within the target flower detection box as the central flower, and the remaining flowers as the marginal flowers.

[0031] Optionally, after the step of mapping the central pixel coordinates into the inflorescence detection box and attributing the corresponding flower to the inflorescence corresponding to the inflorescence detection box according to the inflorescence detection box where the central pixel coordinates are located, the following steps are further included:

[0032] If there is only one flower detection box within an inflorescence, determine the flower corresponding to the flower detection box as the central flower of the inflorescence.

[0033] Optionally, the different identifications are different colors and / or different shapes and / or different annotations.

[0034] In a second aspect, based on the same inventive concept, the present invention further provides a method for automatically identifying the central flower and marginal flowers of fruit trees, including the steps of:

[0035] Construct and train a target detection model for flowers, where the target detection model is used to detect flower targets in images;

[0036] Detect a test image through the target detection model, obtain the detection box coordinates of all the flowers in the test image, and calculate the central pixel coordinates of each flower;

[0037] Use a clustering algorithm with the central pixel coordinates within the test image as clustering features to cluster the flower detection boxes, so as to divide the inflorescence categories to which each flower detection box belongs;

[0038] For all the flowers within the same inflorescence, calculate the average value of the minimum bounding distances between each flower detection box and other flower detection boxes respectively, and determine the flower with the minimum average value as the central flower of the inflorescence, and the remaining flowers as the marginal flowers;

[0039] Visually annotate the detection boxes corresponding to the inflorescences, the central flowers, and the marginal flowers in the test image, and distinguish and display them with different identifications.

[0040] Optionally, building and training the object detection model for flowers includes:

[0041] Building a dataset, where the dataset includes a number of training images, and each training image includes a flower;

[0042] Annotating the flowers in the training images in the dataset, using bounding boxes to mark the boundaries of each flower, and the class label is flower;

[0043] Training a deep learning object detection model using the annotated dataset, and the object detection model is any one of Faster R-CNN, YOLO or SSD.

[0044] Optionally, the calculation formula for the central pixel coordinates of the flower is:

[0045] x c =(x 2 -x 1 ) / 2, y c =(y 2 -y 1 ) / 2;

[0046] Where, [x 1 , y 1 is the upper left coordinate of the detection box, and [x 2 , y 2 is the lower right coordinate of the detection box.

[0047] Optionally, the flower detection box is a rectangular box; the calculation method for the minimum boundary distance between two different flower detection boxes within the same inflorescence includes:

[0048] Calculating the minimum distances dx and dy of the two different flower detection boxes in the x-axis and y-axis directions respectively; dx and dy are calculated based on the following formulas:

[0049] dx = max(0, max(x 1_1 , x 1_2 ) - min(x 2_1 , x 2_2 ));

[0050] dy = max(0, max(y 1_1 , y 1_2 ) - min(y 2_1 , y 2_2 ));

[0051] Calculating the minimum boundary distance between two different flower detection boxes through the Pythagorean theorem formula:

[0052] ;

[0053] Among them, the coordinates of two different flower detection frames are [x 1_1 , y 1_1 , x 2_1 , y 2_1 and [x 1_2 , y 1_2 , x 2_2 , y 2_2 .

[0054] Optionally, for all the flowers within the same inflorescence, the steps of respectively calculating the average value of the minimum bounding distances between each of the flower detection frames and other flower detection frames, and determining the flower with the smallest average value as the central flower of the inflorescence and the remaining flowers as the marginal flowers include:

[0055] For all the flowers within the same inflorescence, respectively calculate the sum of the minimum bounding distances between each flower detection frame and other flower detection frames, and divide the sum of the minimum bounding distances by the number of statistical times to obtain the corresponding average value of the minimum bounding distances; wherein, the number of statistical times is the number of flowers within the inflorescence minus one;

[0056] Loop through and compare the numerical sizes of the average values of the minimum bounding distances corresponding to two flower detection frames within the same inflorescence, and retain the flower detection frame with the smaller average value until the target flower detection frame with the smallest average value is obtained;

[0057] Determine the flower within the target flower detection frame as the central flower and the remaining flowers as the marginal flowers.

[0058] Optionally, after the step of using a clustering algorithm with the central pixel coordinates within the to-be-tested image as the clustering feature to cluster the flower detection frames to divide the inflorescence category to which each flower detection frame belongs, it further includes:

[0059] If there is only one flower detection frame within an inflorescence, determine the flower corresponding to the flower detection frame as the central flower of the inflorescence.

[0060] Optionally, the different identifications are different colors and / or different shapes and / or different annotations.

[0061] Optionally, the clustering algorithm is the KMeans algorithm or the density peak clustering algorithm.

[0062] In a third aspect, based on the same inventive concept, the present invention further provides a method for automatically identifying the central flower and marginal flowers of fruit trees, including the steps of:

[0063] Construct and train a target detection model for flowers, where the target detection model is used to detect flower targets in images;

[0064] Use the target detection model to detect the image to be measured, obtain the coordinates of the detection boxes of all the flowers in the image to be measured, and calculate the central pixel coordinates of each flower;

[0065] Adopt a clustering algorithm, use the central pixel coordinates in the image to be measured as clustering features, and cluster the flower detection boxes to divide the inflorescence categories to which each flower detection box belongs;

[0066] Construct and train a relationship prediction model through scene graph generation technology, and use the trained relationship prediction model to predict the positional relationship of all the flowers within the same inflorescence to determine the central flower and the marginal flowers within the inflorescence;

[0067] Visually annotate the detection boxes corresponding to the inflorescence, the central flower, and the marginal flowers in the image to be measured, and distinguish and display them with different identifiers.

[0068] Optionally, the construction and training of the target detection model for flowers includes:

[0069] Construct a data set, which includes a number of training images, and each training image includes flowers;

[0070] Annotate the flowers in the training images in the data set, use bounding boxes to annotate the boundaries of each flower, and the category label is flower;

[0071] Use the annotated data set to train a deep learning target detection model, and the target detection model is any one of Faster R-CNN, YOLO, or SSD.

[0072] Optionally, the calculation formula for the central pixel coordinates of the flower is:

[0073] x c =(x 2 -x 1 ) / 2, y c =(y 2 -y 1 ) / 2;

[0074] Wherein, [x 1 , y 1 is the upper left coordinate of the detection box, and [x 2 , y 2 is the lower right coordinate of the detection box.

[0075] Optionally, constructing and training a relationship prediction model through the scene graph generation technology, and using the trained relationship prediction model to predict the positional relationship of all the flowers within the same inflorescence to determine the central flower and the marginal flowers within the inflorescence includes:

[0076] Performing surrounding relationship annotation on the central flower and the marginal flowers in the training data to form relationship annotation data;

[0077] Taking a graph neural network as a relationship prediction network, regarding each flower as a node, with the node feature being the flower position feature, constructing edges based on the spatial positional relationship between the flowers, and learning the positional relationship features between the flowers through the message passing mechanism of the graph neural network to construct a relationship prediction model;

[0078] Inputting the relationship annotation data into the relationship prediction model, and using cross-entropy loss as the loss function for the relationship prediction task to perform training to obtain the trained relationship prediction model;

[0079] Taking the detection box position information of all the flowers within the same inflorescence in the to-be-detected image as the model input of the relationship prediction model to obtain the relationship graph structure output by the relationship prediction model;

[0080] Mapping the relationship graph structure onto the to-be-detected image, and identifying and determining the central flower and the marginal flowers within the same inflorescence according to the display position of the relationship graph structure on the to-be-detected image.

[0081] Optionally, after the step of using a clustering algorithm with the central pixel coordinates within the to-be-detected image as the clustering feature to cluster the flower detection boxes to divide the inflorescence categories to which each flower detection box belongs, it further includes:

[0082] If there is only one flower detection box within an inflorescence, determining the flower corresponding to the flower detection box as the central flower of the inflorescence.

[0083] Optionally, the different identifications are different colors and / or different shapes and / or different annotations.

[0084] Optionally, the clustering algorithm is the KMeans algorithm or the density peak clustering algorithm.

[0085] In a fourth aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory, where when the processor executes the computer program, the method described above is implemented.

[0086] In a fifth aspect, the present invention further provides a storage medium, on which a computer program is stored, where when the computer program is executed by a processor, the method described above is implemented.

[0087] The present invention uses a deep learning method combined with a solution that takes into account the positional relationship of flowers to automatically identify the central flower and marginal flowers within the inflorescence of fruit trees. On the one hand, the use of a deep learning model overcomes the problems of traditional image processing algorithms being overly dependent on factors such as color and texture and being easily affected by environmental illumination. On the other hand, the present invention creatively combines the mutual positional relationship of flowers to solve the problem of being unable to accurately identify the central flower and marginal flowers when the central flower and marginal flowers exhibit the same characteristics such as color, shape, and texture.

[0088] The present invention can more accurately predict the central flower and marginal flowers in the image by using the algorithm rule that the average distance between the rectangular frame boundaries of the central flower and other rectangular frame boundaries is the shortest, with the help of the positional relationship between flowers.

[0089] The present invention constructs a relationship prediction model by using the scene graph generation technology in deep learning, and uses the model to identify the mutual relationship of the flowers within the inflorescence to determine the central flower and marginal flowers within the inflorescence. Description of the Drawings

[0090] Figure 1 It is a flowchart of the steps of the method for automatically identifying the central flower and marginal flowers of fruit trees provided in the first embodiment of the present invention;

[0091] Figure 2 It is a schematic diagram of data annotation when constructing the target detection model of flowers and inflorescences in the first embodiment of the present invention;

[0092] Figure 3 It is a schematic diagram of matching flowers and inflorescences of the method for automatically identifying the central flower and marginal flowers of fruit trees provided in the first embodiment of the present invention;

[0093] Figure 4 It is a schematic diagram of calculating the minimum boundary distance between two rectangular frames enclosing flowers of the method for automatically identifying the central flower and marginal flowers of fruit trees provided in the first embodiment of the present invention;

[0094] Figure 5 It is a flowchart of the steps of the method for automatically identifying the central flower and marginal flowers of fruit trees provided in the second embodiment of the present invention;

[0095] Figure 6 It is a schematic diagram of using the clustering algorithm to obtain the inflorescence category to which each flower detection frame belongs in the method for automatically identifying the central flower and marginal flowers of fruit trees provided in the second embodiment of the present invention;

[0096] Figure 7 It is a flowchart of the steps of the method for automatically identifying the central flower and marginal flowers of fruit trees provided in the third embodiment of the present invention;

[0097] Figure 8It is a specific step flowchart for determining the central flower and lateral flowers within the inflorescence in the automatic recognition method of the central flower and lateral flowers of fruit trees provided in Embodiment 3 of the present invention;

[0098] Figure 9 It is a schematic diagram showing the positional relationship of flowers by mapping the relational graph structure of the automatic recognition method of the central flower and lateral flowers of fruit trees provided in Embodiment 3 of the present invention onto the image to be measured;

[0099] Figure 10 It is a schematic diagram of the hardware structure of the electronic device provided in Embodiment 4 of the present invention. Detailed implementation manners

[0100] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0101] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiment", etc. in this specification mean that the described embodiment may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Further, when combining embodiments to describe specific features, structures or characteristics, whether or not there is an explicit description, it has been shown that it is within the knowledge of those skilled in the art to combine such features, structures or characteristics into other embodiments.

[0102] In addition, in the specification and subsequent claims, certain terms are used to refer to specific components or parts. Those of ordinary skill in the art should understand that manufacturers may use different nouns or terms to refer to the same component or part. The specification and subsequent claims do not use the difference in names as a way to distinguish components or parts, but use the difference in functions of components or parts as the criterion for distinction. The terms "including" and "comprising" mentioned throughout the specification and subsequent claims are open-ended terms, so they should be interpreted as "including but not limited to". In addition, the term "connection" herein includes any direct and indirect electrical connection means. Indirect electrical connection means include connection through other devices.

[0103] Before describing the embodiments of the present application in detail, the technical concept of the present application will be briefly described first: First, a target detection model is used to locate flowers / inflorescences. For the problem of inflorescence attribution, a dual-path scheme is innovatively designed to associate flowers with inflorescences through coordinate mapping in the scenario of pre-detecting inflorescences, or a clustering algorithm is used to match inflorescences based on the spatial distribution of flowers when there is no inflorescence annotation. For the crucial determination of central flowers, two complementary techniques are proposed: calculating the mean value of the minimum bounding distances between flowers based on geometric features to locate central flowers through physical proximity, or introducing scene graph generation technology to model the topological relationships between flowers and infer the central position through deep learning. By integrating target detection, spatial clustering, and relationship reasoning technologies, the present application breaks through the dependence on artificial features in traditional image processing, realizes the full-process automation from flower detection, inflorescence attribution to central flower discrimination, not only adapts to application scenarios with different annotation conditions, but also improves the recognition accuracy through dual determination logic, and finally outputs the results with visual annotation, providing accurate data support for fruit tree flower thinning operations.

[0104] The following describes the specific principle of the automatic recognition method for central flowers and marginal flowers of fruit trees in the present application in combination with specific embodiments.

[0105] Embodiment 1

[0106] Figure 1 The automatic recognition method for central flowers and marginal flowers of fruit trees provided in Embodiment 1 of the present invention is shown. This embodiment is specifically applied to the recognition of central flowers and marginal flowers of apple trees, and of course, it can also be applied to the recognition of central flowers and marginal flowers of other fruit trees; the method includes the following steps:

[0107] S101: Construct and train a target detection model for inflorescences and flowers, and the target detection model is used to detect inflorescence targets and flower targets in an image simultaneously.

[0108] In an optional implementation manner, step S101 includes:

[0109] Construct a data set, and the data set includes a number of training images, and each training image includes an inflorescence and a flower; respectively label bounding boxes for the inflorescences and flowers in the training images in the data set; wherein, each flower in the inflorescence is independently labeled as the flower category, and the inflorescence as a whole is labeled as the inflorescence category; specifically in implementation, as Figure 2 shown, for the given data set, image data annotation is performed. Using an image target detection annotation tool, for the two types of objects, inflorescences and flowers, in the image, bounding boxes are labeled. That is, first, each flower in a single inflorescence is labeled with a bounding box one by one, and the bounding box category is flower, and then this inflorescence is labeled with a bounding box, and the bounding box category is inflorescence. According to this method, all inflorescences and the flowers inside them in the training images are labeled with bounding boxes.

[0110] Train a deep learning object detection model using the labeled dataset. The object detection model can be any one of Faster R-CNN (Fast Region-based Convolutional Neural Network), YOLO (You Only Look Once, a deep learning-based object detection algorithm), or SSD (Single Shot MultiBox Detector, an object detection algorithm). Of course, in other examples, other deep learning object detection models can also be used for training. The trained object detection model can detect apple inflorescences and flowers in an image simultaneously.

[0111] S102: Detect the image to be tested through the object detection model, obtain the coordinates of the detection frames of all inflorescences and flowers in the image to be tested, and calculate the central pixel coordinates of each flower.

[0112] The image to be tested refers to an image in which the central flower and the edge flowers need to be detected, which can be either a picture taken by a camera or a frame image taken from video data.

[0113] Use the object detection model to detect all inflorescences and flowers in the image to be tested, obtain the detection results output by the model, and according to the detection results, the coordinates of the detection frame of each flower can be obtained as [x 1 ,y 1 ,x 2 ,y 2 , where the upper left corner coordinates of the target box are [x 1 ,y 1 , and the lower right corner coordinates of the target box are [x 2 ,y 2 ; For this, in an optional implementation provided in this embodiment, the calculation formula for the central pixel coordinates of the flower is:

[0114] x c =(x 2 -x 1 ) / 2, y c =(y 2 -y 1 ) / 2; where [x 1 ,y 1 are the upper left corner coordinates of the detection frame, and [x 2 ,y 2 are the lower right corner coordinates of the detection frame.

[0115] S103: Map the central pixel coordinates into the inflorescence detection frame, and according to the inflorescence detection frame where the central pixel coordinates are located, attribute the corresponding flower to the inflorescence corresponding to the inflorescence detection frame.

[0116] The purpose of this step S103 is to match each flower identified in the figure to the corresponding inflorescence. Specifically, during implementation, the detection box of the inflorescence in the image is regarded as the position threshold, and the central pixel coordinates of the flowers in the figure are mapped onto the detection box of the inflorescence. If the central pixel coordinates of a flower are within the detection box of a certain inflorescence, then the flower and the inflorescence are successfully matched. In this way, the inflorescence to which each flower in the image belongs is recursively matched in turn.

[0117] See Figure 3 , within a detection box of an inflorescence, there are three detection boxes of flowers, and their central pixel coordinates are respectively (x c_1 , y c_1 ), (x c_2 , y c_2 ), and (x c_3 , y c_3 ); all these three central pixel coordinates fall within this detection box of the inflorescence. Therefore, these three flowers are attributed to this inflorescence. Here, the "attribution" referred to is equivalent to matching, that is, at this time, all three flowers are matched to this inflorescence. Based on the matching relationship between the inflorescence and the flowers in this embodiment, a relevant data matching set can be established. For example, the data set of flowers matched by a certain inflorescence is {Flower 1, Flower 2, and Flower 3}.

[0118] S104: For all the flowers within the same inflorescence, calculate the average value of the minimum boundary distances between each flower detection box and other flower detection boxes respectively, and determine the flower with the minimum average value as the central flower of the inflorescence, and the remaining flowers as the side flowers. For example, the data set of flowers matched by a certain inflorescence is {A 1 , A 2 , …, A n} (the flowers are all the flowers detected by the detection model and all have rectangular bounding box coordinates [x 1 , y 1 , x 2 , y 2 ). Calculating the average value of the minimum boundary distances between each flower detection box and other flower detection boxes respectively means that each flower in {A 1 , A 2 , …, A n} calculates the minimum boundary distance with other flowers in {A 1 , A 2 , …, A n} respectively, and then obtains the average value. That is, according to the rectangular bounding box coordinates of {A 1 , A 2 , …, A n}, calculate the rectangular box with the shortest distance between the boundaries of each rectangular box and the boundaries of other rectangular boxes, and determine the flower surrounded by this rectangular box as the central flower of this inflorescence, and the flowers surrounded by other rectangular boxes within this inflorescence as the side flowers of this inflorescence.

[0119] The detection frame in this embodiment is a rectangular frame; the calculation method for the minimum boundary distance between the detection frames of two different flowers within the same inflorescence includes:

[0120] Calculate the minimum distances dx and dy in the x-axis and y-axis directions between the detection frames of the two different flowers respectively; dx and dy are calculated based on the following formulas:

[0121] dx = max(0, max(x 1_1 , x 1_2 ) - min(x 2_1 , x 2_2 ));

[0122] dy = max(0, max(y 1_1 , y 1_2 ) - min(y 2_1 , y 2_2 ));

[0123] Calculate the minimum boundary distance between the detection frames of the two different flowers through the Pythagorean theorem formula:

[0124] ;

[0125] wherein, the coordinates of the detection frames of the two different flowers are [x 1_1 , y 1_1 , x 2_1 , y 2_1 and [x 1_2 , y 1_2 , x 2_2 , y 2_2 .

[0126] See Figure 4 , first calculate the minimum boundary distance between the two rectangular frames enclosing the flowers. Among them, one rectangular frame enclosing the flower is rect 1 , with coordinates [x 1_1 , y 1_1 , x 2_1 , y 2_1 ; the other rectangular frame enclosing the flower is rect 2 , with coordinates [x 1_2 , y 1_2 , x 2_2 , y 2_2 ; calculate the distances dx and dy in the x-axis and y-axis directions of the two rectangular frames according to the above formulas.

[0127] The specific operation steps of the above formula are: First, compare the x coordinates of the upper left corners of the two rectangular frames, and select the larger x coordinate value (named V ax); At the same time, compare the x - coordinates of the lower - right corners of the two rectangular frames, and select the smaller x - coordinate value (named V bx ), the distance V in the x - direction x= V ax -V bx , use max(0, V x ) to ensure that the dx distance is non - negative. Similarly, compare the y - coordinates of the upper - left corners of the two rectangular frames, and select the larger y - coordinate value (named V ay ); At the same time, compare the y - coordinates of the lower - right corners of the two rectangular frames, and select the smaller y - coordinate value (named V by ), the distance V in the y - direction y= V ay -V by , use max(0, V y ) to ensure that the dy distance is non - negative. Then, through the formula dist = sqrt(dx 2 +dy 2 ), calculate the minimum boundary distance dist between the two rectangular frames; sqrt is the square - root operation.

[0128] Using the above - mentioned calculation method, the minimum boundary distance between the two rectangular frames can be obtained, and then the rectangular frame with the shortest average distance from the boundary of the rectangular frame within the single inflorescence to the boundaries of other rectangular frames can be calculated.

[0129] In an optional embodiment, step S104 includes:

[0130] For all the flowers within the same inflorescence, calculate the sum of the minimum boundary distances between each flower detection frame and other flower detection frames respectively, and divide the sum of the minimum boundary distances by the statistical number to obtain the average value of the corresponding minimum boundary distances; where the statistical number is the number of flowers in the inflorescence minus one; loop through and compare the numerical magnitudes of the average values of the minimum boundary distances corresponding to two flower detection frames within the same inflorescence, and retain the flower detection frame with the smaller average value until the target flower detection frame with the minimum average value is obtained; determine the flower within the target flower detection frame as the central flower, and the remaining flowers as the marginal flowers.

[0131] Specifically, the above - mentioned embodiment can be implemented through the following algorithm process:

[0132] I. Input the list of flower - surrounding rectangular frames rectangles obtained by performing flower detection using the detection model, and the format of each rectangular frame is [x 1 , y 1 , x 2 , y 2 .

[0133] II. First, the algorithm determines whether the length of the list rectangles is 1. If so, it is considered that there is only one rectangular box, and then directly returns the coordinates of this rectangular box, including the upper left corner coordinates [x 1 , y 1 and the lower right corner coordinates [x 2 , y 2 ; if it is greater than 1, it proceeds to the next step.

[0134] III. Traverse the list of rectangular boxes rectangles{1, 2,..., n} through two-layer loops. The specific process includes:

[0135] (1) First, take the first rectangular box in the list as the target rectangular box, and use the algorithm of "the calculation method of the minimum boundary distance between two different flower detection boxes in the same inflorescence" mentioned above to calculate the minimum boundary distances dist{1, 2,..., n - 1} between this rectangular box and other rectangular boxes respectively; then, calculate the sum total_distance of these minimum boundary distances, and calculate the average value of total_distance according to the formula "total_distance / (n - 1)", and take it as the average distance avg_distance_ 1 .

[0136] (2) Successively take each rectangular box in the list as the target rectangular box, and in the same way as above, calculate the average distance avg_distance_ x .

[0137] (3) Each time the calculated average distance avg_distance_ x will be compared with the previously calculated average distance avg_distance_ x-1 in terms of numerical size. If avg_distance_x is less than avg_distance_ x-1 , then the target rectangular box corresponding to avg_distance_x is regarded as the rectangular box with the shortest boundary distance from the boundaries of other rectangular boxes. The apple flower surrounded by this rectangular box is determined as the central flower in the target inflorescence, and the apple flowers surrounded by other rectangular boxes in this inflorescence are determined as side flowers.

[0138] Further, after step S103, it also includes: if there is exactly one flower detection box in an inflorescence, the flower corresponding to the flower detection box is determined as the central flower of the inflorescence.

[0139] S105: Visually annotate the detection boxes corresponding to the inflorescences, central flowers, and marginal flowers in the image to be measured, and display them with different identifiers for distinction. Optionally, the different identifiers are different colors and / or different shapes and / or different annotations.

[0140] In this embodiment, by using the deep learning method and considering the positional relationship of the flowers, the dependence on artificial features in traditional image processing is broken through, and the full process automation from flower detection, inflorescence attribution to central flower discrimination is realized. It not only adapts to application scenarios with different annotation conditions, but also improves the recognition accuracy through double judgment logic. Finally, the result is output with visual annotation, providing accurate data support for the flower thinning operation of fruit trees.

[0141] Embodiment 2

[0142] Figure 5 Show the automatic recognition method for the central flower and marginal flower of a fruit tree provided in Embodiment 2, including the steps:

[0143] S201: Construct and train a target detection model for flowers, where the target detection model is used to detect flower targets in images.

[0144] In an optional implementation manner, step S201 includes:

[0145] Construct a data set, where the data set includes a number of training images, and each training image includes flowers; annotate the flowers in the training images in the data set, use bounding boxes to annotate the boundaries of each flower, and the category label is flower; that is, for the given data set, perform image data annotation. Using an image detection annotation tool, for the flowers in the training images, annotate the bounding boxes, and annotate each flower with a bounding box one by one. The category of the bounding box is flower; according to this method, all the flowers in the training images are annotated with bounding boxes.

[0146] Use the annotated data set to train a deep learning target detection model, and the target detection model is any one of Faster R-CNN, YOLO, or SSD. The trained target detection model in this embodiment is used to detect the flowers in the image, that is, the apple tree flowers.

[0147] S202: Detect the image to be measured through the target detection model, obtain the coordinates of the detection boxes of all the flowers in the image to be measured, and calculate the central pixel coordinates of each flower.

[0148] The image to be measured refers to the image in which the central flower and marginal flower need to be detected, which can be either a picture taken by a camera or a frame image taken from video data.

[0149] Use the object detection model to detect all the flowers in the image to be tested, obtain the detection results output by the model, and according to the detection results, the coordinates of the detection box of each flower can be obtained as [x 1 , y 1 , x 2 , y 2 . Among them, the upper left corner coordinates of the target box are [x 1 , y 1 , and the lower right corner coordinates of the target box are [x 2 , y 2 ; In this regard, in an optional implementation manner provided in this embodiment, the calculation formula for the central pixel coordinates of the flower is:

[0150] x c =(x 2 -x 1 ) / 2, y c =(y 2 -y 1 ) / 2; where, [x 1 , y 1 are the upper left corner coordinates of the detection box, and [x 2 , y 2 are the lower right corner coordinates of the detection box.

[0151] S203: Use a clustering algorithm with the central pixel coordinates in the image to be tested as the clustering feature to cluster the flower detection boxes, so as to divide the inflorescence category to which each flower detection box belongs. As Figure 6 shown, take the central pixel coordinates of all flower detection boxes as the clustering feature, and use a clustering algorithm (such as the KMeans algorithm or the Density Peak Clustering (DPC) algorithm) to cluster the central pixel coordinates of the flower detection boxes to obtain the inflorescence category to which each flower detection box belongs, that is, in this embodiment, the clustering algorithm is used to group the flowers with close distances into one inflorescence.

[0152] S204: For all the flowers in the same inflorescence, calculate the average value of the minimum boundary distances between each flower detection box and other flower detection boxes respectively, and determine the flower with the smallest average value as the central flower of the inflorescence, and the remaining flowers as the marginal flowers.

[0153] For example, the dataset of the flowers matched by a certain inflorescence is {A 1 , A 2 , …, A n} (the flowers are all the flowers detected by the detection model, and all have the rectangular bounding box coordinates [x 1 , y 1 , x 2 , y 2 ), calculating the average value of the minimum boundary distances between each flower detection box and other flower detection boxes respectively means {A1 , A 2 , …, A n}, each flower in it is respectively calculated with the minimum boundary distance from the other flowers in {A 1 , A 2 , …, A n}, and then the average value is obtained; that is, according to the rectangular bounding box coordinates of {A 1 , A 2 , …, A n}, calculate the rectangular box with the shortest boundary distance from the boundaries of other rectangular boxes in each rectangular box, and determine the flower surrounded by this rectangular box as the central flower of this inflorescence, and determine the flowers surrounded by other rectangular boxes in this inflorescence as the marginal flowers of this inflorescence.

[0154] The detection box in this embodiment is a rectangular box; the calculation method for the minimum boundary distance between two different flower detection boxes within the same inflorescence includes:

[0155] Calculate the minimum distances dx and dy in the x-axis and y-axis directions of the two different flower detection boxes respectively; dx and dy are calculated based on the following formulas:

[0156] dx = max(0, max(x 1_1 , x 1_2 ) - min(x 2_1 , x 2_2 ));

[0157] dy = max(0, max(y 1_1 , y 1_2 ) - min(y 2_1 , y 2_2 ));

[0158] Calculate the minimum boundary distance between the two different flower detection boxes through the Pythagorean theorem formula:

[0159] ;

[0160] Among them, the coordinates of the two different flower detection boxes are [x 1_1 , y 1_1 , x 2_1 , y 2_1 and [x 1_2 , y 1_2 , x 2_2 , y 2_2 .

[0161] Specifically, first calculate the minimum boundary distance between the two rectangular boxes surrounding the flowers. Among them, one rectangular box surrounding the flower is rect 1 , and the coordinates are [x 1_1 , y 1_1 , x2_1 , y 2_1 ; Another rectangular box surrounding the flower is rect 2 , with coordinates [x 1_2 , y 1_2 , x 2_2 , y 2_2 ; Calculate the distances dx and dy in the x-axis and y-axis directions between the two rectangular boxes according to the above formula.

[0162] The specific operation steps of the above formula are as follows: First, compare the x coordinates of the upper left corners of the two rectangular boxes, and select the larger x coordinate value (named V ax ); At the same time, compare the x coordinates of the lower right corners of the two rectangular boxes, and select the smaller x coordinate value (named V bx ), the distance V in the x direction x= V ax -V bx , use max(0, V x ) to ensure that the dx distance is non-negative. Similarly, compare the y coordinates of the upper left corners of the two rectangular boxes, and select the larger y coordinate value (named V ay ); At the same time, compare the y coordinates of the lower right corners of the two rectangular boxes, and select the smaller y coordinate value (named V by ), the distance V in the y direction y= V ay -V by , use max(0, V y ) to ensure that the dy distance is non-negative. Furthermore, through the formula dist = sqrt(dx 2 + dy 2 ), calculate the minimum boundary distance dist between the two rectangular boxes; sqrt is the square root operation.

[0163] Using the above calculation method, the minimum boundary distance between the two rectangular boxes can be obtained, and then the rectangular box with the shortest average distance from the boundary of other rectangular boxes within a single inflorescence can be calculated.

[0164] In an optional implementation manner, step S204 includes:

[0165] For all the flowers within the same inflorescence, calculate the sum of the minimum bounding distances between each flower detection box and other flower detection boxes respectively, and divide the sum of the minimum bounding distances by the number of statistical times to obtain the average value of the corresponding minimum bounding distance; wherein, the number of statistical times is the number of flowers in the inflorescence minus one; loop through and compare the numerical sizes of the average values of the minimum bounding distances corresponding to two flower detection boxes within the same inflorescence, and retain the flower detection box with the smaller average value until the target flower detection box with the minimum average value is obtained; determine the flower within the target flower detection box as the central flower, and the remaining flowers as the side flowers.

[0166] When specifically implementing, the above implementation manner can be achieved through the following algorithm process:

[0167] I. Input the list of flower bounding rectangles rectangles obtained by performing flower detection using a detection model. Each rectangle format is [x 1 , y 1 , x 2 , y 2 .

[0168] II. The algorithm first determines whether the length of the list rectangles is 1. If it is, it is regarded as having only one rectangle, and then directly returns the coordinates of this rectangle, including the upper-left coordinates [x 1 , y 1 and the lower-right coordinates [x 2 , y 2 ; if it is greater than 1, it proceeds to the next step.

[0169] III. Traverse the rectangle list rectangles{1, 2,..., n} through two nested loops. The specific process includes:

[0170] (1) First, take the first rectangle in the list as the target rectangle, and use the algorithm in the "Calculation method of the minimum bounding distance between two different flower detection boxes within the same inflorescence" mentioned above to calculate the minimum bounding distances dist{1, 2,..., n - 1} between this rectangle and other rectangles respectively; then, calculate the sum total_distance of these minimum bounding distances, and calculate the average value of total_distance according to the formula "total_distance / (n - 1)", and take it as the average distance avg_distance_ 1 .

[0171] (2) Successively take each rectangle in the list as the target rectangle, and in the same way as above, calculate the average distance avg_distance_ x .

[0172] (3) The average distance avg_distance_ calculated each time x will be compared with the average distance avg_distance_ calculated in the previous time x-1 in terms of numerical value. If avg_distance_x is less than avg_distance_ x-1 , then the target rectangle corresponding to avg_distance_x is regarded as the rectangle with the shortest distance between its boundary and the boundaries of other rectangles. The apple flower surrounded by this rectangle is determined as the central flower within the target inflorescence, while the apple flowers surrounded by other rectangles within this inflorescence are determined as marginal flowers.

[0173] Furthermore, after step S203, it further includes: If there is only one flower detection box within an inflorescence, the flower corresponding to the flower detection box is determined as the central flower of the inflorescence.

[0174] S205: Visually annotate the detection boxes corresponding to the inflorescences, central flowers, and marginal flowers in the to-be-detected image, and distinguish and display them with different identifiers. Optionally, the different identifiers are different colors and / or different shapes and / or different annotations.

[0175] In this embodiment, by using a clustering algorithm to dynamically match flowers to inflorescences based on the spatial distribution of flowers, and using the algorithm rule that the average distance between the rectangle boundary of the central flower and the boundaries of other rectangles is the shortest, it is possible to more accurately predict the central flower and marginal flowers in the image with the help of the positional relationship between flowers.

[0176] Embodiment Three

[0177] Figure 7 The automatic recognition method for the central flower and marginal flowers of fruit trees provided in Embodiment Three is shown, including the following steps:

[0178] S301: Construct and train a target detection model for flowers, and the target detection model is used to detect flower targets in images.

[0179] In an optional implementation manner, step S301 includes:

[0180] Construct a data set, the data set includes a number of training images, and each training image includes flowers; annotate the flowers in the training images in the data set, use bounding boxes to annotate the boundaries of each flower, and the category label is flower; that is, for the given data set, perform image data annotation. Using an image detection annotation tool, for the flowers in the training images, annotate bounding boxes, and annotate each flower with a bounding box one by one. The category of the bounding box is flower; in this way, all the flowers in the training images are annotated with bounding boxes.

[0181] Train a deep learning object detection model using the labeled dataset, where the object detection model is any one of Faster R-CNN, YOLO, or SSD. The trained object detection model in this embodiment is used to detect the flowers in the image, that is, the apple tree flowers.

[0182] S302: Detect the image to be tested through the object detection model, obtain the coordinates of the detection boxes of all the flowers in the image to be tested, and calculate the central pixel coordinates of each flower.

[0183] The image to be tested refers to the image in which the central flower and the edge flowers need to be detected, which can be either a picture taken by a camera or a frame image taken from video data.

[0184] Use the object detection model to detect all the flowers in the image to be tested, obtain the detection results output by the model, and according to the detection results, the coordinates of the detection box of each flower can be obtained as [x 1 , y 1 , x 2 , y 2 , where the upper left coordinates of the target box are [x 1 , y 1 , and the lower right coordinates of the target box are [x 2 , y 2 ; In this regard, in an optional implementation manner provided in this embodiment, the calculation formula for the central pixel coordinates of the flower is:

[0185] x c = (x 2 - x 1 ) / 2, y c = (y 2 - y 1 ) / 2; where [x 1 , y 1 are the upper left coordinates of the detection box, and [x 2 , y 2 are the lower right coordinates of the detection box.

[0186] S303: Use a clustering algorithm with the central pixel coordinates in the image to be tested as the clustering feature to cluster the flower detection boxes, so as to divide the inflorescence category to which each flower detection box belongs. Similar to the above Embodiment 2, in this embodiment, the central pixel coordinates of the flower detection box are used as the clustering feature, and a clustering algorithm (such as the KMeans algorithm or the Density Peak Clustering (DPC) algorithm) is used to cluster the central pixel coordinates of the flower detection boxes to obtain the inflorescence category to which each flower detection box belongs, that is, in this embodiment, the clustering algorithm is used to group the flowers with close distances into one inflorescence.

[0187] S304: Construct and train a relationship prediction model through scene graph generation technology, and use the trained relationship prediction model to predict the positional relationships of all the flowers within the same inflorescence to determine the central flower and the marginal flowers within the inflorescence. That is, use scene graph generation technology to construct a relationship prediction model, and use this relationship prediction model to identify the mutual relationships of the flowers within the inflorescence to determine the central flower and the marginal flowers within the inflorescence.

[0188] See Figure 8 , in an optional embodiment, step S304 includes:

[0189] S3041: Perform surrounding relationship annotation on the central flowers and the marginal flowers in the training data to form relationship annotation data. To train the relationship prediction model, it is necessary to perform relationship annotation on the image data. For this, in this embodiment, for the flowers in the training images, the positional relationship between the marginal flowers and the central flowers is defined, that is, the relationship that the marginal flowers "surround" the central flowers. The training images mentioned here are the training data used to train the relationship prediction model, which can be the same as or different from the training data for training the above-mentioned object detection model. For each marginal flower, annotate its relationship with the central flower, and realize the relationship annotation by creating a relationship table in the annotation tool or using a specific annotation syntax. For example, use the format of "marginal flower ID: surround: central flower ID" in the annotation file to record the relationship.

[0190] S3042: Use a graph neural network as the relationship prediction network, regard each flower as a node, the node feature is the flower position feature, and the edges are constructed based on the spatial positional relationships between the flowers. Through the message passing mechanism of the graph neural network, learn the positional relationship features between the flowers to construct a relationship prediction model. Among them, the edges are constructed based on the spatial positional relationships between the flowers. If the distance between the bounding boxes of two flowers is within a certain range, then add an edge between their corresponding nodes. And through the message passing mechanism of the graph neural network, the nodes pass information to each other and learn the positional relationship features between the flowers, that is, the "surrounding" relationship of the marginal flowers to the central flower.

[0191] S3043: Input the relationship annotation data into the relationship prediction model, and use the cross-entropy loss as the loss function for the relationship prediction task to perform training to obtain a trained relationship prediction model. The loss function for the relationship prediction task in this embodiment uses the cross-entropy loss, which can measure the difference between the predicted relationship and the annotated relationship.

[0192] S3044: Use the detection box position information of all the flowers within the same inflorescence in the image to be tested as the model input of the relationship prediction model to obtain the relationship graph structure output by the relationship prediction model. Specifically, when using the trained model for relationship prediction, for the flowers {A 1 , A 2 , …, An} (The flowers are those detected by the detection model). The position information of the flower detection boxes is used as the input to the model to predict the "surrounding" relationship between the side flowers and the central flower. The results output by the model are identified by a graph structure, where the nodes are the flowers and the edges represent the positional relationships between the flowers. Each node contains position information (center coordinates, detection box), and each edge contains the relationship type ("surrounding" relationship) and the confidence score of the relationship.

[0193] S3045: Map the graph structure of the relationship to the image to be tested, and identify and determine the central flower and side flowers within the same inflorescence according to the display position of the graph structure of the relationship on the image to be tested. As Figure 9 shown, use a graphical visualization tool to map the output graph structure of the relationship to the original RGB image of the apple flower, and use arrows for the edges between the flowers to represent the surrounding relationship, so that the positional relationship of the side flowers surrounding the central flower in the inflorescence can be visually displayed, thereby realizing the identification of the central flower and side flowers within the inflorescence.

[0194] Further, after step S203, it further includes: If there is only one flower detection box within an inflorescence, the flower corresponding to the flower detection box is determined as the central flower of the inflorescence.

[0195] S305: Visually annotate the detection boxes corresponding to the inflorescence, the central flower, and the side flowers in the image to be tested, and display them separately with different identifiers. The different identifiers are different colors and / or different shapes and / or different annotations.

[0196] In this embodiment, a relationship prediction model is constructed by using the scene graph generation technology in deep learning, and the relationship prediction model is used to identify the mutual relationships of the flowers within the inflorescence to determine the central flower and side flowers within the inflorescence; it integrates object detection, spatial clustering, and relationship reasoning technologies, breaks through the dependence on artificial features in traditional image processing, realizes the full-process automation from flower detection, inflorescence attribution to central flower discrimination, not only adapts to application scenarios with different annotation conditions, but also improves the recognition accuracy through a dual determination logic, and finally outputs the results with visual annotation, providing accurate data support for the flower thinning operation of fruit trees.

[0197] In summary, the automatic recognition method for the central flower and marginal flowers of fruit trees according to the present invention mainly includes the following steps: First, a target detection model is constructed and trained to identify inflorescence and / or flower targets in an image, and the coordinates of the flower detection box and its central pixel coordinates are obtained. For the division of inflorescence attribution, two implementation methods are proposed: One is to achieve the association by mapping the flower center coordinates into the pre-detected inflorescence box; the other is to use a clustering algorithm to perform unsupervised clustering on the flower detection boxes with the central coordinates as features, and dynamically divide the inflorescence categories. In the determination link of the central flower, two technical paths are proposed: Calculate the average value of the minimum bounding distances between each flower detection box and other boxes within the same inflorescence based on geometric features, and select the flower corresponding to the minimum value as the central flower; or construct a relationship prediction model through the scene graph generation technology, and determine the central flower and marginal flowers based on the inference of the position relationship. Finally, the inflorescence, central flower, and marginal flowers are distinguished through visual annotation, where the inflorescence annotation can be achieved through joint detection, and the inflorescence can also be inversely identified through the clustering result. Thus, the method provided by the present invention integrates target detection, spatial relationship analysis, and machine learning technologies, and realizes the automatic and accurate recognition of the inflorescence structure of fruit trees.

[0198] The present invention also provides a storage medium for storing a computer program of any one of the automatic recognition methods for the central flower and marginal flowers of fruit trees as Figure 1 , Figure 5 , Figure 7 described above. For example, computer program instructions, when executed by a computer, can, through the operation of the computer, call or provide the method and / or technical solution according to the present invention, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here. The program instructions for calling the method of the present invention may be stored in a fixed or removable storage medium, and / or be transmitted through a data stream in a broadcast or other signal-bearing medium and / or be stored in the storage medium of a computer device running according to the program instructions.

[0199] According to an embodiment of the present invention, the present invention also provides a Figure 10The electronic device 300 shown, the electronic device 300 may optionally include a storage medium 100 for storing a computer program and a processor 200 for executing the computer program. Wherein, when the computer program is executed by the processor 200, the automatic recognition method of the central flower and the marginal flower of any one of the above fruit trees is implemented, triggering the electronic device 300 to execute the methods and / or technical solutions based on the foregoing multiple embodiments, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here. It should be noted that the electronic devices in the embodiments of the present invention include mobile electronic devices and non-mobile electronic devices. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer, a netbook, or a personal digital assistant, etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present invention do not make specific limitations.

[0200] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the above steps or functions. Similarly, the software program (including related data structures) of the present invention can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and similar devices. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.

[0201] The present invention can be implemented on a computer as a computer-implemented method, or in dedicated hardware, or in a combination of both. The executable code or a part thereof for the method according to the present invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes non-temporary program code components stored on a computer-readable medium so as to execute the method according to the present invention when the program product is executed on a computer.

[0202] In an alternative embodiment, the computer program includes computer program code components suitable for executing all the steps of the method according to the present invention when the computer program runs on a computer. Optionally, the computer program is embodied on a computer-readable medium.

[0203] It should be noted that, in this text, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0204] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for automatically identifying central and peripheral flowers of fruit trees, characterized in that: Includes steps: Constructing and training an inflorescence and flower target detection model, wherein the target detection model is used to simultaneously detect inflorescence targets and flower targets in an image; Detect the image to be tested by using the target detection model, obtain the detection frame coordinates of all inflorescences and flowers in the image to be tested, and calculate the central pixel coordinates of each of the flowers; Mapping the center pixel coordinates to an inflorescence detection frame, and attributing the corresponding flower to the inflorescence corresponding to the inflorescence detection frame according to the inflorescence detection frame where the center pixel coordinates are located; For all flowers in the same inflorescence, respectively calculate the average value of the minimum boundary distance between each flower detection frame and other flower detection frames, determine the flower with the smallest average value as the central flower of the inflorescence, and determine the remaining flowers as side flowers; The detection frames corresponding to the inflorescence, the central flower and the side flowers in the image to be detected are visually marked and displayed distinguishably with different identifiers.

2. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 1, characterized in that: The construction and training of the inflorescence and flower target detection model includes: Constructing a data set, wherein the data set includes a plurality of training images, each of the training images includes an inflorescence and a flower; Annotating the inflorescence and the flowers in the training image in the data set with bounding boxes respectively; wherein each flower in the inflorescence is independently labeled as a flower category, and the inflorescence as a whole is labeled as an inflorescence category; The labeled data set is used to train a deep learning target detection model, where the target detection model is any one of FasterR-CNN, YOLO or SSD.

3. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 1, characterized in that: The calculation formula for the central pixel coordinates of the flower is: x c =(x2-x1) / 2,y c =(y2-y1) / 2; Among them, [x1, y1] is the coordinate of the upper left corner of the detection box, and [x2, y2] is the coordinate of the lower right corner of the detection box.

4. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 3, characterized in that: The detection frame is a rectangular frame; the calculation method for the minimum boundary distance between two different flower detection frames in the same inflorescence includes: Calculate the minimum distances dx and dy between two different flower detection frames in the x-axis and y-axis directions respectively; dx and dy are calculated based on the following formulas: dx= max(0,max(x 1_1 , x 1_2 )-min(x 2_1 , x 2_2 )); dy= max(0), max(y 1_1 , y 1_2 )-min(y 2_1 , y 2_2 )); The minimum boundary distance between two different flower detection frames is calculated using the Pythagorean theorem formula: ； Among them, the coordinates of the two different flower detection frames are [x 1_1 ,y 1_1 , x 2_1 ,y 2_1 ] and [x 1_2 ,y 1_2 , x 2_2 ,y 2_2 ].

5. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 4, characterized in that: The step of calculating the average value of the minimum boundary distances between each flower detection frame and other flower detection frames for all flowers in the same inflorescence, determining the flower with the smallest average value as the central flower of the inflorescence, and determining the remaining flowers as side flowers comprises: For all flowers in the same inflorescence, the sum of the minimum boundary distances between each flower detection frame and other flower detection frames is calculated respectively, and the sum of the minimum boundary distances is divided by the statistical number of times to obtain the corresponding average value of the minimum boundary distances; wherein the statistical number of times is the number of flowers in the inflorescence minus one; Loop through and compare the average values ​​of the minimum boundary distances corresponding to two flower detection frames in the same inflorescence, and retain the flower detection frame with the smaller average value until a target flower detection frame with the minimum average value is obtained; The flowers within the target flower detection frame are determined as central flowers, and the remaining flowers are determined as side flowers.

6. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 1, characterized in that: After the step of mapping the center pixel coordinates to an inflorescence detection frame, and attributing the corresponding flower to the inflorescence corresponding to the inflorescence detection frame according to the inflorescence detection frame where the center pixel coordinates are located, the method further includes: If there is only one flower detection frame in an inflorescence, the flower corresponding to the flower detection frame is determined as the central flower of the inflorescence.

7. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 1, characterized in that: The different marks are different colors and / or different shapes and / or different annotations.

8. A method for automatically identifying central and peripheral flowers of fruit trees, characterized in that: Includes steps: Constructing and training a flower target detection model, wherein the target detection model is used to detect flower targets in an image; The target detection model is used to detect the image to be tested, the detection frame coordinates of all flowers in the image to be tested are obtained, and the central pixel coordinates of each flower are calculated; Using a clustering algorithm to cluster the flower detection frames with the central pixel coordinates in the image to be detected as clustering features, so as to classify the inflorescence category to which each flower detection frame belongs; For all flowers in the same inflorescence, respectively calculate the average value of the minimum boundary distances between each flower detection frame and other flower detection frames, determine the flower with the smallest average value as the central flower of the inflorescence, and determine the remaining flowers as side flowers; The detection frames corresponding to the inflorescence, the central flower and the side flowers in the image to be detected are visually marked and displayed distinguishably with different identifiers.

9. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 8, characterized in that: The construction and training of the flower target detection model includes: Constructing a data set, wherein the data set includes a plurality of training images, each of the training images includes a flower; Annotate the flowers in the training image in the data set, use a bounding box to annotate the boundary of each flower, and mark the category as flower; The labeled data set is used to train a deep learning target detection model, where the target detection model is any one of FasterR-CNN, YOLO or SSD.

10. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 8, characterized in that: The calculation formula for the central pixel coordinates of the flower is: x c =(x2-x1) / 2,y c =(y2-y1) / 2; Among them, [x1, y1] is the coordinate of the upper left corner of the detection box, and [x2, y2] is the coordinate of the lower right corner of the detection box.

11. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 10, characterized in that: The flower detection frame is a rectangular frame; the calculation method for the minimum boundary distance between two different flower detection frames in the same inflorescence includes: Calculate the minimum distances dx and dy between two different flower detection frames in the x-axis and y-axis directions respectively; dx and dy are calculated based on the following formulas: dx= max(0,max(x 1_1 , x 1_2 )-min(x 2_1 , x 2_2 )); dy= max(0), max(y 1_1 , y 1_2 )-min(y 2_1 , y 2_2 )); The minimum boundary distance between two different flower detection frames is calculated using the Pythagorean theorem formula: ； Among them, the coordinates of the two different flower detection frames are [x 1_1 ,y 1_1 , x 2_1 ,y 2_1 ] and [x 1_2 ,y 1_2 , x 2_2 ,y 2_2 ].

12. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 11, characterized in that: The step of respectively calculating the average value of the minimum boundary distances between each flower detection frame and other flower detection frames for all flowers in the same inflorescence, determining the flower with the smallest average value as the central flower of the inflorescence, and determining the remaining flowers as side flowers comprises: For all flowers in the same inflorescence, the sum of the minimum boundary distances between each flower detection frame and other flower detection frames is calculated respectively, and the sum of the minimum boundary distances is divided by the statistical number of times to obtain the corresponding average value of the minimum boundary distances; wherein the statistical number of times is the number of flowers in the inflorescence minus one; Loop through and compare the average values ​​of the minimum boundary distances corresponding to two flower detection frames in the same inflorescence, and retain the flower detection frame with the smaller average value until a target flower detection frame with the minimum average value is obtained; The flowers within the target flower detection frame are determined as central flowers, and the remaining flowers are determined as side flowers.

13. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 8, characterized in that: After the step of clustering the flower detection frames using the clustering algorithm with the central pixel coordinates in the image to be detected as clustering features to divide the inflorescence category to which each flower detection frame belongs, the method further includes: If there is only one flower detection frame in an inflorescence, the flower corresponding to the flower detection frame is determined as the central flower of the inflorescence.

14. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 8, characterized in that: The different marks are different colors and / or different shapes and / or different annotations.

15. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 8, characterized in that: The clustering algorithm is a KMeans algorithm or a density peak clustering algorithm.

16. A method for automatically identifying central and peripheral flowers of fruit trees, characterized in that: Includes steps: Constructing and training a flower target detection model, wherein the target detection model is used to detect flower targets in an image; The target detection model is used to detect the image to be tested, the detection frame coordinates of all flowers in the image to be tested are obtained, and the central pixel coordinates of each flower are calculated; Using a clustering algorithm to cluster the flower detection frames with the central pixel coordinates in the image to be detected as clustering features, so as to classify the inflorescence category to which each flower detection frame belongs; A relationship prediction model is constructed and trained by scene graph generation technology, and the trained relationship prediction model is used to predict the position relationship of all flowers in the same inflorescence to determine the central flower and the side flowers in the inflorescence; The detection frames corresponding to the inflorescence, the central flower and the side flowers in the image to be detected are visually marked and displayed distinguishably with different identifiers.

17. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 16, characterized in that: The construction and training of the flower target detection model includes: Constructing a data set, wherein the data set includes a plurality of training images, each of the training images includes a flower; Annotate the flowers in the training image in the data set, use a bounding box to annotate the boundary of each flower, and mark the category as flower; The labeled data set is used to train a deep learning target detection model, where the target detection model is any one of FasterR-CNN, YOLO or SSD.

18. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 16, characterized in that: The calculation formula for the central pixel coordinates of the flower is: x c =(x2-x1) / 2,y c =(y2-y1) / 2; Among them, [x1, y1] is the coordinate of the upper left corner of the detection box, and [x2, y2] is the coordinate of the lower right corner of the detection box.

19. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 16, characterized in that: The relationship prediction model is constructed and trained by scene graph generation technology, and the trained relationship prediction model is used to predict the position relationship of all flowers in the same inflorescence to determine the central flower and the side flowers in the inflorescence. Perform relationship annotation on the center flower and the side flower in the training data to form relationship annotation data; The graph neural network is used as a relationship prediction network, each flower is regarded as a node, the node feature is the flower position feature, the edge is constructed based on the spatial position relationship between the flowers, and the position relationship features between the flowers are learned through the message passing mechanism of the graph neural network to build a relationship prediction model; Inputting the relationship annotation data into the relationship prediction model, and using the cross entropy loss as the loss function of the relationship prediction task for training, to obtain the trained relationship prediction model; Using the detection frame position information of all flowers in the same inflorescence in the image to be tested as the model input of the relationship prediction model, and obtaining the relationship graph structure output by the relationship prediction model; The relationship graph structure is mapped onto the image to be tested, and the central flower and the side flowers in the same inflorescence are identified and determined according to the display position of the relationship graph structure on the image to be tested.

20. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 16, characterized in that: After the step of clustering the flower detection frames using the clustering algorithm with the central pixel coordinates in the image to be detected as clustering features to divide the inflorescence category to which each flower detection frame belongs, the method further includes: If there is only one flower detection frame in an inflorescence, the flower corresponding to the flower detection frame is determined as the central flower of the inflorescence.

21. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 16, characterized in that: The different marks are different colors and / or different shapes and / or different annotations.

22. The method for automatically identifying central and peripheral flowers of fruit trees according to claim 16, characterized in that: The clustering algorithm is a KMeans algorithm or a density peak clustering algorithm.

23. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 22 is implemented.

24. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 22 is implemented.

Images