Fruit image processing method and device, electronic equipment and storage medium

Abstract

The invention provides a fruit image processing method and device, electronic equipment and a storage medium. The processing method comprises the following steps: through a trained target detection model, determining first branch attention areas corresponding to a plurality of target fruit branches; and determining a world movement distance of the image acquisition device based on the first position information and the second position information. And determining a first prediction area corresponding to the historical fruit branch based on the world movement distance of the image acquisition device, the first branch attention area corresponding to the historical fruit branch and the movement corresponding relation. And for the first branch attention region corresponding to each target fruit branch, based on the first branch attention region and each first prediction region, determining an overlapping degree between the first branch attention region and each first prediction region. And determining identity information of each target fruit branch based on the overlapping degree. According to the scheme, the accuracy of fruit branch counting is improved.

Description

Technical Field

[0001] This invention relates to the technical field of smart agriculture, and more specifically to a method for processing fruit images, a device for processing fruit images, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In recent years, with the continuous advancement of agricultural planting technology, the advantages of large-scale fruit cultivation have become increasingly significant. Fruit yield forecasting is crucial for growers in production planning, cost accounting, and market positioning. Theoretically, the yield of a particular fruit species can be indirectly estimated by identifying parameters such as the number of fruit branches. In actual planting scenarios, this typically relies on inspection robots to continuously photograph each fruit branch, and the number of branches is obtained by analyzing and processing the images. However, due to the complex fruit growth environment, often affected by factors such as changes in light, leaf shading, fruit clustering, and mutual shading, and the different photographing angles used to train the robot, overlapping images of different fruit branches can occur in the final images, resulting in low accuracy and poor reliability in determining the number of fruit branches.

[0003] Therefore, how to quickly and reliably divide different fruit branches in complex environments has become an urgent problem to be solved. Summary of the Invention

[0004] The present invention was proposed in view of the above-mentioned problems. The present invention provides a method for processing fruit images, a device for processing fruit images, an electronic device, a storage medium, and a computer program product.

[0005] According to one aspect of the present invention, a method for processing fruit images is provided. The method includes: based on a target fruit image comprising multiple target fruit branches, determining, through a trained target detection model, a first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image; for each historical fruit branch in at least one historical fruit branch, determining, based on first position information acquired by an image acquisition device when acquiring the target fruit image and second position information acquired by the image acquisition device when acquiring the historical fruit image corresponding to the historical fruit branch, the world movement distance of the image acquisition device corresponding to the historical fruit branch is determined, wherein the historical fruit branch is a target fruit branch in a historical fruit image that corresponds to identity information, and the identity information of different historical fruit branches is different; based on the world movement distance of the image acquisition device corresponding to the historical fruit branch and the second position information acquired by the image acquisition device when acquiring the historical fruit image corresponding to the historical fruit branch, ... Based on the corresponding first branch attention region in the historical fruit image and the movement correspondence, the first predicted region corresponding to the historical fruit branch in the target fruit image is determined. The movement correspondence is the correspondence between the world movement distance of the image acquisition device in the world coordinate system and the pixel movement distance of the pixel in the pixel coordinate system. For each target fruit branch in the target fruit image, based on the first branch attention region and the first predicted region corresponding to each historical fruit branch, the degree of overlap between the first branch attention region and each first predicted region is determined. Based on the degree of overlap between the first branch attention region corresponding to each target fruit branch and the first predicted region corresponding to each historical fruit branch, the identity information of each target fruit branch is determined.

[0006] For example, based on a target fruit image including multiple target fruit branches, a trained target detection model determines the first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image, including:

[0007] Based on a target fruit image including multiple original fruit branches, the second branch attention region corresponding to each original fruit branch in the target fruit image is determined by the trained target detection model.

[0008] For each of the multiple original fruit branches, based on the target depth image acquired by the image acquisition device for the multiple original fruit branches, the second branch interest region corresponding to the original fruit branch, and the pixel correspondence, the fruit branch depth information corresponding to the original fruit branch is determined. The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image. The fruit branch depth information corresponding to the original fruit branch is used to represent the distance between the original fruit branch and the image acquisition device.

[0009] The original fruit branches whose corresponding fruit branch depth information is greater than a preset depth threshold are filtered out, so that the filtered original fruit branches are used as target fruit branches, and the second branch attention area corresponding to the original fruit branch is used as the first branch attention area corresponding to the target fruit branch.

[0010] For example, based on the degree of overlap between the first branch attention region corresponding to each target fruit branch and the first prediction region corresponding to each historical fruit branch, the identity information of each target fruit branch is determined, including at least one of the following:

[0011] For each target fruit branch, if the overlap between the first branch attention area and the first prediction area corresponding to all historical fruit branches is less than a preset threshold, the identity information of the target fruit branch is determined to be different from the identity information of all historical fruit branches, and the target fruit branch is regarded as a new historical fruit branch.

[0012] For each target fruit branch, if there is at least one first prediction region corresponding to the first branch attention region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to the at least one first prediction region. The first prediction region corresponding to the first branch attention region is a first prediction region whose overlap with the first branch attention region is greater than a preset threshold.

[0013] For example, when the first branch of interest corresponds to at least one first prediction region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to each of the at least one first prediction region, including at least one of the following:

[0014] If there are multiple first prediction regions corresponding to the first branch attention region, and the identity information of the historical fruit branches corresponding to the multiple first prediction regions is not determined based on the target fruit image to which the first branch attention region belongs, the identity information of the target fruit branch is determined to be empty or composite information, wherein the composite information includes the identity information of the historical fruit branches corresponding to the multiple first prediction regions.

[0015] If the first predicted region corresponding to the first branch attention region is the same as at least one of the first predicted regions corresponding to other first branch attention regions in the target fruit image, it is determined that the identity information of the target fruit branch corresponding to the first branch attention region is different from the identity information of the target fruit branch corresponding to other first branch attention regions, and is also different from the identity information of all historical fruit branches. The identity information of the historical fruit branch corresponding to the target predicted region is deleted, wherein the target predicted region is the first predicted region corresponding to the first branch attention region.

[0016] If the first branch attention area corresponds to a first prediction area, and the first prediction area is different from the first branch attention area, the identity information of the target fruit branch corresponding to the first branch attention area is determined to be the identity information of the historical fruit branch corresponding to the first prediction area.

[0017] For example, the processing method further includes:

[0018] Based on the target fruit image, the first fruit interest region corresponding to each target fruit in the target fruit image is determined by the trained instance segmentation model;

[0019] For each target fruit branch, the total number of target fruits included in that target fruit branch is determined based on the total number of the first fruit attention regions corresponding to all target fruits in that target fruit branch.

[0020] For example, based on the target fruit image, the trained instance segmentation model determines the first fruit interest region corresponding to each target fruit in each target fruit branch within the target fruit image, including:

[0021] Based on a target fruit image including multiple original fruit branches, the second fruit interest region corresponding to each original fruit in each original fruit branch in the target fruit image is determined by the trained instance segmentation model.

[0022] For each original fruit in each original fruit branch, based on the target depth image acquired by the image acquisition device for multiple original fruit branches, the second fruit interest region corresponding to the original fruit, and the pixel correspondence, the fruit depth information corresponding to the original fruit is determined. The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image. The fruit depth information corresponding to the original fruit is used to represent the distance between the original fruit and the image acquisition device.

[0023] The original fruits whose corresponding depth information is greater than a preset depth threshold are filtered out, so that the filtered original fruits are used as target fruits, and the second fruit interest region corresponding to the original fruit is used as the first fruit interest region corresponding to the target fruit.

[0024] For example, the total number of target fruits included in the target fruit branch is determined based on the total number of the first fruit attention regions corresponding to each of the target fruits in the target fruit branch, including at least one of the following:

[0025] For each historical fruit branch that has the same identity information as the target fruit branch, the maximum of the following will be used as the number of target fruits corresponding to the target fruit branch: the number of first fruit attention areas corresponding to the target fruit branch, and the number of first fruit attention areas corresponding to historical fruit branches with the same identity information as the target fruit branch.

[0026] For each historical fruit branch that does not have the same identity information as the target fruit branch, the number of the first fruit attention regions corresponding to the target fruit branch is taken as the number of target fruits corresponding to the target fruit branch.

[0027] According to another aspect of the present invention, a fruit image processing apparatus is also provided, the apparatus comprising: a first branch interest region determination module, a world movement distance determination module, a first prediction region determination module, an overlap degree determination module, and an identity information determination module.

[0028] The first branch attention region determination module is used to determine the first branch attention region corresponding to each of the multiple target fruit branches in the target fruit image based on a target fruit image including multiple target fruit branches, using a trained target detection model; the world movement distance determination module is used to determine the world movement distance of the image acquisition device corresponding to each of the at least one historical fruit branches, based on the first position information of the image acquisition device when acquiring the target fruit image and the second position information of the image acquisition device when acquiring the historical fruit image corresponding to the historical fruit branch, wherein the historical fruit branch is a target fruit branch in the historical fruit image that corresponds to identity information, and the identity information of different historical fruit branches is different; the first prediction region determination module is used to determine the world movement distance of the image acquisition device corresponding to each of the at least one historical fruit branches, based on the world movement distance of the image acquisition device corresponding to the historical fruit branch and the second position information of the image acquisition device when acquiring the historical fruit image corresponding to the historical fruit branch, ... The system uses a first branch interest region corresponding to a historical fruit branch in its corresponding historical fruit image and a movement correspondence to determine the first predicted region corresponding to that historical fruit branch in the target fruit image. The movement correspondence is the relationship between the world movement distance of the image acquisition device in the world coordinate system and the pixel movement distance of the pixel in the pixel coordinate system. An overlap determination module is used to determine the degree of overlap between the first branch interest region corresponding to each target fruit branch in the target fruit image and each first predicted region corresponding to each historical fruit branch, based on the first branch interest region and the first predicted region corresponding to each historical fruit branch. An identity information determination module is used to determine the identity information of each target fruit branch based on the degree of overlap between the first branch interest region corresponding to each target fruit branch and the first predicted region corresponding to each historical fruit branch.

[0029] According to another aspect of the present invention, an electronic device is also provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the above-described fruit image processing method.

[0030] According to another aspect of the present invention, a storage medium is also provided, which stores a computer program / instructions, which, when executed, are used to perform the above-described fruit image processing method.

[0031] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when run, are used to execute the above-described fruit image processing method.

[0032] According to the above-described scheme of the present invention, based on a target fruit image including multiple target fruit branches, a trained target detection model can be used to determine the first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image. Then, for each historical fruit branch among at least one historical fruit branch, based on the first position information when the image acquisition device acquired the target fruit image and the second position information when the image acquisition device acquired the historical fruit image corresponding to that historical fruit branch, the world movement distance of the image acquisition device corresponding to that historical fruit branch is determined. Next, for each historical fruit branch among at least one historical fruit branch, based on the world movement distance of the image acquisition device corresponding to that historical fruit branch, the first branch interest region corresponding to that historical fruit branch in its corresponding historical fruit image, and the movement correspondence, the first prediction region corresponding to that historical fruit branch in the target fruit image is determined. Finally, for each first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image, based on the first branch interest region and the first prediction region corresponding to each historical fruit branch, the degree of overlap between the first branch interest region and each first prediction region is determined. Finally, based on the degree of overlap between the first branch interest region corresponding to each target fruit branch and the first prediction region corresponding to each historical fruit branch, the identity information of each target fruit branch is determined. The above scheme can accurately determine each target fruit branch in the target fruit image through a target detection model, effectively reducing the adverse effects of random fruit branch growth posture and mutual occlusion between fruit branches on localization, providing a reliable basis for subsequently determining the identity information of each target fruit branch. Furthermore, based on the world movement distance of the image acquisition device corresponding to the historical fruit branch, the first branch interest region corresponding to the historical fruit branch in its corresponding historical fruit image, and the movement correspondence, the above scheme determines the first prediction region corresponding to the historical fruit branch in the target fruit image. Then, based on the degree of overlap between the first branch interest region corresponding to each target fruit branch and the first prediction region corresponding to the historical fruit branch, the identity information of each target fruit branch is determined. This effectively reduces the occurrence of situations where the same fruit branch is misidentified as different fruit branches due to large differences in imaging effects caused by different shooting angles, achieving accurate tracing of the same fruit branch under different shooting angles, and effectively reducing the occurrence of duplicate counting caused by fruit branches overlapping vertically and laterally to form new shapes. Furthermore, identifying fruit branches based on unique identity information ensures that each fruit branch is counted only once, fundamentally reducing duplicate identification and counting, and improving the accuracy of fruit branch counting. Attached Figure Description

[0033] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0034] Figure 1 A schematic flowchart of a method for processing fruit images according to an embodiment of the present invention is shown;

[0035] Figure 2 A schematic diagram of a fruit image according to an embodiment of the present invention is shown;

[0036] Figure 3 A schematic diagram of a user interface according to an embodiment of the present invention is shown;

[0037] Figure 4 A schematic block diagram of a fruit image processing apparatus according to an embodiment of the present invention is shown;

[0038] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0040] In practical planting scenarios, fruit yield estimation largely relies on inspection robots equipped with image acquisition devices to continuously take pictures. The collected image information is then analyzed to obtain parameters such as the number of fruit branches in the planting area. However, because the same image acquisition device can capture images of multiple fruit branches from different angles, the resulting images can vary significantly. Without effective algorithms to process the image information, the accuracy of the final fruit branch count can drop drastically, failing to meet users' requirements for precise fruit yield estimation.

[0041] In related technologies, the partial overlap of different fruit branches in the image creates new visual morphologies. This makes it difficult for inspection robots to accurately distinguish individual fruit branches during lateral imaging analysis, leading to duplicate counting in the lateral direction. Furthermore, the growth state of fruit branches is significantly random; different fruit branches vary in size, color, and growth posture. Even for the same fruit branch, the imaging effect of the image acquisition device can change drastically under different poses. Therefore, simply comparing the similarity of fruit branches for differentiation cannot effectively distinguish the same fruit branch under different poses, potentially misclassifying the same fruit branch as multiple different branches. This results in the same fruit branch being repeatedly identified, thus affecting the accuracy of fruit yield predictions.

[0042] To at least partially solve the above problems, embodiments of the present invention provide a method for processing fruit images. Combined with... Figure 1 , Figure 1 A schematic flowchart illustrating a method for processing fruit images according to an embodiment of the present invention is shown. Figure 1 As shown, the above-mentioned fruit image processing method may include steps S110 to S150.

[0043] In step S110, based on the target fruit image including multiple target fruit branches, the first branch attention region corresponding to each of the multiple target fruit branches in the target fruit image is determined by the trained target detection model.

[0044] The target fruit branch can be a bunch of fruit of a specified category in the planting area. For example, in the current scenario, if the specified category in the planting area is cherry tomato, then the target fruit branch can be a cherry tomato branch (e.g., a bunch of cherry tomatoes). As another example, in the current scenario, if the specified category in the planting area is grape, then the target fruit branch can be a grape branch (e.g., a bunch of grapes).

[0045] The target fruit image can be an image of a planting area where multiple target fruit branches exist. For example, such as... Figure 2 As shown, Figure 2 A schematic diagram of a fruit image according to an embodiment of the present invention is shown. When the target fruit is a cherry tomato, a color image of the cherry tomato growing area can be acquired using an image acquisition device (e.g., an RGB color camera, a separate camera integrating RGB and depth, a high dynamic range camera, a camera integrated into a mobile robot platform, etc.) as the target fruit image. It is understood that the image acquisition device can be any image acquisition device capable of acquiring color images.

[0046] Object detection models can include single-shot recognition models (i.e., You Only Look Once, YOLO model), region-based convolutional neural networks (i.e., Region Convolutional Neural Network, R-CNN model), single-shot multi-box detectors (i.e., Single Shot MultiBox Detector, SSD), etc.

[0047] The trained object detection model can determine the first branch region of interest for each target fruit branch in a target fruit image containing multiple target fruit branches. For example, the target fruit image can be directly input into the trained object detection model. The trained model can then determine the first branch region of interest for each target fruit branch based on the input target fruit image.

[0048] This invention provides a training method for an object detection model for reference. A first training dataset is obtained. The object detection model is iteratively trained using the first training dataset, and the model parameters are adjusted using a loss function until training is complete. The first training dataset includes first training images for multiple training fruit branches. The label of each first training image can be the corresponding training branch region for each of the multiple training fruit branches in that first training image. It is understood that the corresponding training branch regions for each of the multiple training fruit branches in the first training image can be marked with rectangular boxes. The conditions for completing the training can include the loss value calculated by the loss function stabilizing, the number of iterations reaching a preset number, etc.

[0049] The training fruit branch must have the same fruit category as the target fruit branch. For example, if the target fruit branch corresponds to the fruit category of grapes, then the training fruit branch should also correspond to the fruit category of grapes. If the target fruit branch corresponds to the fruit category of bananas, then the training fruit branch should also correspond to the fruit category of bananas. If the target fruit branch corresponds to the fruit category of tomato, then the training fruit branch should also correspond to the fruit category of tomato.

[0050] The training branch region corresponding to the training fruit branch represents the position of that training fruit branch in the first training image. The shape of the training branch region corresponding to the training fruit branch can be a rectangle, etc. Taking a rectangular shape as an example, this training branch region can be the smallest bounding rectangle of the region corresponding to that training fruit branch in the first training image.

[0051] In step S120, for each historical fruit branch in at least one historical fruit branch, the world movement distance of the image acquisition device corresponding to the historical fruit branch is determined based on the first position information when the image acquisition device acquires the target fruit image and the second position information when the image acquisition device acquires the historical fruit image corresponding to the historical fruit branch.

[0052] Historical fruit branches are target fruit branches with corresponding identity information in historical fruit images. Different historical fruit branches have different identity information. For example, a historical fruit image can be an image of a target fruit branch acquired before the aforementioned target fruit image is acquired, and the target fruit branch in the historical fruit image (i.e., the aforementioned historical fruit branch) can be located in the same planting area as the target fruit branch in the target fruit image. Taking a bunch of cherry tomatoes as an example, bunches of cherry tomatoes located in the same row can be considered as target fruit branches located in the same planting area. The image acquisition device moves from the leftmost end to the rightmost end of a row, acquiring images of the cherry tomato bunches in that row, resulting in a fruit video containing multiple frames of fruit images. If the nth frame of this fruit video is considered the target fruit image, then frames n-1 to nm can be considered as the historical fruit images corresponding to that target fruit image. Here, n and m are positive integers, and m is less than n. The value of m can be determined based on the moving speed of the image acquisition device. For example, the value of m can be negatively correlated with the moving speed of the image acquisition device; that is, the faster the image acquisition device moves, the smaller the value of m, and the slower the image acquisition device moves, the larger the value of m. This is because the faster the image acquisition device moves, the farther the image acquisition time corresponding to the historical fruit image is from the target acquisition time, and the lower the probability that the historical fruit image contains a branch of the target fruit image. The aforementioned target acquisition time is the image acquisition time corresponding to the target fruit image.

[0053] Different historical fruit branches can correspond to different identity information to distinguish them. For example, if a historical fruit image corresponds to three historical fruit branches, then each historical fruit branch can be assigned different identity information. Specifically, the identity information corresponding to the three historical fruit branches mentioned above can be L1, L2, and L3, that is, historical fruit branch L1, historical fruit branch L2, and historical fruit branch L3. It is understood that the above example uses both English letters and numbers to represent the identity information corresponding to the historical fruit branches. At least one of the following methods can also be used to represent the identity information corresponding to the historical fruit branches: numbers, text, graphics, letters, colors, etc. The specific form of the identity information can be determined according to the actual situation.

[0054] The first position information when the image acquisition device acquires an image of the target fruit, and the second position information when the image acquisition device acquires an image of the corresponding historical fruit branch, can be position information in a world coordinate system. The position information in the world coordinate system may include the three-dimensional coordinates (x, y, z) of the image acquisition device, and / or its orientation (e.g., yaw angle, pitch angle, roll angle, etc.), to represent the spatial state of the image acquisition device at the moment of imaging. It is understood that the image acquisition device can determine the aforementioned first and second position information based on its onboard positioning and attitude perception system.

[0055] This invention provides an example for reference. Here, we take the case where the image acquisition device's yaw angle, pitch angle, and roll angle remain unchanged when acquiring images of the target fruit and historical fruit, and its spatial motion can be considered as pure translational motion. The three-dimensional coordinates in the first position information when the image acquisition device acquires the target fruit image are (x1, y1, z1), and the three-dimensional coordinates in the second position information when the image acquisition device acquires the historical fruit image corresponding to historical fruit branch L1 are (x2, y2, z2). Based on the above three-dimensional coordinates (x1, y1, z1) and (x2, y2, z2), the world movement distance of the image acquisition device corresponding to historical fruit branch L1 can be calculated to be d (i.e., )rice.

[0056] In step S130, for each historical fruit branch in at least one historical fruit branch, based on the world movement distance of the image acquisition device corresponding to the historical fruit branch, the first branch interest area corresponding to the historical fruit branch in its corresponding historical fruit image, and the movement correspondence, the first prediction area corresponding to the historical fruit branch in the target fruit image is determined.

[0057] The motion correspondence is the relationship between the world movement distance of the image acquisition device in the world coordinate system and the pixel movement distance of the pixel in the pixel coordinate system. This motion correspondence can be determined based on parameters such as the camera intrinsic and extrinsic parameters of the image acquisition device, the distance between the image acquisition device and the target fruit branch, and the angles between the movement direction of the image acquisition device and the coordinate axes of the world coordinate system. For example, when the image acquisition device is acquiring an image of the target fruit branch, it moves horizontally along the x-axis parallel to the world coordinate system, and the camera intrinsic and extrinsic parameters remain unchanged. The process of determining the motion correspondence can be as follows: the image acquisition device acquires images at multiple discrete positions along its movement direction (here, positions P1, P2, and P3 are used as examples). The selection rule for these discrete positions is that at any position, the image acquired by the image acquisition device contains a region corresponding to the same specified feature point. This specified feature point can be a point on the same target fruit branch, which must be unobstructed and not easily deformed. Images acquired by the image acquisition device at any two different positions are considered as a set of images. Taking positions P1, P2, and P3 as examples, three sets of images can be obtained: the images acquired by the image acquisition device at positions P1 and P2 (hereinafter referred to as the first set of images), the images acquired by the image acquisition device at positions P1 and P3 (hereinafter referred to as the second set of images), and the images acquired by the image acquisition device at positions P2 and P3 (hereinafter referred to as the third set of images). For the first set of images, the pixel coordinates of the feature point in the image corresponding to position P1 and the pixel coordinates of the feature point in the image corresponding to position P2 can be determined. Then, based on the difference between the horizontal and vertical coordinates of the two pixel coordinates and the Pythagorean theorem, the pixel movement distance of the feature point in the pixel coordinate system in the first set of images can be determined. Similarly, the pixel movement distance of the feature point in the pixel coordinate system in the second and third sets of images can be obtained. Based on the distance between two corresponding locations in each of the three image sets (equivalent to the world movement distance of the image acquisition device) and the pixel movement distance for each image set, a linear relationship can be fitted using the least squares method to determine the movement correspondence. It is understood that the above is merely an illustrative example. In practice, the pixel movement distance can be calibrated using parameters such as focal length from the camera's intrinsic parameters to ensure the accuracy of the movement correspondence at different distances. Furthermore, selecting more location points and using more precise algorithms can also ensure the accuracy of the determined movement correspondence.

[0058] This invention provides an example for reference. Historical Fruit Branch L aThe first branch of interest in the corresponding historical fruit image is a rectangular region with a minimum pixel x-coordinate of 1, a maximum pixel x-coordinate of 25, a minimum pixel y-coordinate of 1, and a maximum pixel y-coordinate of 25. If the image acquisition device acquires the historical fruit branch L... a After obtaining the target fruit image by shifting it b meters to the left from the corresponding historical fruit image, the historical fruit branch L can be determined based on the shift correspondence. a The first predicted region in the target fruit image, and the range of its horizontal and vertical coordinate values. If the maximum and / or minimum horizontal coordinate values ​​of the first predicted region exceed the range of horizontal coordinate values ​​of the target fruit image, then the maximum and minimum horizontal coordinate values ​​of the first predicted region are determined based on the range of horizontal coordinate values ​​of the target fruit image (e.g., the horizontal coordinate values ​​in the first predicted region that exceed the range of horizontal coordinate values ​​of the target fruit image are truncated to the boundary of the target fruit image). If the maximum and / or minimum vertical coordinate values ​​of the first predicted region exceed the range of vertical coordinate values ​​of the target fruit image, then the maximum and minimum vertical coordinate values ​​of the first predicted region are determined based on the range of vertical coordinate values ​​of the target fruit image (e.g., the vertical coordinate values ​​in the first predicted region that exceed the range of vertical coordinate values ​​of the target fruit image are truncated to the boundary of the target fruit image). Here, we take the example of the maximum horizontal coordinate value of the first prediction region exceeding the horizontal coordinate value range of the target fruit image as an example. According to the movement correspondence, the minimum pixel horizontal coordinate value of the first prediction region is 250, the maximum pixel horizontal coordinate value is 274, and the horizontal coordinate value range of the target fruit image is 0 to 255. Therefore, the maximum pixel horizontal coordinate of the first prediction region should be the maximum horizontal coordinate value of the target fruit image, that is, 255.

[0059] In step S140, for each of the multiple target fruit branches, the degree of overlap between the first branch attention region and each first prediction region is determined based on the first branch attention region and the first prediction region corresponding to each historical fruit branch in the target fruit image.

[0060] This invention provides an example for reference. Taking the first predicted region corresponding to the historical fruit branch L1, and the degree of overlap between the first branch attention region and the first predicted region as the ratio of the total number of pixels in the overlapping region of the two regions (i.e., the first branch attention region and the first predicted region) to the total number of pixels in the region with the smallest area among the two regions, as an example, if the minimum abscissa value of the first predicted region corresponding to the historical fruit branch L1 is 100, the maximum abscissa value is 140, the minimum ordinate value is 100, and the maximum ordinate value is 140, and the minimum abscissa value of the first branch attention region corresponding to the target fruit branch G1 is 80, the maximum abscissa value is 120, the minimum ordinate value is 80, and the maximum ordinate value is 120, then it can be determined that the overlapping region between the first predicted region and the first branch attention region is a rectangular region with an abscissa value greater than or equal to 100 and less than or equal to 120, and a ordinate value greater than or equal to 100 and less than or equal to 120. Furthermore, it can be determined that there are (120-100+1)*(120-100+1) pixels, i.e., 441 pixels, within this overlapping region. The overlap between the first branch interest region and the first prediction region is (441 / 1681), approximately 26.234%. Here, 1681 is the total number of pixels in the first branch interest region or the first prediction region, which is (120-80+1)*(120-80+1).

[0061] In step S150, the identity information of each target fruit branch is determined based on the degree of overlap between the first branch attention region corresponding to each target fruit branch and the first prediction region corresponding to each historical fruit branch.

[0062] This invention provides an example for reference. Taking the overlap between the target fruit branch G1 and the first prediction region corresponding to the historical fruit branch L1 as 26.234%, and the overlap between the target fruit branch G1 and the first prediction region corresponding to the historical fruit branch L2 as 30%, the above overlap levels can be compared with preset thresholds. If there is an overlap level greater than the preset threshold, the identity information of the historical fruit branch corresponding to that overlap level can be used as the identity information of the target fruit branch, which is equivalent to treating the target fruit branch and the historical fruit branch as the same fruit branch. If the determined overlap levels are all less than the above preset threshold, it is determined that the identity information of the target fruit branch is different from the identity information of all historical fruit branches, and the target fruit branch is considered a new fruit branch. Taking a preset threshold of 95% as an example, as seen above, if the overlap level between the target fruit branch G1 and the historical fruit branches L1 and L2 is less than the above preset threshold, then an identity information different from the identity information of all historical fruit branches should be determined for the target fruit branch G1. It is understandable that the aforementioned preset threshold can be negatively correlated with the moving speed of the image acquisition device; that is, the faster the image acquisition device moves, the smaller the preset threshold value. When the image acquisition device moves slowly, the offset of the first branch interest region corresponding to the same fruit branch in adjacent fruit images is small. To avoid identifying different fruit branches as the same fruit branch, the preset threshold can be appropriately increased. Conversely, when the image acquisition device moves quickly, the offset of the first branch interest region corresponding to the same fruit branch in adjacent fruit images is large. To avoid identifying the same fruit branch as different fruit branches, the preset threshold can be appropriately decreased.

[0063] It is also understandable that, in addition to recording the identity information of each historical fruit branch, it can also record the number of at least one historical fruit image containing the historical fruit branch with the same identity information, as well as the location information of the first branch attention area in each historical fruit image.

[0064] Furthermore, it is understandable that if the target fruit image is the first fruit image obtained and there are no historical fruit branches, then the identity information (e.g., L1, L2, and so on) is automatically assigned according to the order of the horizontal coordinate of the target fruit branch in the image from small to large and the vertical coordinate from large to small. The identity information can also be manually adjusted by the user.

[0065] Furthermore, it can be understood that the total number of fruit branches in the currently acquired image can be determined based on the identity information corresponding to all historical fruit branches. For example, the total number of identity information can be used as the total number of fruit branches in the currently acquired image. Specifically, if there are identity information L1, identity information L2, and identity information L3, then the total number of fruit branches in the currently acquired image can be determined to be 3.

[0066] According to the above-described scheme of the present invention, based on a target fruit image including multiple target fruit branches, a trained target detection model can be used to determine the first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image. Then, for each historical fruit branch among at least one historical fruit branch, based on the first position information of the image acquisition device when acquiring the target fruit image and the second position information of the image acquisition device when acquiring the historical fruit image corresponding to that historical fruit branch, the world movement distance of the image acquisition device corresponding to that historical fruit branch is determined. Next, for each historical fruit branch among at least one historical fruit branch, based on the world movement distance of the image acquisition device corresponding to that historical fruit branch, the first branch interest region corresponding to that historical fruit branch in its corresponding historical fruit image, and the movement correspondence, the first prediction region corresponding to that historical fruit branch in the target fruit image is determined. Finally, for each first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image, based on the first branch interest region and the first prediction region corresponding to each historical fruit branch, the degree of overlap between the first branch interest region and each first prediction region is determined. Finally, based on the degree of overlap between the first branch interest region corresponding to each target fruit branch and the first prediction region corresponding to each historical fruit branch, the identity information of each target fruit branch is determined. The above scheme can accurately determine each target fruit branch in the target fruit image through a target detection model, effectively reducing the adverse effects of random fruit branch growth posture and mutual occlusion between fruit branches on localization, providing a reliable basis for subsequently determining the identity information of each target fruit branch. Furthermore, based on the world movement distance of the image acquisition device corresponding to the historical fruit branch, the first branch interest region corresponding to the historical fruit branch in its corresponding historical fruit image, and the movement correspondence, the above scheme determines the first prediction region corresponding to the historical fruit branch in the target fruit image. Then, based on the degree of overlap between the first branch interest region corresponding to each target fruit branch and the first prediction region corresponding to the historical fruit branch, the identity information of each target fruit branch is determined. This effectively reduces the occurrence of situations where the same fruit branch is misidentified as different fruit branches due to large differences in imaging effects caused by different shooting angles, achieving accurate tracing of the same fruit branch under different shooting angles, and effectively reducing the occurrence of duplicate counting caused by fruit branches forming new shapes through horizontal overlap. Furthermore, identifying fruit branches based on unique identity information ensures that each fruit branch is counted only once, fundamentally reducing duplicate identification and counting, and improving the accuracy of fruit branch counting.

[0067] For example, step S110 above, based on a target fruit image including multiple target fruit branches, determines the first branch attention region corresponding to each of the multiple target fruit branches in the target fruit image through a trained target detection model, including steps S111 to S113.

[0068] In step S111, based on the target fruit image including multiple original fruit branches, the second branch attention region corresponding to each of the multiple original fruit branches in the target fruit image is determined by the trained target detection model.

[0069] The original fruit branch and the target fruit branch must have the same fruit category. For example, if the target fruit branch corresponds to the grape category, then the original fruit branch must also correspond to the grape category. If the target fruit branch corresponds to the banana category, then the original fruit branch must also correspond to the banana category. If the target fruit branch corresponds to the cherry tomato category, then the original fruit branch must also correspond to the cherry tomato category.

[0070] A target fruit image that includes multiple primary fruit branches can be an image of a planting area where multiple primary fruit branches exist. For example, such as... Figure 2 As shown, when the target fruit branch is a cherry tomato, a color image of the cherry tomato planting area can be acquired using an image acquisition device (e.g., an RGB color camera, a high dynamic range camera, a camera integrated into a mobile robot platform, etc.) as the target fruit image including multiple original fruit branches. It is understood that the image acquisition device can be any image acquisition device capable of acquiring color images. For example, a depth camera can be used to simultaneously acquire the target fruit image including multiple original fruit branches, and the target depth image acquired for the multiple original fruit branches in step S112 below.

[0071] The process for determining the region of interest for the second branch described above can be found in step S110 above. This invention will not be elaborated upon here.

[0072] In step S112, for each of the multiple original fruit branches, the fruit branch depth information corresponding to the original fruit branch is determined based on the target depth image acquired by the image acquisition device for the multiple original fruit branches, the second branch interest region corresponding to the original fruit branch, and the pixel correspondence.

[0073] In one example, the target depth image acquired for multiple original fruit branches can be obtained by a depth camera as described in step S111 above. If the image acquisition device acquiring the target fruit image including multiple original fruit branches is not a depth camera, the depth camera can be placed at the same spatial position (e.g., with the same height, pitch angle, and yaw angle) as the image acquisition device when acquiring the target fruit image including multiple original fruit branches. Joint calibration (calibrating the extrinsic parameter matrix and pixel mapping relationship) should be performed on the color image acquisition device and the depth camera to ensure that the target depth image acquired by the depth camera is strictly aligned with the original fruit image at the pixel level. The focal length of the depth camera must be consistent with that of the color image acquisition device to ensure imaging scale matching.

[0074] In another example, a deep learning model can be used to determine the depth value of each pixel in a target fruit image that includes multiple original fruit branches, thereby generating a target depth image for the multiple original fruit branches.

[0075] The pixel correspondence refers to the pixel correspondence between the target fruit image and the target depth image. The fruit branch depth information corresponding to the original fruit branch is used to represent the distance between the original fruit branch and the image acquisition device. For example, the pixel correspondence can be a one-to-one correspondence. Specifically, for example, the size of the target fruit image can be the same as the size of the target depth image, then the pixels in the target fruit image can correspond one-to-one with the pixels in the target depth image.

[0076] The depth information of the original fruit branch can be used to represent the average distance from the original fruit branch to the image acquisition device. For example, the region (hereinafter referred to as the depth region) corresponding to the second branch of interest of the original fruit branch can be determined in the target depth image based on this region. The depth information of the original fruit branch is then determined based on the depth value of each pixel in the depth region corresponding to the original fruit branch. Specifically, the pixel coordinates of the second branch of interest of the original fruit branch in the fruit image are (1,1), (1,2), (2,1), and (2,2). In the case of a one-to-one correspondence between the pixels, the pixel coordinates of the depth region corresponding to the original fruit branch are also (1,1), (1,2), (2,1), and (2,2). The depth information of the original fruit branch can be calculated based on the depth values ​​of the pixels in the depth region corresponding to the original fruit branch. Specifically, the maximum depth value among the depth values ​​of the pixels in the depth region corresponding to the original fruit branch can be used as the depth information of the original fruit branch. Another example is using the average depth value of the pixels in the depth region corresponding to the original fruit branch as the depth information of the original fruit branch. For example, the depth value of the pixel at the center point of the depth region corresponding to the original fruit branch can be used as the depth information of that original fruit branch. Alternatively, multiple pixels can be selected within the depth region corresponding to the original fruit branch, and the average of their depth values ​​can be used as the depth information of that original fruit branch. It is understood that the selected pixels can be chosen based on the specific circumstances.

[0077] In step S113, original fruit branches whose corresponding fruit branch depth information is greater than a preset depth threshold are filtered out, so that the filtered original fruit branches are used as target fruit branches, and the second branch attention area corresponding to the original fruit branch is used as the first branch attention area corresponding to the target fruit branch.

[0078] This invention provides an example for reference. Multiple original fruit branches include original fruit branch A', original fruit branch B', and original fruit branch C'. If the depth information of the fruit branch corresponding to original fruit branch C' is greater than a preset depth threshold, and the depth information of the fruit branches corresponding to original fruit branches A' and B' is less than or equal to the preset depth threshold, original fruit branch C' can be filtered out, and original fruit branches A' and B' can be selected as target fruit branches. The second branch attention region corresponding to original fruit branch A' and the second branch attention region corresponding to original fruit branch B' are each designated as a first branch attention region. It is understood that the aforementioned preset depth threshold can be set by the user according to actual circumstances.

[0079] According to the above-described scheme of the present invention, based on a target fruit image including multiple original fruit branches, a trained target detection model can be used to determine the second branch interest region corresponding to each of the multiple original fruit branches in the target fruit image. For each of the multiple original fruit branches, the fruit branch depth information corresponding to the original fruit branch is determined based on the target depth image acquired by the image acquisition device for the multiple original fruit branches, the second branch interest region corresponding to the original fruit branch, and the pixel correspondence. Then, original fruit branches whose corresponding fruit branch depth information is greater than a preset depth threshold are filtered out, so that the filtered original fruit branches are taken as target fruit branches, and the second branch interest region corresponding to the original fruit branch is taken as the first branch interest region corresponding to the target fruit branch. The above scheme can filter multiple original fruit branches corresponding to the target fruit image according to the depth information to determine the target fruit branch, realizing distance-aware filtering of fruit branches in the planting area. This can effectively reduce the impact of occlusion, background interference, or exceeding the reasonable observation range (such as too far, ground clutter, crosstalk between adjacent plants) on the detection results, which is beneficial to improving the accuracy of fruit branch identification. Based on this, counting and tracking only the branches of the target fruit within the effective operating depth range can avoid duplicate statistics and overestimation of yield, thereby improving the reliability of the yield prediction of the target fruit.

[0080] For example, step S150 above, which determines the identity information of each target fruit branch based on the degree of overlap between the first branch attention area corresponding to each target fruit branch and the first prediction area corresponding to each historical fruit branch, includes step S151a.

[0081] In step S151a, for each target fruit branch, if the overlap between the first branch attention area and the first prediction area corresponding to each of the historical fruit branches is less than a preset threshold, it is determined that the identity information of the target fruit branch is different from the identity information of each of the historical fruit branches, and the target fruit branch is regarded as a new historical fruit branch.

[0082] This invention provides an example for reference. Taking a target fruit image as an example, it includes the first branch attention region corresponding to target fruit branch G1, the first branch attention region corresponding to target fruit branch G2, the first branch attention region corresponding to target fruit branch G3, and the first branch attention region corresponding to target fruit branch G4, and there are three historical fruit branches (e.g., the aforementioned historical fruit branch L1, historical fruit branch L2, and the historical fruit branch with identity information L3). Here, we use target fruit branch G1 as an example. For target fruit branch G1, according to the content in steps S140 and S150 above, the overlap between the first branch attention region corresponding to target fruit branch G1 and the first predicted region corresponding to historical fruit branch L1 is determined to be 26.234%, and the overlap between it and the first predicted region corresponding to historical fruit branch L2 is 30%. If the overlap between the first branch attention region corresponding to target fruit branch G1 and the first predicted region corresponding to historical fruit branch L3 (i.e., the historical fruit branch with identity information L3) is 10%. When the preset threshold is 95%, it can be determined that the overlap between the first branch attention region corresponding to the target fruit branch G1 and the first prediction region corresponding to the three historical fruit branches is less than the preset threshold. Therefore, it can be determined that the target fruit branch G1 is a different fruit branch from the three historical fruit branches. The target fruit branch G1 can be assigned completely different identity information than the three historical fruit branches. For example, the identity information of the target fruit branch G1 can be L4, and the target fruit branch G1 can be regarded as a new historical fruit branch, that is, a historical fruit branch with identity information L4. It should be understood that the above identity information is only for illustrative reference, and it is sufficient to ensure that the identity information corresponding to different fruit branches is different.

[0083] For example, step S150 above, which determines the identity information of each target fruit branch based on the degree of overlap between the first branch attention area corresponding to each target fruit branch and the first prediction area corresponding to each historical fruit branch, includes step S151b.

[0084] In step S151b, for each target fruit branch corresponding to a first branch attention region, if there is at least one first prediction region corresponding to the first branch attention region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to the at least one first prediction region.

[0085] The first predicted region corresponding to the first branch's area of ​​interest is the first predicted region whose overlap with the first branch's area of ​​interest is greater than a preset threshold. For example, if the overlap between the first branch's area of ​​interest and the first predicted region is 98%, and the preset threshold is 95%, then the first predicted region is the first predicted region corresponding to the first branch's area of ​​interest.

[0086] This invention provides an example for reference. Referring again to the example in step S151a above, using target fruit branch G2 as an example, for target fruit branch G2, according to the content in step S140 above, the overlap between the first branch attention area corresponding to target fruit branch G2 and the first prediction area corresponding to historical fruit branch L1 is determined to be 30%, the overlap between target fruit branch G2 and the first prediction area corresponding to historical fruit branch L2 is 20%, and the overlap between target fruit branch G2 and the first prediction area corresponding to historical fruit branch L3 is 98%. When the preset degree threshold is 95%, it can be determined that the overlap between the first branch attention area corresponding to target fruit branch G2 and the first prediction area corresponding to historical fruit branch L3 is greater than the preset degree threshold. Therefore, the first prediction area corresponding to historical fruit branch L3 is taken as the first prediction area corresponding to the first branch attention area corresponding to target fruit branch G2. The identity information (i.e., L3) of the historical fruit branch corresponding to the first predicted region whose overlap with the first branch's region of interest exceeds a preset threshold can be used as the identity information of the target fruit branch (i.e., target fruit branch G2) corresponding to the first branch's region of interest. In other words, the identity information of the target fruit branch G2 is determined to be L3. This is equivalent to considering the target fruit branch G2 and the historical fruit branch with identity information L3 as the same fruit branch.

[0087] According to the above-described scheme of the present invention, for each target fruit branch corresponding to a first branch interest region, if the overlap between the first branch interest region and the first prediction regions corresponding to all historical fruit branches is less than a preset threshold, it can be determined that the identity information of the target fruit branch is different from the identity information of all historical fruit branches, and the target fruit branch is regarded as a new historical fruit branch. For each target fruit branch corresponding to a first branch interest region, if there is at least one first prediction region corresponding to the first branch interest region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to the at least one first prediction region. The above scheme can reduce the occurrence of duplicate counting or identity confusion caused by changes in the shooting angle of the image acquisition device or temporary occlusion of fruit branches, which helps to ensure that each fruit branch is counted only once. In agricultural scenarios with dense fruiting and frequent foliage occlusion, the above method can improve the cross-frame deduplication accuracy of fruit branches, thereby improving the reliability of fruit branch count and providing a solid data foundation for high-precision yield prediction.

[0088] For example, in step S151b above, when there is at least one first prediction region corresponding to the first branch attention region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to each of the at least one first prediction region, including step S151b1a.

[0089] In step S151b1a, if there are multiple first prediction regions corresponding to the first branch interest region, and the identity information of the historical fruit branches corresponding to the multiple first prediction regions is not determined based on the target fruit image to which the first branch interest region belongs, the identity information of the target fruit branch is determined to be empty or composite information.

[0090] The composite information includes the identity information of the historical fruit branches corresponding to each of the multiple first prediction regions. For example, if the overlap between the first branch attention region corresponding to the target fruit branch and the three first prediction regions is greater than a preset threshold, then the identity information of the target fruit branch can include the identity information of the historical fruit branches corresponding to the three first prediction regions, i.e., composite information.

[0091] This invention provides an example for reference. The example in step S151b above can be continued, with the addition of historical fruit branches with identity information L5 (hereinafter referred to as historical fruit branch L5), L6 (hereinafter referred to as historical fruit branch L6), and L7 (hereinafter referred to as historical fruit branch L7). Taking target fruit branch G3 as an example, for target fruit branch G3, according to the content in step S140 above, the overlap between the first branch attention area corresponding to target fruit branch G3 and the first prediction area corresponding to each of the historical fruit branches L1, L2, and L3 is less than a preset threshold. Furthermore, the overlap between the first branch attention area corresponding to target fruit branch G3 and the first prediction area corresponding to each of the historical fruit branches L5, L6, and L7 is greater than the preset threshold. In the above situation, it can be determined whether the overlap between the first branch interest region corresponding to other target fruit branches in the target fruit image corresponding to target fruit branch G3 and the first predicted region corresponding to historical fruit branch L5, the first predicted region corresponding to historical fruit branch L6, and the first predicted region corresponding to historical fruit branch L7 is greater than a preset threshold. If not, it falls under the case described above where "the identity information of the historical fruit branches corresponding to multiple first predicted regions is not determined based on the target fruit image to which the first branch interest region belongs." In this case, it can be determined that the identity information of target fruit branch G3 is empty, or that the identity information is a composite of L5, L6, and L7. If it exists, a specified first predicted region (which can be a first predicted region whose overlap with the first branch interest region corresponding to other target fruit branches is greater than a preset threshold) can be filtered out from the first predicted regions corresponding to target fruit branch G3. Then, based on the remaining first predicted regions corresponding to target fruit branch G3, the identity information of target fruit branch G3 is determined. For example, if the first prediction region specified above is the first prediction region corresponding to historical fruit branch L5, then the remaining first prediction regions corresponding to target fruit branch G3 include: the first prediction regions corresponding to historical fruit branch L6 and the first prediction regions corresponding to historical fruit branch L7. Based on the remaining first prediction regions corresponding to target fruit branch G3, it is determined whether the identity information of target fruit branch G3 is empty or composite information (i.e., the identity information is a composite of L6 and L7). For yet another example, if the first prediction regions specified above are the first prediction regions corresponding to historical fruit branch L5 and historical fruit branch L6, then the remaining first prediction region corresponding to target fruit branch G3 is the first prediction region corresponding to historical fruit branch L7.In this case, the first branch attention area corresponding to the target fruit branch corresponds to only one first prediction area. The identity information of the historical fruit branch corresponding to this first prediction area can be used as the identity information of the target fruit branch, that is, the identity information of the target fruit branch G3 is L7. For example, if the specified first prediction area is the first prediction area corresponding to historical fruit branch L5, the first prediction area corresponding to historical fruit branch L6, and the first prediction area corresponding to historical fruit branch L7, then the target fruit branch G3 does not have any corresponding remaining first prediction areas. Referring to step S151a above, it can be determined that the identity information of the target fruit branch G3 is different from the identity information of all historical fruit branches (e.g., identity information L8), and the target fruit branch G3 is treated as a new historical fruit branch.

[0092] For example, step S151b above, when there is at least one first prediction region corresponding to the first branch attention region, determines the identity information of the target fruit branch based on the identity information of the historical fruit branches corresponding to each of the at least one first prediction region, including step S151b1b.

[0093] In step S151b1b, if the first predicted region corresponding to the first branch attention region is the same as at least one of the first predicted regions corresponding to other first branch attention regions in the target fruit image, it is determined that the identity information of the target fruit branch corresponding to the first branch attention region is different from the identity information of the target fruit branch corresponding to other first branch attention regions, and is also different from the identity information of all historical fruit branches, and the identity information of the historical fruit branch corresponding to the target predicted region is deleted.

[0094] The target prediction region is the first prediction region corresponding to the first branch's region of interest. For example, if two first branch regions of interest correspond to the same first prediction region, then that first prediction region can be the target prediction region.

[0095] This invention provides an example for reference. The example in step S151b above can be continued, with the addition of a historical fruit branch (hereinafter referred to as historical fruit branch L9) with identity information L9, and target fruit branches G4 and G5. Here, the overlap between the first branch attention area corresponding to target fruit branch G4 and the first prediction area corresponding to historical fruit branch L9 is greater than a preset threshold. Furthermore, when the overlap between the first branch attention area corresponding to target fruit branch G5 and the first prediction area corresponding to historical fruit branch L9 is greater than the preset threshold, the first prediction area corresponding to historical fruit branch L9 can be the target prediction area corresponding to the first branch attention area corresponding to target fruit branch G4. This situation is equivalent to two different fruit branches being captured as one fruit branch when the image acquisition device acquires the fruit image corresponding to historical fruit branch L9, due to the shooting angle. In this case, target fruit branch G4 can be assigned identity information that is different from all other historical fruit branches (e.g., L...). 10 At the same time, it can also assign the target fruit branch G5 an identity that is different from all other historical fruit branches (e.g., L). 11 The identity information L9 (i.e., the identity information of the historical fruit branch corresponding to the target prediction region) can be deleted.

[0096] For example, in step S151b above, when there is at least one first prediction region corresponding to the first branch attention region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to each of the at least one first prediction region, including step S151b1c.

[0097] In step S151b1c, if the first branch attention area corresponds to a first prediction area and is different from the first prediction areas corresponding to other first branch attention areas, the identity information of the target fruit branch corresponding to the first branch attention area is determined to be the identity information of the historical fruit branch corresponding to the first prediction area.

[0098] For details, please refer to step S151b1a above, where the first prediction region specified above refers to the first prediction region corresponding to historical fruit branch L5 and the first prediction region corresponding to historical fruit branch L6.

[0099] The above-described solution according to embodiments of the present invention is beneficial in solving the problem of determining the identity information of a target fruit branch in various situations, such as one-to-many, many-to-one, and one-to-one relationships, that may occur between the first branch interest region and the first prediction region in a target fruit image. This solution improves the robustness of identity management in high-density, high-dynamic planting areas, effectively suppressing identity mismatches and duplicate counts caused by occlusion recovery, detection splits, or transient missed detections. This enhances the stability of fruit branch number statistics and provides a reliable identity tracking basis for long-term, continuous, and high-precision yield prediction.

[0100] For example, the method for processing fruit images further includes steps S210 and S220.

[0101] In step S210, based on the target fruit image, the first fruit interest region corresponding to each target fruit in the target fruit image is determined by the trained instance segmentation model.

[0102] The instance segmentation model can be an instance segmentation model trained based on a masked region convolutional neural network.

[0103] The trained instance segmentation model can determine the first fruit interest region corresponding to each target fruit in each target fruit branch within the target fruit image, based on the aforementioned target fruit image containing multiple target fruit branches. For example, a target fruit image containing multiple target fruit branches can be input into the trained instance segmentation model, which can then determine the first fruit interest region corresponding to each target fruit based on the input target fruit image.

[0104] This invention provides a training method for an instance segmentation model for reference. A second training dataset is obtained. The instance segmentation model is iteratively trained using the second training dataset, and the model parameters are adjusted using a loss function until training is complete. The second training dataset includes multiple training images of training fruits. The label of each second training image can be the corresponding training fruit region within that image. It is understood that the corresponding training fruit regions within the second training image can be represented by the mask region of a fruit mask image or a set of pixel coordinates. The conditions for training completion can include the loss value calculated by the loss function stabilizing and the number of iterations reaching a preset number.

[0105] The training fruit must belong to the same fruit category as the target fruit. For example, if the target fruit belongs to the grape category, then the training fruit must also belong to the grape category. If the target fruit belongs to the banana category, then the training fruit must also belong to the banana category. If the target fruit belongs to the tomato category, then the training fruit must also belong to the tomato category.

[0106] The training fruit region corresponding to a training fruit is used to represent the position of that training fruit in the second training image, and the areas within the edges of each training fruit region do not overlap. For example, the region corresponding to a training fruit in the second training image may include the part of the training fruit that is not occluded by other training fruits, but does not include the part of the training fruit that is occluded by other training fruits.

[0107] In step S220, for each target fruit branch, the total number of target fruits included in the target fruit branch is determined based on the total number of the first fruit attention regions corresponding to all target fruits in the target fruit branch.

[0108] This invention provides an example for reference. Given that the total number of first fruit interest regions corresponding to each target fruit in target fruit branch G1 is 10, it can be determined that the total number of target fruits included in target fruit branch G1 is 10.

[0109] See Figure 3 As shown, Figure 3 A schematic diagram of a user interface according to an embodiment of the present invention is shown. The user interface can display the first branch interest region (i.e., the first branch of interest region for each target fruit branch in the target fruit image) on the user interface. Figure 3 Each black box in the image represents a region of interest for the first branch, and the total number of target fruit branches in all fruit images acquired by the image acquisition device (i.e., Figure 3 The 128 in the image represent the total number of target fruit branches in all fruit images acquired by the image acquisition device until the identification information of all target fruit branches in the target fruit image is determined. Figure 3 The 128 in the image represent the total number of target fruits in all fruit images acquired by the image acquisition device up to the point where the total number of target fruits is 1500. It should be understood that the user interface can also display parameters such as the identity information corresponding to each target fruit branch in the target fruit image, and the number of target fruits in each target fruit branch.

[0110] According to the above-described scheme of the present invention, based on the target fruit image, a trained instance segmentation model can be used to determine the first fruit interest region corresponding to each target fruit in each target fruit branch within the target fruit image. Then, for each target fruit branch, the total number of target fruits included in that branch can be determined based on the total number of the first fruit interest regions corresponding to all target fruits within that branch. This scheme, by determining the number of target fruits within each target fruit branch using a trained instance segmentation model, helps improve the accuracy of target fruit yield prediction.

[0111] For example, step S210 above includes steps S211 to S213.

[0112] In step S211, based on the target fruit image including multiple original fruit branches, the second fruit interest region corresponding to each original fruit in each original fruit branch in the target fruit image is determined by the trained instance segmentation model.

[0113] The original fruit and the target fruit must belong to the same fruit category. For example, if the target fruit belongs to the grape category, then the original fruit must also belong to the grape category.

[0114] The process for determining the second fruit region of interest described above can be found in step S210 above, which describes the process for determining the first fruit region of interest. This invention will not be elaborated upon here.

[0115] In step S212, for each original fruit in each original fruit branch, the fruit depth information corresponding to the original fruit is determined based on the target depth image acquired by the image acquisition device for multiple original fruit branches, the second fruit interest region corresponding to the original fruit, and the pixel correspondence.

[0116] The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image. The fruit depth information corresponding to the original fruit is used to represent the distance between the original fruit and the image acquisition device. For details, please refer to the content in step S112 above, which will not be repeated here.

[0117] The process of determining the fruit depth information corresponding to the original fruit can refer to the process of determining the fruit branch depth information corresponding to the original fruit branch in step S112 above, and will not be repeated here.

[0118] In step S213, original fruits whose corresponding fruit depth information is greater than a preset depth threshold are filtered out, so that the filtered original fruits are used as target fruits, and the second fruit interest region corresponding to the original fruit is used as the first fruit interest region corresponding to the target fruit.

[0119] The above process can be referred to in step S113 for determining the first branch interest area, and will not be repeated here. The preset depth threshold used to screen the original fruit can be the same as the preset depth threshold for screening the original fruit branches in step S113 above, and can also be determined according to the actual situation.

[0120] According to the above-described scheme of the present invention, multiple original fruits corresponding to the target fruit image can be screened based on depth information to identify the target fruit. This achieves distance-aware screening of fruits within the planting area, effectively reducing the impact of occlusion, background interference, or exceeding the reasonable observation range (such as being too far away, ground clutter, or crosstalk from adjacent plants) on the detection results, thus improving the accuracy of fruit identification. Furthermore, counting and tracking only target fruits within the effective operating depth range avoids duplicate counting and overestimation of yield, thereby improving the reliability of the target fruit yield prediction.

[0121] For example, step S220 may include step S221a.

[0122] In step S221a, for each historical fruit branch that has the same identity information as the target fruit branch, the maximum of the following is taken as the number of target fruits corresponding to the target fruit branch: the number of first fruit attention areas corresponding to the target fruit branch, and the number of first fruit attention areas corresponding to historical fruit branches that have the same identity information as the target fruit branch.

[0123] This invention provides an example for reference. If the identity information of the target fruit branch is determined to be L1, then the maximum value between the number of target fruits in historical fruit branches with the same identity information L1 and the number of target fruits in the target fruit branch can be used as the number of target fruits corresponding to that target fruit branch. For example, if the identity information corresponding to the target fruit branch is determined to be L1, the number of first fruit attention areas corresponding to that target fruit branch is 10, and the number of first fruit attention areas corresponding to historical fruit branches with the same identity information as that target fruit branch is 12, then 12 can be used as the number of target fruits corresponding to that target fruit branch.

[0124] For example, step S220 may include step S221b.

[0125] In step S221b, for each historical fruit branch that does not have the same identity information as the target fruit branch, the number of the first fruit attention regions corresponding to the target fruit branch is taken as the number of target fruits corresponding to the target fruit branch.

[0126] The present invention provides an example for reference. The number of first fruit attention regions corresponding to the target fruit branch is 10, and there is no historical fruit branch with the same identity information as the target fruit branch (for example, the case of target fruit branch G1 in step S151a above), then the number of target fruits corresponding to the target fruit branch is 10.

[0127] According to the above-described solution of the present invention, the occurrence of inaccurate target fruit counts in a single frame of fruit image due to changes in lighting, foliage occlusion, fruit posture rotation, or temporary missed detections can be effectively reduced, which is beneficial to improving the accuracy of target fruit yield prediction. Furthermore, when a new fruit branch is detected, the number of target fruits in the new fruit branch can be updated in a timely manner, ensuring that the number of target fruits in the newly added fruit branch is not missed in the count.

[0128] This invention also provides a fruit image processing device. Figure 4 A schematic block diagram of a fruit image processing apparatus 300 according to an embodiment of the present invention is shown. (In conjunction with...) Figure 4 As shown, the processing device 300 may include a first branch interest area determination module 310, a world movement distance determination module 320, a first prediction area determination module 330, an overlap degree determination module 340, and an identity information determination module 350.

[0129] The first branch attention region determination module 310 is used to determine the first branch attention region corresponding to each of the multiple target fruit branches in the target fruit image based on the target fruit image including multiple target fruit branches, through a trained target detection model.

[0130] The world movement distance determination module 320 is used to determine the world movement distance of the image acquisition device corresponding to each of the at least one historical fruit branches, based on the first position information when the image acquisition device acquires the target fruit image and the second position information when the image acquisition device acquires the historical fruit image corresponding to the historical fruit branch. The historical fruit branch is the target fruit branch in the historical fruit image that has identity information, and the identity information of different historical fruit branches is different.

[0131] The first prediction region determination module 330 is used to determine, for each historical fruit branch in at least one historical fruit branch, the first prediction region corresponding to the historical fruit branch in the target fruit image based on the world movement distance of the image acquisition device corresponding to the historical fruit branch, the first branch interest region corresponding to the historical fruit branch in its corresponding historical fruit image, and the movement correspondence, wherein the movement correspondence is the correspondence between the world movement distance of the image acquisition device in the world coordinate system and the pixel movement distance of the pixel in the pixel coordinate system.

[0132] The overlap determination module 340 is used to determine the degree of overlap between the first branch attention region and each first prediction region for each target fruit branch in the target fruit image, based on the first branch attention region and the first prediction region corresponding to each historical fruit branch.

[0133] The identity information determination module 350 is used to determine the identity information of each target fruit branch based on the degree of overlap between the first branch attention area corresponding to each target fruit branch and the first prediction area corresponding to each historical fruit branch.

[0134] For example, the first branch attention area determination module 310 includes: a second branch attention area determination module, a fruit branch depth information determination module, and a first determination module.

[0135] The second branch attention region determination module can be used to determine the second branch attention region corresponding to each of the multiple original fruit branches in the target fruit image based on a target fruit image containing multiple original fruit branches, through a trained target detection model.

[0136] The fruit branch depth information determination module can be used to determine the fruit branch depth information corresponding to each of multiple original fruit branches based on the target depth image acquired by the image acquisition device for the multiple original fruit branches, the second branch interest region corresponding to the original fruit branch, and the pixel correspondence. The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image, and the fruit branch depth information corresponding to the original fruit branch is used to represent the distance between the original fruit branch and the image acquisition device.

[0137] The first determining module can be used to filter out original fruit branches whose corresponding fruit branch depth information is greater than a preset depth threshold from multiple original fruit branches, so as to take the filtered original fruit branches as target fruit branches, and take the second branch attention area corresponding to the original fruit branch as the first branch attention area corresponding to the target fruit branch.

[0138] For example, the identity information determination module 350 includes at least one of the following: a second determination module and a third determination module.

[0139] The second determining module can be used to determine that the identity information of the target fruit branch is different from the identity information of all historical fruit branches when the overlap between the first branch attention area corresponding to each target fruit branch and the first prediction area corresponding to each of the historical fruit branches is less than a preset threshold, and then treat the target fruit branch as a new historical fruit branch.

[0140] The third determining module can be used to determine the identity information of the target fruit branch based on the identity information of the historical fruit branches corresponding to each of the first branch attention regions corresponding to each target fruit branch, when there is at least one first prediction region corresponding to the first branch attention region. The first prediction region corresponding to the first branch attention region is a first prediction region whose degree of overlap with the first branch attention region is greater than a preset degree threshold.

[0141] For example, the third determining module may include at least one of the following: a first submodule, a second submodule, and a third submodule.

[0142] The first submodule can be used to determine whether the identity information of the target fruit branch is empty or composite information when there are multiple first prediction regions corresponding to the first branch interest region, and the identity information of the historical fruit branches corresponding to the multiple first prediction regions is not determined based on the target fruit image to which the first branch interest region belongs. The composite information includes the identity information of the historical fruit branches corresponding to the multiple first prediction regions.

[0143] The second submodule can be used to determine, when the first predicted region corresponding to the first branch attention region is the same as at least one of the first predicted regions corresponding to other first branch attention regions in the target fruit image, that the identity information of the target fruit branch corresponding to the first branch attention region is different from the identity information of the target fruit branch corresponding to other first branch attention regions, and is different from the identity information of all historical fruit branches, and delete the identity information of the historical fruit branch corresponding to the target predicted region, wherein the target predicted region is the first predicted region corresponding to the first branch attention region.

[0144] The third submodule can be used to determine the identity information of the target fruit branch corresponding to the first branch attention area as the identity information of the historical fruit branch corresponding to the first prediction area when the first branch attention area corresponds to a first prediction area and is different from the first prediction areas corresponding to other first branch attention areas.

[0145] For example, the processing device 300 may further include: a first fruit interest area determination module 360 ​​and a total number determination module 370.

[0146] The first fruit interest region determination module 360 ​​can be used to determine the first fruit interest region corresponding to each target fruit in each target fruit branch in the target fruit image based on the target fruit image and through the trained instance segmentation model.

[0147] The total number determination module 370 can be used to determine the total number of target fruits included in each target fruit branch based on the total number of the first fruit attention regions corresponding to all target fruits in the target fruit branch.

[0148] For example, the first fruit interest area determination module 360 ​​may include: a second fruit interest area determination module, a fruit depth information determination module, and a fourth determination module.

[0149] The second fruit interest region determination module can be used to determine the second fruit interest region in the target fruit image for each original fruit in each of the multiple original fruit branches, based on a target fruit image that includes multiple original fruit branches, through a trained instance segmentation model.

[0150] The fruit depth information determination module can be used to determine the fruit depth information corresponding to each original fruit in each original fruit branch based on the target depth image acquired by the image acquisition device for multiple original fruit branches, the second fruit interest area corresponding to the original fruit, and the pixel correspondence. The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image, and the fruit depth information corresponding to the original fruit is used to represent the distance between the original fruit and the image acquisition device.

[0151] The fourth determination module can be used to filter out original fruits whose corresponding fruit depth information is greater than a preset depth threshold from multiple original fruits, so as to take the filtered original fruits as target fruits, and take the second fruit interest region corresponding to the original fruit as the first fruit interest region corresponding to the target fruit.

[0152] For example, the total number determination module may include at least one of the following: a first total number determination module and a second total number determination module.

[0153] The first total number determination module can be used to determine the number of target fruits corresponding to each target fruit branch for each historical fruit branch that has the same identity information as the target fruit branch. The maximum of the following is used as the number of target fruits corresponding to the target fruit branch: the number of first fruit attention areas corresponding to the target fruit branch, and the number of first fruit attention areas corresponding to historical fruit branches that have the same identity information as the target fruit branch.

[0154] The second total number determination module can be used to determine the number of the first fruit attention area corresponding to the target fruit branch as the number of the target fruit branch for each historical fruit branch that does not have the same identity information as the target fruit branch.

[0155] According to another aspect of the present invention, an electronic device is also provided. Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 5 As shown, the electronic device 400 includes a processor 410 and a memory 420. The memory 420 stores a computer program, and the computer program instructions are executed by the processor 410 to perform the above-mentioned fruit image processing method.

[0156] According to another aspect of the present invention, a storage medium storing a computer program / instructions is also provided. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media. The computer program / instructions are used by a processor during runtime to execute the above-described fruit image processing method.

[0157] Those skilled in the art can understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the relevant description of the fruit image processing method. For the sake of brevity, it will not be described in detail here.

[0158] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention thereto. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0160] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0161] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0162] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0163] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0164] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0165] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the fruit image processing apparatus according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0166] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0167] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for processing fruit images, characterized in that, The method includes: Based on a target fruit image including multiple target fruit branches, a trained target detection model is used to determine the first branch attention region corresponding to each of the multiple target fruit branches in the target fruit image. For each historical fruit branch in at least one historical fruit branch, Based on the first position information when the image acquisition device acquires the target fruit image and the second position information when the image acquisition device acquires the historical fruit image corresponding to the historical fruit branch, the world movement distance of the image acquisition device corresponding to the historical fruit branch is determined. The historical fruit branch is the target fruit branch in the historical fruit image that has identity information, and the identity information of different historical fruit branches is different. Based on the world movement distance of the image acquisition device corresponding to the historical fruit branch, the first branch attention area corresponding to the historical fruit branch in its corresponding historical fruit image, and the movement correspondence, the first prediction area corresponding to the historical fruit branch in the target fruit image is determined. The movement correspondence is the correspondence between the world movement distance of the image acquisition device in the world coordinate system and the pixel movement distance of the pixel in the pixel coordinate system. For each of the multiple target fruit branches, the degree of overlap between the first branch attention region and each of the first prediction regions is determined based on the first branch attention region and the first prediction region corresponding to each historical fruit branch in the target fruit image. The identity information of each target fruit branch is determined based on the degree of overlap between the first branch attention area corresponding to each target fruit branch and the first prediction area corresponding to each historical fruit branch.

2. The method as described in claim 1, characterized in that, The step of determining the first branch interest region corresponding to each of the multiple target fruit branches in the target fruit image based on a trained target detection model includes: Based on a target fruit image including multiple original fruit branches, a trained target detection model is used to determine the second branch attention region corresponding to each of the multiple original fruit branches in the target fruit image. For each of the plurality of original fruit branches, based on the target depth image acquired by the image acquisition device for the plurality of original fruit branches, the second branch interest region corresponding to the original fruit branch, and the pixel correspondence, the fruit branch depth information corresponding to the original fruit branch is determined. The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image. The fruit branch depth information corresponding to the original fruit branch is used to represent the distance between the original fruit branch and the image acquisition device. The original fruit branches whose corresponding fruit branch depth information is greater than a preset depth threshold are filtered out, so that the filtered original fruit branches are used as target fruit branches, and the second branch attention area corresponding to the original fruit branch is used as the first branch attention area corresponding to the target fruit branch.

3. The method as described in claim 1, characterized in that, The process of determining the identity information of each target fruit branch based on the degree of overlap between the first branch attention region corresponding to each target fruit branch and the first prediction region corresponding to each historical fruit branch includes at least one of the following: For each target fruit branch, if the overlap between the first branch attention area and the first prediction area corresponding to all historical fruit branches is less than a preset threshold, the identity information of the target fruit branch is determined to be different from the identity information of all historical fruit branches, and the target fruit branch is regarded as a new historical fruit branch. For each target fruit branch, if there is at least one first prediction region corresponding to the first branch attention region, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to the at least one first prediction region. The first prediction region corresponding to the first branch attention region is a first prediction region whose overlap with the first branch attention region is greater than a preset threshold.

4. The method as described in claim 3, characterized in that, When there is at least one first prediction region corresponding to the first branch's area of ​​interest, the identity information of the target fruit branch is determined based on the identity information of the historical fruit branches corresponding to each of the at least one first prediction region, including at least one of the following: If there are multiple first prediction regions corresponding to the first branch attention region, and the identity information of the historical fruit branches corresponding to each of the multiple first prediction regions is not determined based on the target fruit image to which the first branch attention region belongs, the identity information of the target fruit branch is determined to be empty or composite information, wherein the composite information includes the identity information of the historical fruit branches corresponding to each of the multiple first prediction regions. If the first predicted region corresponding to the first branch attention region is the same as at least one of the first predicted regions corresponding to other first branch attention regions in the target fruit image, it is determined that the identity information of the target fruit branch corresponding to the first branch attention region is different from the identity information of the target fruit branch corresponding to other first branch attention regions, and is also different from the identity information of all historical fruit branches. The identity information of the historical fruit branch corresponding to the target predicted region is deleted, wherein the target predicted region is the first predicted region corresponding to the first branch attention region. If the first branch attention area corresponds to a first prediction area, and the first prediction area is different from the first branch attention area, the identity information of the target fruit branch corresponding to the first branch attention area is determined to be the identity information of the historical fruit branch corresponding to the first prediction area.

5. The method as described in claim 1, characterized in that, The method further includes: Based on the target fruit image, the first fruit attention region corresponding to each target fruit in each target fruit branch in the target fruit image is determined by the trained instance segmentation model. For each target fruit branch, the total number of target fruits included in that target fruit branch is determined based on the total number of the first fruit attention regions corresponding to all target fruits in that target fruit branch.

6. The method as described in claim 5, characterized in that, Based on the target fruit image, the step of determining the first fruit interest region corresponding to each target fruit in each target fruit branch in the target fruit image through a trained instance segmentation model includes: Based on a target fruit image including multiple original fruit branches, a trained instance segmentation model is used to determine the second fruit interest region corresponding to each original fruit in each of the multiple original fruit branches in the target fruit image. For each original fruit in each original fruit branch, based on the target depth image acquired by the image acquisition device for the multiple original fruit branches, the second fruit interest region corresponding to the original fruit, and the pixel correspondence, the fruit depth information corresponding to the original fruit is determined. The pixel correspondence is the pixel correspondence between the target fruit image and the target depth image. The fruit depth information corresponding to the original fruit is used to represent the distance between the original fruit and the image acquisition device. The original fruits whose corresponding depth information is greater than a preset depth threshold are screened out, so that the screened original fruits are used as target fruits, and the second fruit interest region corresponding to the original fruit is used as the first fruit interest region corresponding to the target fruit.

7. The method as described in claim 5, characterized in that, The determination of the total number of target fruits included in the target fruit branch based on the total number of the first fruit attention regions corresponding to all target fruits in the target fruit branch includes at least one of the following: For each historical fruit branch that has the same identity information as the target fruit branch, the maximum of the following will be used as the number of target fruits corresponding to the target fruit branch: the number of first fruit attention areas corresponding to the target fruit branch, and the number of first fruit attention areas corresponding to historical fruit branches with the same identity information as the target fruit branch. For each historical fruit branch that does not have the same identity information as the target fruit branch, the number of the first fruit attention regions corresponding to the target fruit branch is taken as the number of target fruits corresponding to the target fruit branch.

8. A fruit image processing apparatus, characterized in that, The device includes: The first branch attention region determination module is used to determine the first branch attention region corresponding to each of the multiple target fruit branches in the target fruit image based on a target fruit image including multiple target fruit branches, through a trained target detection model. The world movement distance determination module is used to determine the world movement distance of the image acquisition device corresponding to each historical fruit branch for at least one historical fruit branch, based on the first position information when the image acquisition device acquires the target fruit image and the second position information when the image acquisition device acquires the historical fruit image corresponding to the historical fruit branch. The historical fruit branch is a target fruit branch in the historical fruit image that has corresponding identity information, and the identity information of different historical fruit branches is different. The first prediction region determination module is used to determine the first prediction region corresponding to the historical fruit branch in the target fruit image for each historical fruit branch in at least one historical fruit branch, based on the world movement distance of the image acquisition device corresponding to the historical fruit branch, the first branch attention region corresponding to the historical fruit branch in its corresponding historical fruit image, and the movement correspondence relationship, wherein the movement correspondence relationship is the correspondence between the world movement distance of the image acquisition device in the world coordinate system and the pixel movement distance of the pixel in the pixel coordinate system; The overlap degree determination module is used to determine the degree of overlap between the first branch attention region and each of the first prediction regions in the target fruit image for each of the multiple target fruit branches and the first branch attention region corresponding to each historical fruit branch. The identity information determination module is used to determine the identity information of each target fruit branch based on the degree of overlap between the first branch attention area corresponding to each target fruit branch and the first prediction area corresponding to each historical fruit branch.

9. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the fruit image processing method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program / instructions, characterized in that, The computer program / instructions, when running, are used to perform the fruit image processing method as described in any one of claims 1 to 7.

11. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the fruit image processing method as described in any one of claims 1 to 7.

Images