Intelligent estimation method and system for number of fruits in sheltered environment, terminal and medium

Abstract

The invention provides an intelligent estimation method and system for the number of fruits in a sheltered environment, a terminal and a medium, and the method comprises the steps: enabling a mobile robot to cruise in a fruit detection region, and synchronously obtaining an RGB image and a depth map; deploying a lightweight model at the edge end, and performing occlusion area detection according to the images and outputting a result; and the cloud deploys a diffusion model, iteratively generates a heat map based on an edge end result, and further obtains fruit number estimation. According to the method, high-precision shielding processing is realized, and after a heat map and watershed combined algorithm is adopted, the segmentation IoU of a double-fruit overlapping scene is greatly improved. Meanwhile, through cooperative computing of the edge end and the cloud end, single-frame processing delay is remarkably reduced, the real-time operation requirement of the mobile robot can be fully met, and a basis is provided for efficient and accurate orchard automatic picking and yield statistics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fruit quantity estimation, in particular to a method and system for intelligently estimating the number of fruits in a blocked environment, a terminal and a medium. BACKGROUND

[0002] In the existing orchard automatic picking and yield statistical system, traditional visual detection methods (such as YOLO, Mask R-CNN) face many severe challenges. Due to the dense distribution of fruits in the orchard and the frequent branch and leaf blocking, these methods have a very prominent problem of missed detection in the blocked scene. Specifically, local blocking makes feature extraction impossible, for example, when a fruit is exposed only 1 / 3 of its surface area, the model cannot obtain its complete features; the boundary of overlapping fruits is difficult to accurately segment, and multiple fruits in contact are often misjudged as a single fruit; at the same time, changes in lighting conditions can also seriously affect the texture recognition accuracy. Related experimental data shows that when the fruit blocking rate exceeds 40%, the detection accuracy of traditional methods drops sharply to below 65%, which undoubtedly seriously hinders the efficiency of orchard automation, making it difficult to achieve the desired effect of automatic picking and yield statistics.

[0003] Existing solutions also have their own limitations: detection models based on RGB images mainly rely on visible light images for bounding box regression, but in the case of branch and leaf blocking or fruit overlap, they cannot reconstruct the geometric features of the blocked part, resulting in a large increase in the missed detection rate. For example, in a dense citrus scene, the missed detection rate is as high as 32%. Multi-sensor fusion methods obtain depth information to assist positioning, but point cloud data produces a lot of noise in complex blocked environments, and its computationally intensive algorithms are difficult to implement in real time, with a processing delay of more than 500ms, which cannot meet the timeliness requirements of orchard automation. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a method and system for intelligently estimating the number of fruits in a blocked environment, a terminal and a medium, which solves the problem of high missed detection rate, low work efficiency and difficulty in adapting to complex orchard environments caused by the use of traditional visual detection, multi-sensor fusion and traditional image processing methods due to blocking, lighting changes, data processing timeliness and other problems.

[0005] To achieve the above object and other related objects, the present application provides a method for intelligently estimating the number of fruits in a blocked environment, which comprises: acquiring an RGB image and a depth map synchronously collected by a mobile robot during cruising in a fruit detection area; detecting a blocked area based on a light-weight model deployed on an edge, outputting a blocked area detection result according to the RGB image and the depth map; generating a heat map based on a diffusion model deployed on a cloud, outputting a fruit number estimation result according to the blocked area detection result from the edge.

[0006] In an embodiment of the present application, the blocked area detection based on the RGB image and the depth map and the output of the blocked area detection result comprises: extracting a potential blocked area from the depth map, and splicing the RGB image to obtain a first blocked area detection result; extracting a ROI area from the RGB image based on a deep learning algorithm to obtain a second blocked area detection result; splicing the first blocked area detection result and the second blocked area detection result to generate the blocked area detection result.

[0007] In an embodiment of the present application, the extraction of the potential blocked area from the depth map and the splicing of the RGB image to obtain the first blocked area detection result comprises: extracting a depth mutation area from the depth map by using a background difference method, marking the depth mutation area as a potential blocked area, and constructing a blocked mask matrix; splicing three channel data of the RGB image and the blocked mask matrix to form four-channel image data after splicing, which is taken as the first blocked area detection result.

[0008] In an embodiment of the present application, the generation of the heat map based on the blocked area detection result from the edge and the output of the fruit number estimation result comprises: diffusion iteration denoising based on the blocked area detection result from the edge, outputting a high-confidence fruit position heat map; adaptive threshold segmentation of the high-confidence fruit position heat map, generating a binary result; separation of overlapping targets based on the binary result by using a morphological watershed algorithm, and simultaneously combining depth information in the blocked area detection result as a spatial priori to optimize the segmentation boundary to generate a predicted blocked fruit center coordinate, and estimating the number of fruits through connected domain analysis.

[0009] In an embodiment of the present application, the training of the noise prediction network of the diffusion model by using a training set of multiple typical blocked modes comprises: reconstructing fruit positions by forward noise addition and reverse denoising based on each sample in the training set with labeled real fruit positions, and calculating the error of the real fruit positions and the reconstructed fruit positions by using an L1 loss function to optimize the network parameters of the noise prediction network.

[0010] In an embodiment of the present application, the method further comprises: storing the fruit quantity estimation result output by the diffusion model in a shielding mode memory bank, regularly fine-tuning the network parameters of the diffusion model, and automatically triggering model fine-tuning when a new shielding mode is detected.

[0011] In an embodiment of the present application, the edge end transmits the encrypted shielding area detection result to the cloud end through WebSocket.

[0012] To achieve the above object and other related objects, the present application provides a fruit quantity intelligent estimation system in a shielding environment, which comprises: a data acquisition module, configured to acquire RGB images and depth maps synchronously collected by a mobile robot during cruising in a fruit detection area; a shielding area detection module, connected to the data acquisition module, configured to perform shielding area detection based on a lightweight model deployed on an edge end according to the RGB images and the depth maps, and output a shielding area detection result; and a fruit quantity estimation module, connected to the shielding area detection module, configured to generate a heat map through noise iteration based on a diffusion model deployed on a cloud end according to the shielding area detection result from the edge end, and obtain a fruit quantity estimation result.

[0013] To achieve the above object and other related objects, the present application provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are configured to store a computer program; the one or more processors are connected to the memories and configured to run the computer program to execute the fruit quantity intelligent estimation method in a shielding environment.

[0014] To achieve the above object and other related objects, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is run by one or more processors to execute the method.

[0015] As described above, the present application is a fruit quantity intelligent estimation method, system, terminal and medium in a shielding environment, which has the following beneficial effects: the present application synchronously acquires RGB images and depth maps by cruising a mobile robot in a fruit detection area. A lightweight model is deployed on an edge end to perform shielding area detection according to these images and output results. A diffusion model is deployed on a cloud end to generate a heat map through noise iteration based on the results from the edge end, and then obtain fruit quantity estimation. The present application realizes high-precision shielding processing. After adopting a heat map and watershed joint algorithm, the segmentation IoU of double-fruit overlapping scenes is greatly improved. At the same time, through collaborative calculation of the edge end and the cloud end, single-frame processing delay is significantly reduced, which can fully meet the real-time operation requirements of the mobile robot, and provides a basis for realizing efficient and accurate orchard automatic picking and yield statistics. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1A flowchart of a method for intelligently estimating the number of fruits in a blocked environment is shown in an embodiment of the present application.

[0017] Figure 2 A flowchart of a method for intelligently estimating the number of fruits in a blocked environment is shown in an embodiment of the present application.

[0018] Figure 3 A structure diagram of a system for intelligently estimating the number of fruits in a blocked environment is shown in an embodiment of the present application.

[0019] Figure 4 A structure diagram of an electronic terminal is shown in an embodiment of the present application. DETAILED DESCRIPTION

[0020] Other advantages and novel features of the present application will become apparent from the following detailed description of the application when considered in conjunction with the drawings. The application can be put into practice / embodied in many different embodiments, and the following detailed description is not intended to limit the scope of the application, but merely to set forth the preferred embodiments of the application. The embodiments described herein are presented by way of example only and are not intended to limit the scope of the application. The various embodiments can be combined in different ways to provide other embodiments of the application. It should be noted that the following detailed description is intended to be read in connection with the accompanying drawings, which are described below.

[0021] It should be noted that in the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure the present application. It should be noted that the use of particular terminology when describing certain features or aspects of the application should not be taken to imply that the terminology is being redefined herein to be restricted to include any specific characteristics of the features or aspects of the application with which that terminology is associated. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0022] Throughout this specification, when it is said that a certain part is "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" in which other elements are interposed therebetween. In addition, when it is said that a certain part "includes" a certain constituent element, other constituent elements are not excluded unless specifically stated to the contrary, and it means that other constituent elements can also be included.

[0023] The terms first, second, third, etc. that are mentioned herein are used to describe various portions, components, regions, layers, and / or segments, but are not limited thereto. These terms are only used to distinguish one portion, component, region, layer, or segment from another portion, component, region, layer, or segment. Thus, a first portion, component, region, layer, or segment described below can be referred to as a second portion, component, region, layer, or segment without departing from the scope of the present application.

[0024] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including" when used herein, specify the presence of stated features, operations, elements, components, items, and / or groups but do not preclude the presence or addition of one or more other features, operations, elements, components, items, and / or groups thereof. As used herein, the terms "or" and "and / or" are to be interpreted as inclusive, i.e., as meaning one or any combination of items. Thus, "A, B or C" or "A, B and / or C" means any of the following: A; B; C; A and B; A and C; B and C; A, B and C. An exception to this definition will occur only when a combination of elements, functions, or operations are in some way inherently mutually exclusive.

[0025] The present application provides a method for intelligent estimation of fruit quantity in an occluded environment. A mobile robot is used to cruise in the fruit detection area to synchronously acquire RGB images and depth maps. A lightweight model is deployed on the edge to detect occluded areas based on these images and output the results. A diffusion model is deployed on the cloud to iteratively generate heat maps based on the edge results, and then obtain the fruit quantity estimation. The present application realizes high-precision occlusion processing. After using the heat map and watershed algorithm, the segmentation IoU of the double-fruit overlapping scene is greatly improved. At the same time, through the collaborative calculation of the edge and the cloud, the single-frame processing delay is significantly reduced, which can fully meet the real-time operation requirements of the mobile robot, and provides a basis for realizing efficient and accurate orchard automatic picking and yield statistics.

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The present application can be embodied in various different forms, and is not limited to the embodiments described herein.

[0027] As Figure 1 A flowchart of a method for intelligent estimation of fruit quantity in an occluded environment is shown.

[0028] The method comprises:

[0029] Step S1: Obtain the RGB images and depth maps synchronously collected by the mobile robot cruising in the fruit detection area.

[0030] In detail, the mobile robot cruises in the fruit detection area according to a preset path or a self-planned path. The path planning can be optimized according to the layout of the orchard, the distribution of the trees, and the requirements of the detection task. The robot is equipped with a visual sensor and a ToF depth sensor; the visual sensor can be a high-resolution RGB camera for collecting color images of fruits. The ToF depth sensor is used to measure three-dimensional point cloud data. The ToF sensor calculates depth information by emitting modulated light waves and measuring the time difference of the return of the light waves. The ToF sensor works synchronously with the RGB camera, ensuring that the RGB images and depth maps collected at the same time point and position are spatially aligned.

[0031] Step S2: Based on the lightweight model deployed at the edge, occlusion area detection is performed according to the RGB image and the depth map, and an occlusion area detection result is output.

[0032] In an embodiment, the occlusion area detection according to the RGB image and the depth map and the output of the occlusion area detection result include:

[0033] Potential occlusion areas are extracted from the depth map, and a first occlusion area detection result is obtained by splicing with the RGB image; wherein the first occlusion area detection result is a four-channel image;

[0034] Based on a deep learning algorithm, a ROI region is extracted from the RGB image to obtain a second occlusion area detection result; specifically, a pre-trained deep learning model (such as YOLO, Faster R-CNN, etc.) is used to extract a region of interest (ROI) from the RGB image. The input RGB image is input into the deep learning model, and the model outputs the detected object bounding box. These bounding boxes represent possible object regions, some of which may be occluded fruits. By analyzing these bounding boxes, possible occlusion areas are extracted. The extracted ROI region is generated into a binary image representing possible occlusion areas. This binary image is the second occlusion area detection result.

[0035] The first occlusion area detection result (four-channel image) and the second occlusion area detection result (binary image) are spliced to generate a comprehensive occlusion area detection result.

[0036] In an embodiment, the potential occlusion areas are extracted from the depth map, and the first occlusion area detection result is obtained by splicing with the RGB image, including:

[0037] The background difference method is used to extract the depth mutation region from the depth map, and the depth mutation region is marked as a potential occlusion region to construct an occlusion mask matrix. Specifically, the background model is established for the depth map, the current depth map is differentiated with the background depth model, and the depth change of each pixel point is calculated. A depth difference threshold is set. When the depth difference value exceeds the threshold, it is considered that the pixel point has a depth mutation. These depth mutation regions are likely to be occlusion regions. The detected depth mutation region is marked as a potential occlusion region, and a binary occlusion mask matrix is generated. The pixel value of the occlusion region is 1, and the pixel value of the non-occlusion region is 0.

[0038] The three channel data of the RGB image and the occlusion mask matrix are spliced to form a spliced four-channel image data as the first occlusion region detection result. Specifically, the three channel data of the RGB image is taken as the first three channels of a new image. The occlusion mask matrix is taken as the fourth channel of the new image. The spliced four-channel image is the first occlusion region detection result. This four-channel image not only contains the color information of the RGB image, but also contains the occlusion information of the depth map, providing more rich features for subsequent occlusion region detection.

[0039] In an embodiment, a lightweight model is trained using a training set of multiple typical occlusion patterns. The typical occlusion patterns include: lateral branch occlusion, foreground fruit occlusion, multiple cross occlusion, etc. Each sample in the training set is labeled with an occlusion region label. A lightweight model architecture suitable for edge deployment, such as YOLO-Occlusion, is trained using a training set of multiple typical occlusion patterns. A suitable loss function (such as cross-entropy loss, Dice loss, etc.) is used to optimize the model parameters to ensure that the model can accurately detect the occlusion region and obtain the final lightweight model.

[0040] Step S3: Based on the diffusion model deployed in the cloud, a heat map is generated by iterative noise generation based on the occlusion region detection result from the edge, and a fruit number estimation result is obtained.

[0041] In an embodiment, based on the lightweight model deployed in the edge, the occlusion region is detected based on the RGB image and the depth map, and the occlusion region detection result is output, which includes:

[0042] Based on the occlusion region detection result from the edge, a high-confidence fruit position heat map is output by diffusion iterative denoising; specifically, a pre-trained noise prediction network is used for diffusion iterative denoising, and after T times of diffusion iterative denoising, a high-confidence fruit position heat map is output.

[0043] An adaptive threshold segmentation is performed on the high-confidence fruit position heat map to generate a binary result; specifically, the mean μ H and the standard deviation σ HAdaptive threshold segmentation formula is adopted:

[0044] τ=μ H +α·σ H ; (1)

[0045] Wherein, alpha is an adjustment factor (default alpha = 1.5);The pixel value greater than threshold tau in the heat map is set to 1, and the rest is set to 0, to generate binary results.

[0046] Based on the binary results, the morphological watershed algorithm is used to separate overlapping targets, and the depth information in the occlusion area detection result is used as spatial prior to optimize the segmentation boundary to generate the predicted occluded fruit center coordinates. The geometric center of each segmentation region is calculated or other suitable algorithms are used to determine the center position of each fruit. Through connected component analysis, the predicted fruit center coordinates are counted, and finally the number of fruits is estimated. Connected component analysis can identify and count independent regions in the image, that is, the region corresponding to each fruit, so as to accurately count the total number of fruits.

[0047] In an embodiment, the noise prediction network in the diffusion model adopts U-Net++ as the main architecture, and embeds a cross-scale attention module. This architecture design aims to enhance the model's ability to capture multi-scale features, so as to more effectively handle complex occlusion and noise problems. U-Net++ is an improved U-Net architecture that further enhances feature transmission and fusion by adding skip connections and nested skip connections, improving the performance of the model. Specifically, it includes an encoder and a decoder; The encoder uses multi-level convolutional blocks to gradually reduce the spatial resolution, and extracts multi-scale features after each down-sampling. The embedded cross-scale attention module can automatically learn the relationship between features of different scales, enhancing the model's attention to key features and improving the accuracy of noise prediction. In addition, after each level of down-sampling in the encoder, a conditional injection layer is inserted to gate and fuse the depth map features with the diffusion step number embedding vector t. This design enables the model to better utilize depth information and time step information, thereby more accurately predicting noise.

[0048] The noise prediction network in the diffusion model is trained using a training set of various typical occlusion patterns, including lateral branch leaf occlusion, foreground fruit occlusion, multiple intersection occlusion, etc. The noise prediction network is trained using a forward noise addition and reverse noise removal process. The forward process gradually adds Gaussian noise to the true fruit position map in each training sample, and the reverse noise removal process gradually removes the noise through the noise prediction network, combining with the depth map condition to reconstruct the fruit position. The goal of the network is to recover a clear fruit position map from the noisy data. The L1 loss function is used to optimize the network parameters to minimize the difference between the predicted noise and the true noise. In the training data, simulated occlusions (such as random elliptical masks) are randomly generated to simulate the occlusion conditions in actual scenes. In this way, the model is forced to learn local completion capabilities, improving the robustness and accuracy of the model in handling occlusion problems.

[0049] In an embodiment, the method further comprises storing the fruit quantity estimation results output by the diffusion model in an occlusion pattern memory library. The occlusion pattern memory library can be designed as a database or file system to store the results of each detection, including RGB images, depth maps, occlusion area detection results, fruit quantity estimation results, etc. By setting a fixed time interval (such as every week or every month), the network parameters of the diffusion model are periodically fine-tuned, and the model fine-tuning is automatically triggered when a new occlusion pattern is detected. During the fine-tuning process, 10% of the old samples are retained, and the last layer of the network is updated to prevent the model from forgetting old knowledge when learning new knowledge, ensuring the stability and generalization ability of the model.

[0050] In an embodiment, the edge device is responsible for transmitting the occlusion area detection results to the cloud through the WebSocket protocol. To ensure the confidentiality and integrity of the data, the data during transmission will be encrypted. This process ensures the security of the data during transmission, preventing data leakage or tampering.

[0051] The present application deploys a lightweight model on the edge, which can realize real-time occlusion area detection of RGB images and depth maps, quickly output preliminary results, reduce data transmission volume and delay, and enhance privacy protection and reliability. The diffusion model is deployed on the cloud, which uses its powerful computing power for noise iteration and heat map generation to complete fruit quantity estimation, facilitating centralized management and optimization of the model. The two work together to fully leverage their respective advantages, achieving efficient, real-time, and secure fruit detection and quantity estimation suitable for a variety of complex application scenarios.

[0052] Similar to the principles of the above embodiments, the present application provides an intelligent fruit quantity estimation system in an occlusion environment.

[0053] The following specific embodiments are provided in conjunction with the accompanying drawings:

[0054] As Figure 3A structural schematic diagram of a fruit quantity intelligent estimation system in a shielding environment according to an embodiment of the present application is shown.

[0055] The system comprises:

[0056] A data acquisition module 1 is configured to acquire an RGB image and a depth map synchronously collected by a mobile robot during cruising in a fruit detection area.

[0057] A shielding area detection module 2 is connected to the data acquisition module 1 and configured to perform shielding area detection based on a lightweight model deployed at an edge end according to the RGB image and the depth map, and output a shielding area detection result.

[0058] A fruit quantity estimation module 3 is connected to the shielding area detection module 2 and configured to generate a heat map through noise iteration based on a diffusion model deployed at a cloud end according to the shielding area detection result from the edge end, and obtain a fruit quantity estimation result.

[0059] Since the implementation principle of the fruit quantity intelligent estimation system in the shielding environment has been described in the foregoing embodiments, no repeated description is given here.

[0060] The fruit quantity intelligent estimation method in the shielding environment provided by the embodiment of the present application can be implemented on a terminal side or a server side. As for the hardware structure of an electronic terminal, please refer to Figure 4 , which is an optional hardware structure schematic diagram of an electronic terminal 1000 provided by the embodiment of the present application. The terminal 1000 can be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 1000 comprises at least one processor 1001, a memory 1002, at least one network interface 10010 and a user interface 1009. Each component in the apparatus is coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to realize the connection communication between the components. The bus system 1005 comprises a data bus, a power supply bus, a control bus and a state signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system in Figure 4 .

[0061] The user interface 1009 can comprise a display, a keyboard, a mouse, a trackball, a click gun, a key, a button, a touchpad or a touch screen, etc.

[0062] It is to be understood that the memory 1002 can be volatile or nonvolatile memory, or both. In one example, the non-volatile memory can be read only memory (ROM), programmable read only memory (PROM), and the like, which are used as external cache. By way of example, and not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), and the like. The memory described in the embodiments of the present application is intended to include, but not be limited to, these and any other suitable type of memory.

[0063] The memory 1002 in the embodiments of the present application is configured to store various types of data to support the operation of the terminal 1000. Examples of these data include: any executable programs for operating on the terminal 1000, such as an operating system 10021 and an application program 10022; the operating system 10021 contains various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application program 10022 can contain various application programs, such as a media player (MediaPlayer), a browser (Browser), and the like, for implementing various application services. The method for intelligently estimating the number of fruits in a sheltered environment provided by the embodiments of the present application can be included in the application program 10022.

[0064] The method disclosed by the embodiments of the present application can be applied to the processor 1001 or implemented by the processor 1001. The processor 1001 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 1001. The processor 1001 can be a general processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 1001 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor 1001 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided by the embodiments of the present application, the hardware decoding processor can be directly embodied to complete the execution, or the hardware and software modules in the decoding processor can be combined to complete the execution. The software module can be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines the hardware to complete the steps of the foregoing method.

[0065] In the exemplary embodiments, the terminal 1000 can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), or the like for executing the foregoing method.

[0066] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by the hardware of the computer program. The foregoing computer program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various media that can store program codes.

[0067] In the embodiments provided in the present application, the computer readable and writable storage medium can include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, a flash memory, a U disk, a mobile hard disk, or any other medium capable of storing desired program code in the form of instructions or data structures and capable of being accessed by a computer. In addition, any connection can be appropriately referred to as a computer readable medium. For example, if instructions are sent from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave is included in the definition of the medium. However, it should be understood that the computer readable and writable storage medium and the data storage medium do not include connections, carriers, signals or other transitory media, but are intended for non-transitory, tangible storage media. As used in the application, magnetic disks and optical disks include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks and Blu-ray discs, in which magnetic disks typically magnetically copy data, and optical disks optically copy data with a laser.

[0068] In summary, the fruit quantity intelligent estimation method, system, terminal and medium under the shielding environment of the present application realize high-precision shielding processing. After adopting the algorithm of heat map and watershed, the segmentation IoU of the double-fruit overlapping scene is greatly improved. At the same time, through the collaborative calculation of the edge and the cloud, the single-frame processing delay is significantly reduced, which can fully meet the real-time operation demand of the mobile robot, and provides a basis for realizing efficient and accurate orchard automatic picking and yield statistics. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0069] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical thought disclosed in the present application should be covered by the claims of the present application.

Claims

1. A method for intelligently estimating the number of fruits under shading conditions, characterized in that, The method includes: Acquire RGB images and depth maps simultaneously collected by the mobile robot as it cruises through the fruit detection area; Based on a lightweight model deployed at the edge, occlusion region detection is performed according to RGB images and depth maps, and the occlusion region detection results are output. Based on the diffusion model deployed in the cloud, a heatmap is generated by iterative noise reduction based on the occlusion area detection results from the edge, and the fruit quantity estimation results are obtained.

2. The intelligent fruit quantity estimation method under shading conditions as described in claim 1, characterized in that, The step of detecting occlusion regions based on RGB images and depth maps, and outputting the occlusion region detection results, includes: Potential occlusion regions are extracted from the depth map and stitched with the RGB image to obtain the first occlusion region detection result; Based on deep learning algorithms, the ROI region is extracted from RGB images to obtain the detection results of the second occlusion region; The detection results of the first occluded area and the detection results of the second occluded area are combined to generate the occluded area detection result.

3. The intelligent fruit quantity estimation method under shading conditions according to claim 2, characterized in that, The step of extracting potential occlusion regions from the depth map and stitching them with the RGB image to obtain the first occlusion region detection result includes: The background subtraction method is used to extract depth abrupt change regions from the depth map, and these regions are marked as potential occlusion regions to construct an occlusion mask matrix. The three channels of the RGB image are concatenated with the occlusion mask matrix to form a concatenated four-channel image, which is used as the detection result of the first occlusion region.

4. The intelligent fruit quantity estimation method under shading conditions according to claim 1, characterized in that, The step of generating a heatmap through noise iteration based on the occlusion region detection results from the edge, and obtaining the fruit quantity estimation results, includes: Based on the detection results of occlusion areas from the edge, diffusion iteration denoising is performed to output a high-confidence fruit location heatmap; Adaptive threshold segmentation is applied to the high-confidence fruit location heatmap to generate binarized results; Based on the binarization results, the morphological watershed algorithm is used to separate overlapping targets. At the same time, the depth information in the occlusion region detection results is combined as a spatial prior to optimize the segmentation boundary, generate the predicted center coordinates of the occluded fruit, and estimate the number of fruits through connected component analysis.

5. The intelligent fruit quantity estimation method under shading conditions according to claim 4, characterized in that, The noise prediction network of the diffusion model is trained using training sets with various typical occlusion patterns. This includes: reconstructing the fruit position based on each sample with the real fruit position marked in the training set by forward noise addition and reverse noise removal, and using the L1 loss function to calculate the error of the real fruit position and the reconstructed fruit position to optimize the network parameters of the noise prediction network.

6. The intelligent fruit quantity estimation method under shading conditions according to claim 4, characterized in that, The method further includes: storing the fruit quantity estimation results output by the diffusion model in an occlusion pattern memory bank, periodically fine-tuning the network parameters of the diffusion model, and automatically triggering model fine-tuning when a new occlusion pattern is detected.

7. The intelligent fruit quantity estimation method under shading conditions as described in claim 1, characterized in that, The edge device transmits encrypted occlusion area detection results to the cloud via WebSocket.

8. A smart fruit quantity estimation system under shading conditions, characterized in that, The system includes: The data acquisition module is used to acquire RGB images and depth maps synchronously collected by the mobile robot as it cruises in the fruit detection area; The occlusion region detection module, connected to the data acquisition module, is used to perform occlusion region detection based on the RGB image and depth map using a lightweight model deployed at the edge, and output the occlusion region detection results. The fruit quantity estimation module, connected to the occlusion area detection module, is used to generate a heatmap by performing noise iteration based on the occlusion area detection results from the edge, according to the diffusion model deployed in the cloud, to obtain the fruit quantity estimation result.

9. An electronic terminal, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by one or more processors, performs the method as described in any one of claims 1 to 7.

Images