Leaf projection area calculation method and system for seedling automatic sorting scene

Abstract

The application discloses a kind of leaf projection area calculation method and system for seedling automation sorting scene, belong to agricultural visual detection and automation equipment technical field.The method includes the following steps: extracting the front view feature map and the overhead view feature map from the front view image and the overhead view image of seedling;And the front view feature map is aligned by overhead view feature map through polar line;According to the front view feature map and the overhead view feature map after alignment respectively, the front view two-dimensional skeleton and the overhead view two-dimensional skeleton are predicted, and the front view two-dimensional skeleton and the overhead view two-dimensional skeleton are fused based on the relationship of double camera calibration and are constructed as sparse three-dimensional skeleton priori;According to the elevation information and three-dimensional skeleton priori, the occluded area in the overhead view feature map is completed, and the completed overhead view feature map is obtained;According to the completed overhead view feature map, complete leaf mask is obtained;According to complete leaf mask, the physical projection area is calculated.The application can quickly and accurately calculate the projection area of seedling leaf, to complete seedling grading.

Description

Technical Field

[0001] This application belongs to the field of agricultural visual inspection and automation equipment technology, specifically relating to a method and system for calculating the projected area of ​​leaves for automated seedling sorting scenarios. Background Technology

[0002] In automated seedling sorting lines of modern facility agriculture, the projected area of ​​seedling leaves is one of the core indicators for judging growth grade and sorting decisions. These seedlings can include grafted vegetable seedlings, tissue culture seedlings of flowers, and tobacco seedlings, among others.

[0003] However, in real seedling cultivation scenarios, seedling leaves are generally crossed, overlapping, and curled, making it difficult to perceive the areas where leaves are obscured.

[0004] In traditional technologies, some solutions use depth sensors or binocular dense point cloud reconstruction to obtain 3D information in order to solve the occlusion problem, but these solutions have the following drawbacks: Depth mapping equipment is expensive and lacks stability in high-humidity, high-light greenhouse environments; dense point cloud reconstruction requires a huge amount of computation, with a single operation typically taking more than a second, which cannot match the high-speed cycle requirement of several plants per second in automated sorting production lines; non-modal completion methods that rely on RGB-D are limited by the quality of the depth map and lack spatial structural constraints for the completion results of slender, curled leaves. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for calculating the projected area of ​​leaves in automated seedling sorting scenarios, which can quickly and accurately calculate the projected area of ​​seedling leaves to complete seedling grading.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a method for calculating the projected area of ​​leaves in an automated seedling sorting scenario, the method comprising: A frontal feature map is extracted from the frontal view image of the seedling, and a top view feature map is extracted from the top view image of the seedling; wherein the frontal view image and the top view image are acquired by orthogonally arranged frontal and top view cameras, respectively; Based on the dual-camera calibration relationship, the front view feature map is subjected to a geometric transformation under epipolar constraints using the top view feature map as a reference, to obtain an aligned front view feature map that corresponds to the top view feature map pixel by pixel in the spatial dimension. Based on the aligned frontal feature map and the top view feature map, the frontal 2D skeleton and the top view 2D skeleton are predicted respectively, and the frontal 2D skeleton and the top view 2D skeleton are fused together based on the dual-camera calibration relationship to construct a sparse 3D skeleton prior. The elevation information of each leaf of the seedling is obtained from the front view feature map, and the occluded area in the top view feature map is completed based on the elevation information and the prior of the three-dimensional skeleton to obtain the completed top view feature map. The complete blade mask is obtained based on the completed top-view feature map; The physical projected area is calculated based on the complete blade mask.

[0007] Secondly, embodiments of this application provide a leaf projection area calculation system for automated seedling sorting scenarios. The system includes a top-view camera and a front-view camera arranged orthogonally in space, as well as a computer device. The computer device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0008] Thirdly, embodiments of this application provide a computer device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0011] Compared with the prior art, this application has the following beneficial effects: (1) By acquiring dual-view images through orthogonally arranged top-view and front-view cameras, the occlusion relationship is identified using the elevation information provided by the front-view images, and the occlusion area is accurately restored by combining the prior guidance of the three-dimensional skeleton to complete the occluded area.

[0012] (2) The sparse three-dimensional skeleton prior is used to replace the dense point cloud reconstruction. Only a few skeleton key points are needed to obtain the three-dimensional spatial prior required for guiding occlusion completion, which significantly reduces the amount of computation and meets the real-time requirements of the automated sorting production line while ensuring area accuracy.

[0013] (3) Only two ordinary cameras, a front-view camera and a top-view camera, are used. No depth sensor, lidar or structured light equipment is required. The hardware cost is low and it has better stability than active depth sensors in agricultural environments such as high humidity and strong light in greenhouses.

[0014] (4) The spatial orientation information of stem, petiole or leaf vein provided by the three-dimensional skeleton prior is used to guide the completion process, so that the completion result has physical rationality and avoids the morphological distortion problem caused by the pure data-driven method. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a method for calculating the projected area of ​​leaves in an automated seedling sorting scenario, provided by some embodiments of this application. Figure 2 These are internal structural diagrams of a computer device provided in some embodiments of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0018] In one exemplary embodiment, this application proposes a method for calculating the leaf projected area for automated seedling sorting. The following, in conjunction with the accompanying drawings, provides a detailed description of the method for calculating the leaf projected area for automated seedling sorting provided by this application through specific embodiments and application scenarios.

[0019] Reference Figure 1 The method includes steps 102-112. Wherein: Step 102: Extract the frontal feature map from the frontal view image of the seedling and extract the top view feature map from the top view image of the seedling; wherein the frontal view image and the top view image are acquired by orthogonally arranged frontal and top view cameras, respectively.

[0020] For images: In some embodiments, before deployment of the production system (i.e., an automated seedling sorting line), a checkerboard calibration board is used to perform joint calibration within the common field of view of the two cameras, obtaining the intrinsic parameter matrix, extrinsic parameter matrix (including rotation matrix and translation vector), and fundamental matrix characterizing the epipolar geometry of the two cameras for each camera. The two cameras are orthogonally arranged, i.e., their optical axes are at approximately 90° apart, and they share the same world coordinate system.

[0021] In some embodiments, the camera is rigidly fixed, and the calibration parameters can remain unchanged during production system operation.

[0022] In some embodiments, the camera may be an RGB camera.

[0023] In some embodiments, a top-view camera is mounted directly above the conveyor belt of the production system, with its optical axis perpendicular to the conveyor belt plane, to acquire top-view images of the seedlings, providing information on the distribution of the seedling leaves in the XY plane. A front-view camera is mounted to the side and in front of the conveyor belt, with its optical axis horizontally pointing towards the area through which the seedlings pass, to acquire front-view images of the seedlings, providing information on the layering relationship of the seedling leaves in the Z-axis (height) direction. The two cameras are spatially orthogonal with an optical axis angle of approximately 90°, sharing the same world coordinate system.

[0024] In some embodiments, the same hardware trigger signal (such as photoelectric sensor trigger) is sent to both cameras to ensure that the acquisition time of the two images is strictly synchronized, eliminating inter-frame deviation caused by seedling movement.

[0025] In some embodiments, before deployment of the production system, a checkerboard calibration board is used to perform joint calibration within the common field of view of the two cameras to obtain the calibration relationship between the top-view camera and the front-view camera. The calibration relationship between the two cameras includes an intrinsic parameter matrix, an extrinsic parameter matrix, and a fundamental matrix characterizing the epipolar geometry of the two cameras. The fundamental matrix is ​​a 3×3 matrix with a rank of 2.

[0026] It should be noted that, since the camera is rigidly fixed, the calibration parameters remain unchanged during production line operation. Simultaneously, the same hardware trigger signal controls the top-view camera and the front-view camera to synchronously acquire RGB images of the seedlings, ensuring that the acquisition time of the two images is strictly synchronized to eliminate inter-frame deviation caused by seedling movement.

[0027] For feature map extraction: The top-view image and the front-view image are processed by a feature encoder (e.g., a ResNet-50 backbone network) to obtain top-view feature maps and front-view feature maps.

[0028] Step 104: Based on the dual-camera calibration relationship, perform a geometric transformation on the front view feature map under epipolar constraints using the top view feature map as a reference to obtain an aligned front view feature map that corresponds to the top view feature map pixel by pixel in the spatial dimension.

[0029] In some embodiments, the calibrated intrinsic parameters and distortion coefficients are used to perform distortion correction and epipolar correction on the two original images, so that the corresponding epipolar lines are located on the same horizontal scan line in the top and front view images, simplifying the calculation of subsequent feature search along the epipolar lines. Among them, the calibration parameters, especially the fundamental matrix, are used for epipolar line intersection alignment and subsequent 3D skeleton mapping, serving as the mathematical basis for all geometric constraints.

[0030] In some embodiments, the dual-camera calibration relationship includes a fundamental matrix; the step of performing an epipolar-constrained geometric transformation on the frontal feature map based on the dual-camera calibration relationship and using the top-view feature map as a reference to obtain an aligned frontal feature map that corresponds pixel-by-pixel to the top-view feature map in the spatial dimension includes: For any spatial location in the top-view feature map, the epipolar equation corresponding to that spatial location in the front-view feature map is calculated based on the fundamental matrix.

[0031] Sampling along the epipolar equation on the frontal feature map to obtain The feature vectors corresponding to each sampling point; wherein, the It is an integer greater than 1.

[0032] right Each feature vector is weighted and aggregated using one-dimensional learnable attention weights, and the aggregated result is used as the aligned frontal feature corresponding to any spatial location.

[0033] The aligned frontal features corresponding to each spatial location are reconstructed into an aligned frontal feature map, wherein the aligned frontal feature map corresponds to the top-view feature map pixel by pixel in the spatial dimension.

[0034] Specifically: Given the fundamental matrix obtained through joint calibration, for any spatial location in the top-view feature map, the epipolar equation corresponding to that spatial location in the front-view feature map is calculated using the fundamental matrix. That is, the feature point in the front view that matches the top-view location must be located on the epipolar line.

[0035] On the frontal feature map, it is generated at equal intervals with sub-pixel precision along the epipolar direction. The location of each sampling point is determined. Feature vectors at each sampling point are obtained from the frontal feature map using bilinear interpolation, ensuring the entire sampling process is differentiable and supporting end-to-end backpropagation training. For the epipolar line... Each sampled feature is weighted and aggregated using one-dimensional learnable attention weights to form aligned frontal features. Finally, the aligned frontal features corresponding to each spatial location are reconstructed into an aligned frontal feature map, which corresponds pixel-by-pixel with the top-view feature map in the spatial dimension.

[0036] This embodiment reduces the search space for cross-view feature matching from two dimensions to one dimension, which not only significantly reduces the amount of computation, but more importantly, eliminates erroneous matching far from the epipolar line from a geometric perspective, fundamentally solving the problem of chaotic dual-view feature correspondence under complex occlusion.

[0037] Step 106: Based on the aligned front view feature map and the top view feature map, predict the front view 2D skeleton and the top view 2D skeleton respectively, and fuse the front view 2D skeleton and the top view 2D skeleton based on the dual camera calibration relationship to construct a sparse 3D skeleton prior.

[0038] In some embodiments, fusing the frontal 2D skeleton and the top-view 2D skeleton based on the dual-camera calibration relationship to construct a sparse 3D skeleton prior includes: Key points of the skeleton are extracted from the top-view two-dimensional skeleton and the front-view two-dimensional skeleton, and the topological connection relationship between each key point of the skeleton is obtained.

[0039] Based on the dual-camera calibration relationship, the common-view anchor point is mapped to three-dimensional space to obtain the three-dimensional coordinates of the common-view anchor point; wherein, the common-view anchor point is a matching skeleton key point in the top-view two-dimensional skeleton and the front-view two-dimensional skeleton.

[0040] The three-dimensional coordinates of the unilaterally visible point are determined based on the topological connection relationship and the three-dimensional coordinates of adjacent common-view anchor points; wherein, the unilaterally visible point is a skeleton key point that is visible only in the top-view two-dimensional skeleton or only in the front-view two-dimensional skeleton.

[0041] A sparse three-dimensional skeleton prior is generated based on the three-dimensional coordinates of all skeleton key points and the topological connection relationship; wherein, the three-dimensional skeleton prior represents the spatial orientation of the stem, petiole or leaf midrib of the seedling with a sparse linear structure.

[0042] In some embodiments, the matching skeletal key points are determined in the following manner: For the key skeleton points in the top-view 2D skeleton, their corresponding epipolar lines in the front view image are calculated according to the dual-camera calibration relationship. The key skeleton points in the front-view 2D skeleton located in the neighborhood of the epipolar lines are determined as matching candidates. The matching result is determined after topological consistency verification.

[0043] The three-dimensional coordinates of the unilaterally visible point are determined in the following way: Based on the three-dimensional coordinates of adjacent common anchor points, interpolation is performed along the topological connection relationship according to the arc length ratio, and extrapolation is performed in combination with the extension direction of the blade main vein to determine the value.

[0044] Specifically: First, a lightweight skeleton prediction head is attached to both the top-view feature map and the aligned front-view feature map, outputting a top-view 2D skeleton heatmap and a front-view 2D skeleton heatmap. The 2D skeleton refers to the center line of the main vein of the stem and the first-order leaf veins, which is a connected curve with a width of approximately 1-3 pixels. Key points of the skeleton, including bifurcation points, petiole connection points, and leaf tips, are detected from the 2D skeleton heatmap using non-maximum suppression, and the topological connectivity between these key points is obtained.

[0045] Then, the key points of the skeleton were divided into two categories for processing.

[0046] The first category consists of common-view anchor points, which are key points that can be identified in both the top-view and front-view 2D skeletons. For key points in the top-view 2D skeleton, their corresponding epipolar lines in the front-view image are calculated based on the dual-camera calibration relationship. Key points in the front-view 2D skeleton located within the neighborhood of the epipolar lines are identified as matching candidates, and the matching results are determined after topological consistency verification. For each pair of matched common-view anchor points, the 3D spatial coordinates of the point are solved using the linear triangulation method based on the calibration projection matrix.

[0047] The second category is single-sided visible points, which are key points visible only in the top-view 2D skeleton or only in the front-view 2D skeleton. For these points, dual-view matching is not forced. Instead, the 3D coordinates of adjacent shared-view skeleton key points are used as a reference. Interpolation is performed along the skeleton topological connection path according to the arc length ratio. The extension direction of the blade midrib is then combined with extrapolation to infer its 3D coordinates.

[0048] Finally, B-spline curves are fitted to all the 3D keypoints obtained from backprojection according to the skeleton topological connection relationship to form a continuous 3D skeleton prior curve in space. This 3D skeleton prior represents the spatial orientation of seedling stems, petioles or leaf midribs with a sparse linear structure. It does not need to cover all points on the leaf surface, but only needs to reflect the main orientation. The computational cost is significantly less than that of dense point cloud reconstruction.

[0049] This embodiment avoids dense point cloud reconstruction of the entire seedling by extracting and inverting only sparse skeleton key points, reducing the data volume by 3-4 orders of magnitude. This allows the acquisition of 3D priors to be completed within milliseconds on an embedded platform. The 3D skeleton is used for non-modal mask growth, providing physically reasonable extension directions and range boundaries for pixel completion in occluded areas.

[0050] Step 108: Obtain the elevation information of each leaf of the seedling based on the front view feature map, and complete the occluded area in the top view feature map based on the elevation information and the prior of the three-dimensional skeleton to obtain the completed top view feature map.

[0051] In some embodiments, the elevation information is a pixel-level relative elevation score map; the step of completing the occluded region in the top-view feature map based on the elevation information and the prior 3D skeleton to obtain the completed top-view feature map includes: The three-dimensional skeleton is projected a priori onto the top-view image space to generate skeleton guiding features.

[0052] Calculate the spatial gradient of the relative elevation score map, and determine the region with a gradient magnitude greater than a preset threshold as the leaf layer occlusion boundary.

[0053] A uniformly distributed grid of reference points is generated on the top-view feature map. Based on the blade stacking and occlusion boundary, the sampling offset of each reference point is generated according to the relative elevation score map and the skeleton guiding feature, so that each reference point is shifted from the high-score region to the low-score region to obtain deformable sampling points.

[0054] The occluded areas in the top-view feature map are completed based on the deformed sampling points to obtain the completed top-view feature map.

[0055] In some embodiments, the step of completing the occluded region in the top-view feature map based on the deformed sampling points to obtain the completed top-view feature map includes: By using bilinear interpolation, sampling features are obtained at the locations of the deformed sampling points in the top-view feature map. These sampling features are then used as keys and values ​​to perform multi-head attention calculations with the query corresponding to the top-view feature map. The attention aggregation result is then used as the completed top-view feature map.

[0056] Specifically: From the aligned frontal feature map, a pixel-level relative height score map is predicted using an elevation estimation head. The values ​​are normalized to between 0 and 1, with high-value regions corresponding to higher-positioned leaves (occluders) and low-value regions corresponding to lower-positioned leaves (occluded leaves). The spatial gradient is calculated on the height score map. Regions with gradient magnitudes exceeding a preset threshold are defined as the boundary transition zone of leaf stacking, which also represents the spatial location of occlusion; this is defined as the leaf stacking occlusion boundary. The preset threshold is an empirical value.

[0057] The spatial gradient of the height score map is concatenated with the epipolar lines, and then processed by a migration prediction network to output a sampling offset for deformable attention. This shifts the sampling points from the occluded leaf region to the occluded leaf region. The offset generation is constrained by the elevation gradient; the offset magnitude is larger at the occlusion boundary, and the direction points to the lower height side, i.e., the spatial direction of the occluded leaf.

[0058] By using bilinear interpolation, sampling features are obtained at the locations of each deformed sampling point in the top-view feature map. These sampling features are then used as keys and values ​​to perform multi-head attention calculations with the query corresponding to the top-view feature map. The attention aggregation result is then used as the completed top-view feature map.

[0059] The generation of offsets is also constrained by the 3D skeleton prior. The 3D skeleton prior is reprojected onto the top view image plane according to the projection matrix of the top view camera to generate a skeleton guide map, where each pixel value of the skeleton guide map indicates the distance and direction from that location to the nearest skeleton line.

[0060] The skeleton-guided features and the relative elevation score map are concatenated and then input into the offset prediction network to jointly constrain the generation of the sampling offset. This ensures that the reference point is offset along the blade extension direction indicated by the prior of the 3D skeleton, and that the offset sampling point does not exceed the reasonable blade extension range indicated by the skeleton.

[0061] Unlike standard deformable attention systems where offsets are learned entirely through data, the offset generation in this embodiment is explicitly guided by the physical elevation information captured by the front-view camera. This gives attention sampling a clear geometric meaning: at the occlusion boundary, the focus of feature aggregation is "squeezed" from the surface of the occluded object to the location of the occluded object, thus decoupling the occluded and occluded regions at the feature level. Simultaneously, the introduction of a 3D skeleton prior further constrains the offset direction, making the completion result physically plausible.

[0062] Step 110: Obtain the complete blade mask based on the completed top-view feature map.

[0063] In some embodiments, obtaining the complete blade mask based on the completed top-view feature map includes: Based on the completed top-view feature map, the visible mask of the top-view visible area is decoded and output.

[0064] Based on the joint input of the completed top-view feature map and the prior of the 3D skeleton, a non-modal mask is decoded and output, wherein the generation direction of the non-modal mask is constrained by the direction of the prior of the 3D skeleton.

[0065] The complete leaf mask is obtained by taking the union of the visible mask and the non-modal mask.

[0066] Specifically: This embodiment sets up two parallel decoding branches.

[0067] The first decoding branch is the visible mask branch. It takes the completed top-view feature map as input, passes through a decoder consisting of transposed convolutional layers and skip connections, and outputs a visible mask with the same resolution as the input image. This mask represents the leaf area that the camera can directly see from the top-view perspective.

[0068] The second decoding branch is a non-modal masking branch. It takes the completed top-view feature map and the 3D skeleton prior as joint input, and outputs a non-modal mask through a decoder with the same structure but independent parameters. The core mechanism of this decoder is skeleton-constrained mask growth, which is as follows: The decoder reprojects the 3D skeleton prior onto the top-view image plane according to the projection matrix of the top-view camera to generate a skeleton guide map, where each pixel value indicates the distance and direction from the nearest skeleton line to the current position; The decoder performs pixel-level expansion in the effective direction indicated by the skeleton guide map, gradually growing and completing the occluded blade region according to the skeleton topology, while being constrained by the boundary of the visible mask, so that the growth does not exceed the reasonable extension range of the blade shape.

[0069] Then, the visible mask and the non-modal mask are fused (union) to obtain the final complete leaf mask, which contains both the top-view visible area and the completed area, resulting in a complete set of seedling leaf pixels.

[0070] During training, the visible mask branch can be supervised using standard segmentation loss, with ground truth provided by manually labeled visible regions.

[0071] In some embodiments, the supervision signal for the non-modal mask branch comes from complete manual annotation (including complete annotation of the occluded portion of the occluded leaf). Its loss function includes segmentation loss (mask shape supervision), skeleton consistency loss (constraining the generation direction of the non-modal mask to be consistent with the skeleton guidance direction), and visible region consistency loss (constraining the visible portion of the non-modal mask to be consistent with the output of the visible mask branch). Through the joint constraints of the above multi-task loss functions, the two branches share fusion features and mutually constrain each other, ensuring that the non-modal mask remains consistent with the visible mask branch in the visible region and grows reasonably along the skeleton direction in the invisible region.

[0072] This embodiment unifies visible mask decoding and non-modal mask derivation within an end-to-end framework. The two branches share fusion features and are mutually constrained by skeleton consistency loss and visible region consistency loss, avoiding error accumulation in step-by-step processing. The 3D skeleton prior indicates the reasonable extension direction and range of the occluded blade region in the 2D image space, making the completion result physically reasonable and avoiding blade orientation errors, breaks, or adhesion problems caused by unconstrained image completion.

[0073] Step 112: Calculate the physical projected area based on the complete blade mask.

[0074] In some embodiments, calculating the physical projected area based on the complete blade mask includes: The pixel area conversion ratio is obtained based on the calibration parameters of the top-view camera. The number of pixels contained in the complete leaf mask and the physical area corresponding to each pixel are calculated to obtain the physical projected area of ​​the seedling's leaf.

[0075] Specifically: During the inference phase, after obtaining the complete leaf mask, the total number of pixels contained in the mask is counted. Combining this with the calibration parameters of the top-view camera, the number of pixels is multiplied by the physical area corresponding to each pixel to obtain the physical projected area of ​​the seedling leaf. Specifically, the calculation method is: Projected area = Pixel area conversion ratio × Number of physical pixels.

[0076] In some embodiments, when the ability to output leaf-by-leaf instances is available, the area of ​​each leaf can be calculated separately and then summed to obtain the total projected area of ​​the entire plant.

[0077] In some embodiments, after the area calculation is completed, the calculated projected area can be compared with the threshold according to a preset sorting grade threshold (for example, seedlings are divided into three grades: A, B, and C according to the leaf projection area) to automatically determine the growth grade of the seedling.

[0078] Finally, the area value and / or sorting grade signal are sent to the sorting actuator (such as a pneumatic push rod, robotic arm or diversion mechanism) through the output interface to trigger the corresponding execution action, push the seedling to the corresponding grade collection channel, and complete the automated sorting.

[0079] In this embodiment, area calculation relies on the calibration parameters of the top-view camera to establish a mapping relationship between pixels and actual physical dimensions. The final output is a physical area value and sorting grade signal that can be directly used for sorting decisions, facilitating direct interface with the execution mechanism of automated sorting production lines and achieving a high degree of integration. The entire process, from dual-view image acquisition to sorting grade output, forms a complete closed loop, requiring no manual intervention.

[0080] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0081] In an exemplary embodiment, this application also provides a leaf projection area calculation system for automated seedling sorting scenarios. The system includes a top-view camera and a front-view camera arranged orthogonally in space, as well as a computer device. The computer device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the above-described method embodiments.

[0082] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 2 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for calculating the projected area of ​​leaves in an automated seedling sorting scenario. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0083] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0084] In one embodiment, a computer-readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps in the above-described method embodiments.

[0085] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0086] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0088] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for calculating the projected area of ​​leaves in automated seedling sorting scenarios, characterized in that, The method for calculating the leaf projected area for automated seedling sorting includes: A frontal feature map is extracted from the frontal view image of the seedling, and a top view feature map is extracted from the top view image of the seedling; wherein the frontal view image and the top view image are acquired by orthogonally arranged frontal and top view cameras, respectively. Based on the dual-camera calibration relationship, the front view feature map is subjected to epipolar constraint geometric transformation using the top view feature map as a reference to obtain the aligned front view feature map that corresponds to the top view feature map pixel by pixel in the spatial dimension. Based on the aligned frontal feature map and the top view feature map, the frontal 2D skeleton and the top view 2D skeleton are predicted respectively, and the frontal 2D skeleton and the top view 2D skeleton are fused together based on the dual-camera calibration relationship to construct a sparse 3D skeleton prior. The elevation information of each leaf of the seedling is obtained from the front view feature map, and the occluded area in the top view feature map is completed based on the elevation information and the prior of the three-dimensional skeleton to obtain the completed top view feature map. The complete blade mask is obtained based on the completed top-view feature map; The physical projected area is calculated based on the complete blade mask.

2. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 1, characterized in that, The dual-camera calibration relationship includes a fundamental matrix; the step of performing an epipolar-constrained geometric transformation on the frontal feature map based on the dual-camera calibration relationship and using the top-view feature map as a reference to obtain an aligned frontal feature map that corresponds pixel-by-pixel to the top-view feature map in the spatial dimension includes: For any spatial location in the top-view feature map, the epipolar equation corresponding to that spatial location in the front-view feature map is calculated based on the fundamental matrix; Sampling along the epipolar equation on the frontal feature map yields... The feature vectors corresponding to each sampling point; wherein, the It is an integer greater than 1; right Each feature vector is weighted and aggregated using a one-dimensional learnable attention weight, and the aggregation result is used as the aligned frontal feature corresponding to any spatial location. The aligned frontal features corresponding to each spatial location are reconstructed into an aligned frontal feature map, wherein the aligned frontal feature map corresponds to the top-view feature map pixel by pixel in the spatial dimension.

3. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 1, characterized in that, The process of fusing the frontal 2D skeleton and the top 2D skeleton into a sparse 3D skeleton prior based on the dual-camera calibration relationship includes: Extract skeleton key points from the top-view two-dimensional skeleton and the front-view two-dimensional skeleton respectively, and obtain the topological connection relationship between each skeleton key point; Based on the dual-camera calibration relationship, the common-view anchor points are mapped to three-dimensional space to obtain the three-dimensional coordinates of the common-view anchor points; wherein, the common-view anchor points are the matching skeleton key points in the top-view two-dimensional skeleton and the front-view two-dimensional skeleton; The three-dimensional coordinates of the unilaterally visible point are determined based on the topological connection relationship and the three-dimensional coordinates of adjacent common-view anchor points; wherein, the unilaterally visible point is a skeleton key point that is visible only in the top-view two-dimensional skeleton or only in the front-view two-dimensional skeleton; A sparse three-dimensional skeleton prior is generated based on the three-dimensional coordinates of all skeleton key points and the topological connection relationship; wherein, the three-dimensional skeleton prior represents the spatial orientation of the stem, petiole or leaf midrib of the seedling with a sparse linear structure.

4. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 3, characterized in that, The matching skeletal key points are determined in the following way: For the skeleton key points in the top-view two-dimensional skeleton, their corresponding epipolar lines in the front view image are calculated according to the dual-camera calibration relationship. Skeleton key points in the front-view two-dimensional skeleton located in the neighborhood of the epipolar lines are determined as matching candidates. The matching result is determined after topological consistency verification. The three-dimensional coordinates of the unilaterally visible point are determined in the following way: Based on the three-dimensional coordinates of adjacent common anchor points, interpolation is performed along the topological connection relationship according to the arc length ratio, and extrapolation is performed in combination with the extension direction of the blade main vein to determine the value.

5. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 1, characterized in that, The elevation information is a pixel-level relative elevation score map; the step of completing the occluded areas in the top-view feature map based on the elevation information and the prior 3D skeleton to obtain the completed top-view feature map includes: The three-dimensional skeleton is projected a priori onto the top-view image space to generate skeleton guiding features; Calculate the spatial gradient of the relative elevation score map, and determine the region with a gradient magnitude greater than a preset threshold as the leaf stacking occlusion boundary; A uniformly distributed grid of reference points is generated on the top-view feature map. Based on the blade stacking and occlusion boundary, the sampling offset of each reference point is generated according to the relative elevation score map and the skeleton guiding feature, so that each reference point is shifted from the high-score area to the low-score area to obtain deformed sampling points. The occluded areas in the top-view feature map are completed based on the deformed sampling points to obtain the completed top-view feature map.

6. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 5, characterized in that, The step of completing the occluded region in the top-view feature map based on the deformed sampling points to obtain the completed top-view feature map includes: By using bilinear interpolation, sampling features are obtained at the locations of the deformed sampling points in the top-view feature map. These sampling features are then used as keys and values ​​to perform multi-head attention calculations with the query corresponding to the top-view feature map. The attention aggregation result is then used as the completed top-view feature map.

7. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 1, characterized in that, The step of obtaining the complete blade mask based on the completed top-view feature map includes: Based on the completed top-view feature map, decode and output the visible mask of the top-view visible area; Based on the joint input of the completed top-view feature map and the prior of the three-dimensional skeleton, a non-modal mask is decoded and output, wherein the generation direction of the non-modal mask is constrained by the direction of the prior of the three-dimensional skeleton. The complete leaf mask is obtained by taking the union of the visible mask and the non-modal mask.

8. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 7, characterized in that, The visible mask is output by the first decoding branch, and the non-modal mask is output in parallel by the second decoding branch; When training the second decoding branch, the skeleton consistency loss is used to constrain the pixel extension direction of the non-modal mask to be consistent with the extension direction of the three-dimensional skeleton prior, and the visible region consistency loss is used to constrain the visible part of the non-modal mask to be consistent with the visible mask output by the first decoding branch.

9. The method for calculating the leaf projected area for automated seedling sorting scenarios according to claim 1, characterized in that, The calculation of the physical projected area based on the complete blade mask includes: The pixel area conversion ratio is obtained based on the calibration parameters of the top-view camera. The number of pixels contained in the complete leaf mask and the physical area corresponding to each pixel are calculated to obtain the physical projected area of ​​the seedling's leaf.

10. A leaf projection area calculation system for automated seedling sorting scenarios, characterized in that, The leaf projection area calculation system for automated seedling sorting includes a top-view camera and a front-view camera arranged orthogonally in space, and a computer device; the computer device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the program or instructions are executed by the processor, they implement the steps of the leaf projection area calculation method for automated seedling sorting as described in any one of claims 1-9.

Images