Plant phenotype processing method and plant phenotype processing equipment based on recovery scale

Abstract

The invention provides a plant phenotype processing method and plant phenotype processing equipment based on a recovery scale. The method comprises the following steps: placing a scale at a position adjacent to a plant, and carrying out image acquisition on an acquisition area containing the plant and the scale to obtain an image sequence; performing multi-view reconstruction on the image sequence to obtain a plant point cloud and a scale point cloud; a scale annular point cloud is extracted from the scale point cloud, space coordinate analysis is carried out according to the scale annular point cloud, and a rotation matrix and a scale factor are determined; performing attitude adjustment on the plant point cloud according to the rotation matrix to obtain an attitude-adjusted plant point cloud; performing scale recovery processing on the plant point cloud after attitude adjustment according to a scale factor to obtain a final plant point cloud; performing phenotype analysis according to the final plant point cloud to obtain plant phenotype parameters. The accurate rotation matrix and the scale factor are determined according to the scale, and corresponding attitude adjustment and scale recovery are performed on the obtained plant point cloud, so that the obtained final plant point cloud is more accurate, and the accuracy of phenotype analysis is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method and apparatus for processing plant phenotypes based on restoration scale. Background Technology

[0002] In existing technologies, the process of scale restoration of an image generally involves predicting the size and position of the corresponding pixels in the image, and then using the predicted size and position to determine the proportional relationship between the actual size of the objects in the image to restore the scale of each object in the image.

[0003] However, this scale recovery method has a large error and low accuracy, and cannot be applied to plant phenotyping fields where high precision in scale recovery is required. Summary of the Invention

[0004] In view of this, the purpose of this disclosure is to propose a plant phenotyping method and apparatus based on the recovery scale to solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, this disclosure provides a plant phenotypic treatment method based on the restoration scale, comprising:

[0006] A ruler is placed adjacent to the plant, and images are captured in the area containing the plant and the ruler to obtain an image sequence. The image sequence is reconstructed from multiple perspectives to obtain plant point clouds and scale point clouds; Extract scale ring point cloud from the scale point cloud, perform spatial coordinate analysis based on the scale ring point cloud, and determine the rotation matrix and scale factor; The plant point cloud is then subjected to attitude adjustment according to the rotation matrix to obtain an attitude-adjusted plant point cloud. The plant point cloud after attitude adjustment is scaled according to the scale factor to obtain the final plant point cloud. Phenotypic analysis was performed on the final plant point cloud to obtain plant phenotypic parameters.

[0007] Based on the same inventive concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0008] As described above, the plant phenotyping method and apparatus based on restoration scale provided in this disclosure pre-places a scale at the adjacent positions of the plant. This allows for image acquisition of the acquisition area containing both the plant and the scale, resulting in an image sequence containing both the plant and the scale. To restore the plant size using the scale, the image sequence containing both the plant and the scale is reconstructed from multiple perspectives, thereby reconstructing plant point clouds and scale point clouds. The scale ring-shaped portion of the scale point cloud is extracted to obtain a scale ring-shaped point cloud. Since the actual size corresponding to the scale is known, spatial coordinate analysis can be performed on the scale ring-shaped point cloud. This allows us to determine the spatial relationship between the scale ring point cloud and the actual size of the scale, thereby identifying the accurate rotation matrix and scale factor. The resulting plant point cloud can then be attitude-adjusted according to the rotation matrix, resulting in a more accurate attitude position. Furthermore, the attitude-adjusted point cloud undergoes scale recovery processing according to the scale factor, making the final plant point cloud more closely resemble the actual size of the plant, thus improving its accuracy. Consequently, phenotypic analysis based on this final plant point cloud becomes more accurate and beneficial for plant phenotypic analysis, yielding more accurate plant phenotypic parameters. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic flowchart of a plant phenotyping method based on the recovery scale according to an embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating the obtaining of a scale point cloud and a plant point cloud of a single plant according to an embodiment of this disclosure. Figure 3 This is a schematic diagram illustrating the determination of the rotation matrix according to an embodiment of the present disclosure; Figure 4 A schematic diagram of rotating the scale ring point cloud according to an embodiment of this disclosure; Figure 5 A schematic diagram illustrating the modification of the circular model according to an embodiment of this disclosure; Figure 6 This is a schematic diagram of the rotation and scale correction of plant point clouds according to an embodiment of the present disclosure; Figure 7 This is a schematic diagram illustrating the determination of plant point clouds for multiple plants according to an embodiment of this disclosure. Figure 8This is a schematic diagram of the structure of a plant phenotyping device based on the recovery scale according to an embodiment of the present disclosure; Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] Definitions: SfM: Structure from Motion, a motion recovery structure algorithm.

[0014] SAM 2: Segment Anything 2, a semantic segmentation model.

[0015] MVS: Multi-View Stereo, a multi-view stereo vision algorithm.

[0016] MagicRing: A circular ruler.

[0017] HSV: Hue, Saturation, Value, a color space.

[0018] SOR: Statistical Outlier Removal.

[0019] PCA: Principal components analysis.

[0020] RANSAC: Random Sample Consensus.

[0021] K-Net: K-net work, is a unified framework for image segmentation.

[0022] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0023] The plant phenotypic treatment method based on the restoration scale proposed in this disclosure, such as Figure 1 As shown, it includes: Step 101: Place a ruler at the adjacent position of the plant, and collect images of the collection area containing the plant and the ruler to obtain an image sequence.

[0024] In practice, the scale is a calibrated object of a known size (e.g., a piece of paper of a predetermined size or an object with a MagicRing, preferably an object with a MagicRing).

[0025] Place the ruler close to the plant, ensuring that its placement does not interfere with image acquisition of the plant, and that the ruler does not leave the acquisition area during image acquisition.

[0026] Then, a camera is used to capture video of the collection area containing plants and rulers. The obtained video is then uniformly framed to obtain an image sequence.

[0027] Step 102: Perform multi-view reconstruction on the image sequence to obtain plant point clouds and scale point clouds.

[0028] In practice, to ensure the three-dimensional reconstruction of the image sequence, the image sequence will be reconstructed into point clouds according to the plant's perspective and the scale's perspective, respectively, to obtain accurate plant point clouds and scale point clouds.

[0029] Step 103: Extract the scale ring point cloud from the scale point cloud, perform spatial coordinate analysis based on the scale ring point cloud, and determine the rotation matrix and scale factor.

[0030] In practice, the scale point cloud is filtered and denoised to extract a clean and complete scale ring point cloud. Then, spatial coordinates can be analyzed based on the scale ring point cloud, and the rotation matrix and scale factor of the scale ring point cloud relative to the actual size of the scale can be determined based on the actual position and size of the scale ring point cloud relative to the scale.

[0031] Step 104: Adjust the pose of the plant point cloud according to the rotation matrix to obtain the pose-adjusted plant point cloud (e.g., ...). Figure 6 (As shown).

[0032] In practice, since the plant and the scale are imaged within the same acquisition area, the rotation matrix and scale factor obtained from the scale are also applicable to the plant point cloud. The plant point cloud can then be adjusted in position and orientation according to the rotation matrix to obtain an accurately orientation-adjusted plant point cloud, ensuring that the orientation-adjusted plant point cloud is suitable for plant phenotypic analysis.

[0033] Step 105: Perform scale recovery processing on the pose-adjusted plant point cloud according to the scale factor to obtain the final plant point cloud (e.g., ...). Figure 6 (As shown).

[0034] In practice, the plant point cloud after attitude adjustment will be proportionally adjusted according to the scale factor to restore the plant point cloud to the size of the real plant. This way, the final plant point cloud obtained is more in line with the size of the real plant and the accuracy of the final plant point cloud is improved.

[0035] Step 106: Perform phenotypic analysis based on the final plant point cloud to obtain plant phenotypic parameters.

[0036] The above method involves pre-placing a ruler at the adjacent locations of the plant. This allows for image acquisition of the area containing both the plant and the ruler, resulting in an image sequence. To restore the plant's size using the ruler, the image sequence is reconstructed from multiple perspectives, generating plant and ruler point clouds. The ring-shaped portion of the ruler point cloud is then extracted to obtain a ruler-ring point cloud. Since the actual size of the ruler is known, spatial coordinate analysis of the ruler-ring point cloud allows for the determination of its relative dimensions. Given the actual size of the scale, the accurate rotation matrix and scale factor are determined. The resulting plant point cloud can then be attitude-adjusted according to the rotation matrix, resulting in a more accurate attitude position. Furthermore, the attitude-adjusted plant point cloud is scaled back according to the scale factor, making the final plant point cloud more consistent with the actual size of the plant, thus improving its accuracy. Consequently, phenotypic analysis based on this final plant point cloud will be more accurate and beneficial for plant phenotypic analysis, yielding more accurate plant phenotypic parameters.

[0037] In some embodiments, step 102 includes: Step 1021: Based on the image sequence, perform segmentation processing to obtain the plant region and the scale region.

[0038] In practice, a semantic segmentation model (e.g., SAM 2) is used to segment each frame of the image sequence, separating the plant region and the scale region.

[0039] Step 1022: Based on the plant region and the scale region, perform spatial reconstruction processing of the plant's corresponding viewpoint to obtain plant point cloud and scale point cloud.

[0040] The above-mentioned determination of plant areas falls into two categories: one is a plant area corresponding to a single plant, and the other is a plant area corresponding to multiple plants (for example, the image acquisition area in a community contains multiple plants).

[0041] In some embodiments, for the plant area corresponding to a single plant, step 1022 includes: Step A1: Determine the camera pose and sparse point cloud of the image based on the image sequence (e.g., ...). Figure 2 (As shown).

[0042] In practice, the image sequence is processed using the Structure for Motion (SfM) algorithm to estimate the camera pose and generate the corresponding sparse point cloud (e.g., ...). Figure 2 (As shown).

[0043] Step A2: In response to the plant region including a single plant, the sparse point cloud of the image is combined with the plant region and the scale region respectively, and 3D reconstruction is performed according to the camera pose to obtain the plant point cloud and the scale point cloud (e.g., ...). Figure 2 (As shown).

[0044] In practice, for the plant region of a single plant, the sparse point cloud obtained above is combined with the plant region and the scale region respectively, and the multi-view stereo vision algorithm (MVS) is used to reconstruct the three dimensions according to the camera pose, thereby obtaining accurate plant point cloud and scale point cloud.

[0045] The above scheme ensures that the plant point cloud and the scale point cloud are reconstructed under the same camera pose constraint, so that the plant point cloud and the scale point cloud are in a unified relative coordinate system, which facilitates subsequent adjustment of pose and size.

[0046] In some embodiments, for a plant area corresponding to multiple plants, step 1022 includes: Step B1: In response to the fact that the plant region includes multiple plants, the plant region is segmented to obtain plant instances containing multiple plants.

[0047] Step B2: Assign a corresponding color to each plant in a plant instance containing multiple plants, and determine the plant mask for each plant based on the color. The colors of different plants are different.

[0048] In practice, since the plant region contains multiple plants, it needs to be segmented. Image segmentation algorithms (such as K-Net) are used to segment the plant region into instances, resulting in plant instances of multiple plants. Each plant is then assigned a corresponding color, thus obtaining a plant mask of that color for each plant. Because different plants have different colors, they can be easily distinguished based on color.

[0049] Step B3: Determine the plant point cloud corresponding to each plant based on the plant mask of each plant.

[0050] In some embodiments, step B3 includes: Step B31: Obtain the scale information and rotation angle corresponding to the scale, and construct a transformation matrix based on the scale information and rotation angle. .

[0051] Step B32: Map the pixel coordinates corresponding to the plant mask of each plant according to the transformation matrix to obtain the physical coordinates of each plant (e.g., ...). Figure 7 (As shown).

[0052] In practice, a local coordinate system is constructed with the center of the scale (e.g., the center of the MagicRing) as the origin. This is used to determine the plant mask for the i-th plant in the t-th frame of the image sequence. Determine the geometric center in pixel space. Determine the plant physical coordinates corresponding to the plant mask of the i-th plant. The formula is: .

[0053] The plant mask for each plant is processed according to the above process, resulting in a set of plant physical coordinates for each plant (e.g., ...). Figure 7 (As shown) , where n is the number of plants corresponding to the plant region.

[0054] Step B33 involves spatially sorting and adjusting the distance constraints of the plant's physical coordinates to obtain the corresponding identity identifier for each plant.

[0055] In practice, spatial sorting and location adjustment involves setting the physical coordinates of each plant. According to dictionary order (preferred) Axis, second The axis is rearranged to make the plant index more stable in terms of topology.

[0056] Cost function calculation: Calculate the first Frame and the The distance matrix is ​​obtained by calculating the Euclidean distance between the physical coordinates of each plant in each frame. Distance matrix The elements corresponding to two different plants (i and j) in the data. Defined as:

[0057] Consistent Association: By minimizing the global matching cost Each plant is identified by its own unique identifier (e.g., ID). This establishes a stable correspondence between multiple frames of images, ensuring that each plant has a unique ID in the time series.

[0058] Step B34: Determine the camera pose, and perform 3D reconstruction of the plant instance containing multiple plants based on the camera pose to obtain a plant point cloud containing multiple plants (e.g., ...). Figure 7 (As shown).

[0059] In practice, based on the plant instances (e.g., multi-view 2D instances) and their corresponding camera poses that have been matched with the above, the structure of motion recovery (SfM) algorithm and the multi-view stereo vision (MVS) algorithm are used to synthesize plant point clouds containing multiple plants (e.g., dense 3D point clouds containing multiple plants).

[0060] Step B35 involves extracting individual plant points from the plant point cloud containing multiple plants according to their respective identifiers, resulting in the plant point cloud for each plant (e.g., ...). Figure 7 (As shown).

[0061] In practice, the unique identifiers of each plant (e.g., the ID index of the plant instance obtained in the 2D stage) are used as spatial constraints to precisely cut out each independent 3D plant individual from the plant point cloud containing multiple plants, thus obtaining the plant point cloud corresponding to each plant.

[0062] Using the above scheme, the physical coordinates of the plant mask corresponding to each plant can be determined based on the transformation matrix constructed by the scale. In this way, the identity of each plant can be accurately determined. Then, according to the spatial constraints corresponding to the identity, each independent three-dimensional plant individual is accurately cut out from the plant point cloud containing multiple plants after three-dimensional reconstruction, so as to obtain the accurate plant point cloud corresponding to each plant.

[0063] Step B4: Determine the camera pose and sparse point cloud of the image based on the image sequence.

[0064] In practice, the image sequence is processed using the Structure for Motion Recovery (SfM) algorithm to estimate the camera pose and generate the corresponding sparse point cloud.

[0065] Step B5: Combine the sparse point cloud of the image with the scale region, and perform 3D reconstruction according to the camera pose to obtain the scale point cloud.

[0066] In practice, the sparse point cloud of the image obtained above is combined with the scale region, and the multi-view stereo vision algorithm (MVS) is used to reconstruct the three dimensions according to the camera pose, thereby obtaining an accurate scale point cloud.

[0067] The above scheme can accurately segment plant regions containing multiple plants, obtaining plant point clouds for each plant, which facilitates subsequent pose and size adjustments. Furthermore, it can perform 3D reconstruction of the scale region based on the camera pose and the sparse point cloud of the image, thus ensuring that the obtained scale point cloud is more accurate.

[0068] For each plant point cloud identified above, steps 104 to 106 will be executed (and / or, the following is an expansion of steps 104, 105, and 106).

[0069] In some embodiments, step 103 includes: Step 1031: Extract the point cloud representing the target color of the scale ring from the scale point cloud, and use the point cloud of the target color as the scale ring point cloud.

[0070] In practice, since the scale is a target color (e.g., red, green, or blue, preferably red), the process of extracting the scale ring point cloud is as follows: (1) Generate an initial mask using the color threshold corresponding to the target color to obtain a preliminary target color point cloud. .

[0071] (2) Initial target color point cloud Perform statistical filtering (Statistical Outlier Removal) to remove isolated noise.

[0072] (3) Use Euclidean clustering to extract the largest connected clusters to obtain the final pure target color point cloud of MagicRing. ,in For actual quantity, Represents the three-dimensional real quantity, where N is the number of point clouds.

[0073] The final pure target color point cloud of MagicRing As a scale ring-shaped point cloud.

[0074] Step 1032: Perform principal component analysis on the scale ring point cloud to determine multiple principal eigenvectors, construct a new coordinate system by projecting the multiple principal eigenvectors, and determine the rotation matrix based on the positional relationship between the original coordinate system and the new coordinate system corresponding to the scale ring point cloud.

[0075] In some embodiments, step 1032 includes: Step 10321: Perform principal component analysis on the scale ring point cloud to determine three principal eigenvectors.

[0076] In specific implementation, such as Figure 3 As shown, for the scale ring point cloud (e.g., Performing Principal Component Analysis (PCA) yields three principal components: the first principal component (1st PC), the second principal component (2nd PC), and the third principal component (3rd PC), which correspond to three principal eigenvectors. .

[0077] Step 10322: Determine the projected area of ​​the scale ring point cloud on each principal feature vector, and select the principal feature vector with the smallest projected area as the target feature vector.

[0078] Step 10323: Take the coordinate axis corresponding to the target feature vector as the new vertical coordinate axis, and take the coordinate axes corresponding to the remaining two main feature vectors (excluding the target feature vector) as the new planar coordinate axes.

[0079] In practice, to identify the normal direction corresponding to the scale ring point cloud, the projected area of ​​the scale ring point cloud in the direction corresponding to each principal eigenvector is calculated. The specific calculation formula is as follows: .

[0080] Scale ring point cloud The normal to the plane is the Z-axis of the new coordinate system (e.g.) Figure 3 R in z (θ)), the coordinate axis of the principal eigenvector corresponding to the minimum projected area. The X and Y axes of the new coordinate system (i.e., the new plane coordinate axes) are the coordinate axes corresponding to the other two principal eigenvectors.

[0081] Step 10324: Construct a new coordinate system according to the new vertical coordinate axis and the new planar coordinate axis (e.g., Figure 3 (As shown).

[0082] Step 10325: Determine the original coordinate system corresponding to the scale ring point cloud, and determine the rotation axis and rotation angle based on the rotation relationship between the original vertical coordinate axis corresponding to the original coordinate system and the new vertical coordinate axis.

[0083] In practice, due to point clouds Being in the original coordinate system, it is necessary to calculate the rotation matrix from the original coordinate system to the new coordinate system. Let the Z-axis of the original coordinate system be... Calculated using Rodrigues' rotation formula Rotate to the Z-axis of the new coordinate system The rotation matrix. First, calculate the rotation axis. and rotation angle : .

[0084] Step 10326, construct the rotation matrix based on the rotation axis and rotation angle (e.g., Figure 3 (As shown).

[0085] In practice, according to Rodrigues' formula, the rotation matrix is: ;in It is the identity matrix. For the axis of rotation An antisymmetric matrix.

[0086] The above method can obtain an accurate rotation matrix, which facilitates the rotation processing of plant point clouds based on the rotation matrix.

[0087] Step 1033: After rotating the scale ring point cloud according to the rotation matrix, a new scale ring point cloud is obtained. The proportional relationship between the new scale ring point cloud and the original scale ring point cloud is determined to obtain the scale factor.

[0088] In some embodiments, step 1033 includes: Step 10331: Transpose the rotation matrix to obtain the transposed rotation matrix.

[0089] Step 10332: Multiply the transposed rotation matrix with the scale ring point cloud to obtain a new scale ring point cloud (e.g., Figure 4 (As shown).

[0090] In practice, the new scale annular point cloud The corresponding formula is: .

[0091] Step 10333: Project the new scale ring point cloud onto a two-dimensional plane defined by the new plane coordinate axes (e.g., the two-dimensional plane formed by the XY axes) to obtain the scale ring point cloud projection. .

[0092] Step 10334: Perform preliminary fitting and denoising of the circular model based on the scale annular point cloud projection, removing noise points in the scale annular point cloud projection that deviate from the circular model, and obtaining the set of interior points corresponding to the circular model (e.g., ...). Figure 5 (As shown).

[0093] In practice, the RANSAC algorithm is used to determine the circular model. ,in, Center coordinate, Using the radius as the internal point, noise points deviating from the circular model in the scale ring point cloud projection are removed to obtain the set of internal points corresponding to the circular model. The specific criteria for internal point selection are as follows: ,in, for The i-th point in the middle, This is the distance threshold.

[0094] The specific process is as follows: Randomly select 3 non-collinear points from all scale ring point cloud projections. Use these 3 points to uniquely determine a circle (usually by solving the intersection of perpendicular bisectors or linearizing the circle equation) as the initial circle model. Use the center and radius of the obtained initial circle model to check the consistency of other points. Count the number of interior points according to the above judgment criteria. Repeat this process multiple times and select the initial circle model with the most interior points as the final circle model.

[0095] Step 10335: Correct the set of interior points using the least squares method to obtain the corrected circle model (e.g., ...). Figure 5 (As shown).

[0096] In practice, the selected set of interior points The circle parameters are refined using the least squares method, and the algebraic distance is minimized to obtain the corrected circle model.

[0097] The specific refinement formula is as follows: ,in, for The i-th point in the array.

[0098] Step 10336: Determine the target diameter corresponding to the corrected circle model, retrieve the actual diameter corresponding to the scale, and use the ratio of the actual diameter to the target diameter as the scale factor.

[0099] In practical implementation, the target diameter for , scale factor The calculation formula is: ,in, This is the actual diameter.

[0100] The above method ensures higher accuracy of the obtained scale factor, which, together with the rotation matrix, can accurately reconstruct the scale of plant point clouds.

[0101] In some embodiments, step 106 includes: Step 1061: Determine at least one of the plant height and crown width for the final plant point cloud.

[0102] And / or, in step 1062, determine the plant leaves based on the final plant point cloud and calculate the total leaf area.

[0103] And / or, in step 1063, determine the plant canopy based on the final plant point cloud, and determine the canopy projection area of ​​the plant canopy.

[0104] Plant phenotypic parameters include at least one of the following: Plant height, crown width, total leaf area, and canopy projection area.

[0105] The above scheme, based on the accurate rotation matrix and scale factor determined by the scale point cloud corresponding to the scale, allows for rotation according to the rotation matrix and scale recovery processing according to the scale factor. This results in a final plant point cloud that better reflects the true morphology and size of the plant. Consequently, phenotypic analysis based on the final plant point cloud yields more accurate plant phenotypic parameters.

[0106] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0107] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides a plant phenotyping device based on the recovery scale.

[0109] refer to Figure 8 The device includes: The image acquisition module 201 is configured to place a ruler at a position adjacent to the plant, acquire images of the acquisition area containing the plant and the ruler, and obtain an image sequence. Point cloud reconstruction module 202 is configured to perform multi-view reconstruction on the image sequence to obtain plant point cloud and scale point cloud; Spatial analysis module 203 is configured to extract scale ring point cloud from the scale point cloud, perform spatial coordinate analysis based on the scale ring point cloud, and determine rotation matrix and scale factor; The attitude adjustment module 204 is configured to adjust the attitude of the plant point cloud according to the rotation matrix to obtain the attitude-adjusted plant point cloud. The scale recovery module 205 is configured to perform scale recovery processing on the pose-adjusted plant point cloud according to the scale factor to obtain the final plant point cloud. Phenotypic analysis module 206 is configured to perform phenotypic analysis based on the final plant point cloud to obtain plant phenotypic parameters.

[0110] In some embodiments, the point cloud reconstruction module 202 is specifically configured as follows: Based on the image sequence, segmentation processing is performed to obtain the plant region and the scale region; Based on the plant region and the scale region, spatial reconstruction processing is performed on the corresponding viewpoint of the plant to obtain plant point clouds and scale point clouds.

[0111] In some embodiments, the point cloud reconstruction module 202 is further configured as follows: In response to the fact that the plant region includes multiple plants, the plant region is segmented to obtain plant instances containing multiple plants; Assign a corresponding color to each plant in a plant instance containing multiple plants, and determine the plant mask for each plant based on the color. The colors of different plants are different. The plant point cloud corresponding to each plant is determined based on the plant mask of each plant. The camera pose and sparse point cloud of the image are determined based on the image sequence; The sparse point cloud of the image is combined with the scale region, and the scale point cloud is obtained by 3D reconstruction according to the camera pose.

[0112] In some embodiments, the point cloud reconstruction module 202 is further configured as follows: Obtain the scale information and rotation angle corresponding to the scale, and construct a transformation matrix based on the scale information and rotation angle; The pixel coordinates corresponding to the plant mask of each plant are mapped according to the transformation matrix to obtain the physical coordinates of each plant. The physical coordinates of each plant are spatially sorted and distance-constrained to obtain the unique identifiers for each plant. Determine the camera pose, and reconstruct the three-dimensional plant instance containing multiple plants based on the camera pose to obtain a plant point cloud containing multiple plants. Plant point clouds containing multiple plants are extracted individually according to the unique identifiers of each plant to obtain the plant point cloud for each plant.

[0113] In some embodiments, the point cloud reconstruction module 202 is further configured as follows: The camera pose and sparse point cloud of the image are determined based on the image sequence; In response to the presence of a single plant in the plant region, the sparse point cloud of the image is combined with the plant region and the scale region respectively, and the plant point cloud and the scale point cloud are obtained by 3D reconstruction according to the camera pose.

[0114] In some embodiments, the spatial analysis module 203 is specifically configured as follows: Extract the point cloud representing the target color of the scale ring from the scale point cloud, and use the point cloud of the target color as the scale ring point cloud; Principal component analysis is performed on the scale ring point cloud to determine multiple principal eigenvectors. The multiple principal eigenvectors are then projected to construct a new coordinate system. Based on the positional relationship between the original coordinate system and the new coordinate system corresponding to the scale ring point cloud, the rotation matrix is ​​determined. After rotating the scale ring point cloud according to the rotation matrix, a new scale ring point cloud is obtained. The proportional relationship between the new scale ring point cloud and the original scale ring point cloud is determined to obtain the scale factor.

[0115] In some embodiments, the spatial analysis module 203 is further configured to: Principal component analysis was performed on the scale ring point cloud to determine three principal eigenvectors; Determine the projected area of ​​the scale ring point cloud on each principal feature vector, and select the principal feature vector with the smallest projected area as the target feature vector; The coordinate axis corresponding to the target feature vector is taken as the new vertical coordinate axis, and the coordinate axes corresponding to the remaining two principal feature vectors (excluding the target feature vector) among the three principal feature vectors are taken as the new planar coordinate axes. Construct a new coordinate system based on the new vertical coordinate axis and the new planar coordinate axis; Determine the original coordinate system corresponding to the scale ring point cloud, and determine the rotation axis and rotation angle based on the rotation relationship between the original vertical coordinate axis and the new vertical coordinate axis corresponding to the original coordinate system. The rotation matrix is ​​constructed based on the rotation axis and rotation angle.

[0116] In some embodiments, the spatial analysis module 203 is further configured to: The rotation matrix is ​​transposed to obtain the transposed rotation matrix; Multiply the transposed rotation matrix with the scale ring point cloud to obtain a new scale ring point cloud; The new scale ring point cloud is projected onto the two-dimensional plane defined by the new plane coordinate axes to obtain the scale ring point cloud projection. Based on the scale ring point cloud projection, a preliminary fitting and denoising process for the circular model is performed to remove noise points that deviate from the circular model in the scale ring point cloud projection, thereby obtaining the set of interior points corresponding to the circular model. The set of interior points is corrected using the least squares method to obtain the corrected circle model; Determine the target diameter corresponding to the corrected circle model, retrieve the actual diameter corresponding to the scale, and use the ratio of the actual diameter to the target diameter as the scale factor.

[0117] In some embodiments, the phenotypic analysis module 206 is specifically configured as follows: For the final plant point cloud, determine at least one of the following: plant height, crown width, and vertical distribution curve; and / or, Based on the final plant point cloud, determine the plant leaves and calculate the total leaf surface area; and / or, The plant canopy is determined based on the final plant point cloud; and the ground projection area of ​​the plant canopy is determined; and / or, Determine the principal projected area of ​​the ground corresponding to the final plant point cloud; and / or, The final plant point cloud is segmented into organs, and the convex hull area of ​​the ground projection surface corresponding to each segmented organ is determined.

[0118] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0119] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0120] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0121] Figure 9 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0122] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0123] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0124] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0125] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0126] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0127] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0128] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0129] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0130] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0131] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0132] Based on the same concept, corresponding to any of the above embodiments, this disclosure also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0133] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0134] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0135] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0136] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0137] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0138] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0139] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0140] The embodiments disclosed herein are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this disclosure. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for plant phenotyping based on recovery scale, characterized by, include: A ruler is placed adjacent to the plant, and images are captured in the area containing the plant and the ruler to obtain an image sequence. The image sequence is reconstructed from multiple perspectives to obtain plant point clouds and scale point clouds; Extract scale ring point cloud from the scale point cloud, perform spatial coordinate analysis based on the scale ring point cloud, and determine the rotation matrix and scale factor; The plant point cloud is then subjected to attitude adjustment according to the rotation matrix to obtain an attitude-adjusted plant point cloud. The plant point cloud after attitude adjustment is scaled according to the scale factor to obtain the final plant point cloud. Phenotypic analysis was performed on the final plant point cloud to obtain plant phenotypic parameters.

2. The method of claim 1, wherein, The process of reconstructing the image sequence from multiple perspectives to obtain plant point clouds and scale point clouds includes: Based on the image sequence, segmentation processing is performed to obtain the plant region and the scale region; Based on the plant region and the scale region, spatial reconstruction processing is performed on the corresponding viewpoint of the plant to obtain plant point clouds and scale point clouds.

3. The method of claim 2, wherein, The reconstruction process of the plant point cloud and the scale point cloud includes: In response to the fact that the plant region includes multiple plants, the plant region is segmented to obtain plant instances containing multiple plants; Assign a corresponding color to each plant in a plant instance containing multiple plants, and determine the plant mask for each plant based on the color. The colors of different plants are different. The plant point cloud corresponding to each plant is determined based on the plant mask of each plant. The camera pose and sparse point cloud of the image are determined based on the image sequence; The sparse point cloud of the image is combined with the scale region, and the scale point cloud is obtained by 3D reconstruction according to the camera pose.

4. The method of claim 3, wherein, The process of determining the plant point cloud corresponding to each plant based on the plant mask of each plant includes: Obtain the scale information and rotation angle corresponding to the scale, and construct a transformation matrix based on the scale information and rotation angle; The pixel coordinates corresponding to the plant mask of each plant are mapped according to the transformation matrix to obtain the physical coordinates of each plant. The physical coordinates of each plant are spatially sorted and distance-constrained to obtain the unique identifiers for each plant. Determine the camera pose, and reconstruct the plant instance containing multiple plants in 3D based on the camera pose to obtain a plant point cloud containing multiple plants. Plant point clouds containing multiple plants are extracted individually according to the unique identifiers of each plant to obtain the plant point cloud for each plant.

5. The method of claim 2, wherein, The reconstruction process of the plant point cloud and the scale point cloud includes: The camera pose and sparse point cloud of the image are determined based on the image sequence; In response to the presence of a single plant in the plant region, the sparse point cloud of the image is combined with the plant region and the scale region respectively, and the plant point cloud and the scale point cloud are obtained by 3D reconstruction according to the camera pose.

6. The method of claim 1, wherein, The step of extracting scale ring point cloud from the scale point cloud, performing spatial coordinate analysis based on the scale ring point cloud, and determining the rotation matrix and scale factor includes: Extract the point cloud representing the target color of the scale ring from the scale point cloud, and use the point cloud of the target color as the scale ring point cloud; Principal component analysis is performed on the scale ring point cloud to determine multiple principal eigenvectors. The multiple principal eigenvectors are then projected to construct a new coordinate system. Based on the positional relationship between the original coordinate system and the new coordinate system corresponding to the scale ring point cloud, the rotation matrix is ​​determined. After rotating the scale ring point cloud according to the rotation matrix, a new scale ring point cloud is obtained. The proportional relationship between the new scale ring point cloud and the original scale ring point cloud is determined to obtain the scale factor.

7. The method of claim 6, wherein, The process of performing principal component analysis on the scale ring point cloud to determine multiple principal eigenvectors, constructing a new coordinate system from these principal eigenvectors through projection processing, and determining the rotation matrix based on the positional relationship between the original coordinate system and the new coordinate system corresponding to the scale ring point cloud includes: Principal component analysis was performed on the scale ring point cloud to determine three principal eigenvectors; Determine the projected area of ​​the scale ring point cloud on each principal feature vector, and select the principal feature vector with the smallest projected area as the target feature vector; The coordinate axis corresponding to the target feature vector is taken as the new vertical coordinate axis, and the coordinate axes corresponding to the remaining two principal feature vectors (excluding the target feature vector) among the three principal feature vectors are taken as the new planar coordinate axes. Construct a new coordinate system based on the new vertical coordinate axis and the new planar coordinate axis; Determine the original coordinate system corresponding to the scale ring point cloud, and determine the rotation axis and rotation angle based on the rotation relationship between the original vertical coordinate axis and the new vertical coordinate axis corresponding to the original coordinate system. The rotation matrix is ​​constructed based on the rotation axis and rotation angle.

8. The method of claim 6, wherein, After rotating the scale ring point cloud according to the rotation matrix, a new scale ring point cloud is obtained. The proportional relationship between the new scale ring point cloud and the original scale ring point cloud is determined to obtain the scale factor, including: The rotation matrix is ​​transposed to obtain the transposed rotation matrix; Multiply the transposed rotation matrix by the scale ring point cloud to obtain a new scale ring point cloud; The new scale ring point cloud is projected onto the two-dimensional plane defined by the new plane coordinate axes to obtain the scale ring point cloud projection. Based on the scale ring point cloud projection, a preliminary fitting and denoising process for the circular model is performed to remove noise points that deviate from the circular model in the scale ring point cloud projection, thereby obtaining the set of interior points corresponding to the circular model. The set of interior points is corrected using the least squares method to obtain the corrected circle model; Determine the target diameter corresponding to the corrected circle model, retrieve the actual diameter corresponding to the scale, and use the ratio of the actual diameter to the target diameter as the scale factor.

9. The method of claim 1, wherein, The step of performing phenotypic analysis based on the final plant point cloud to obtain plant phenotypic parameters includes: Determine at least one of the plant height and crown width from the final plant point cloud; and / or, Based on the final plant point cloud, determine the plant leaves and calculate the total leaf surface area; and / or, The plant canopy is determined based on the final plant point cloud, and the ground projection area of ​​the plant canopy is determined.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the method as claimed in any one of claims 1 to 9 when executing the program.

Images