Method, device and equipment for extracting point cloud data of target plants from multiple plants

Abstract

The present application provides a method, device and equipment for extracting point cloud data of a target plant from multiple plants. The two-dimensional characteristics of the depth image are used to simplify the problem of three-dimensional point cloud separation into a two-dimensional image processing task, and the plant positions of the depth images of multiple plants are identified and processed. The depth image containing one plant is extracted as the depth image of the target plant, and then the threshold segmentation algorithm is used to remove non-target areas such as the ground and weeds from the depth image of the target plant, and the foreground area is retained. Based on this, the point cloud data of the target plant can be accurately and efficiently restored.

Description

Technical Field

[0001] The present application relates to the field of plant phenotyping technology, and in particular to a method, device and equipment for extracting point cloud data of a target plant from multiple plants. Background Art

[0002] As a three-dimensional expression of plant structure information, point cloud can accurately reflect the morphological characteristics of plants, such as plant height, crown width, and biomass. These characteristics play an important guiding role in the fields of plant breeding, plant monitoring, and crop management.

[0003] Based on the above situation, the existing technology often uses a three-dimensional point cloud segmentation method, but this method is difficult to meet the needs of separating multiple plants, and thus cannot accurately and efficiently extract the point cloud of the target plant from multiple plant point clouds in a complex environment. Summary of the invention

[0004] In view of this, the purpose of the present application is to propose a method, device and equipment for extracting point cloud data of a target plant from multiple plants, so as to solve the above-mentioned technical problems.

[0005] Based on the above purpose, the first aspect of the present application provides a method for extracting point cloud data of a target plant from multiple plants, comprising:

[0006] Acquire depth images of multiple plants;

[0007] Identify and process the position of each target plant in the depth image of multiple plants to obtain a position identification result of the target plant, and generate a region of interest of the target plant based on the position identification result of the target plant and a preset region of interest size;

[0008] For each target plant's region of interest, do the following:

[0009] The region of interest of the target plant is processed using a threshold segmentation algorithm to determine the foreground region corresponding to the region of interest;

[0010] According to the foreground area corresponding to the area of ​​interest, the depth images of the multiple plants are extracted and processed to obtain the depth image corresponding to the foreground area;

[0011] The depth image corresponding to the foreground area is processed by point cloud recovery to obtain the point cloud data of the target plant.

[0012] Based on the same inventive concept, the second aspect of the present application provides a device for extracting point cloud data of a target plant from multiple plants, comprising:

[0013] A depth image acquisition module is configured to acquire depth images of multiple plants;

[0014] A region of interest generation module is configured to identify and process the position of each target plant in the depth image of multiple plants, obtain a position identification result of the target plant, and generate a region of interest of the target plant based on the position identification result of the target plant and a preset region of interest size;

[0015] The target plant processing module is configured to perform the following operations for each target plant: use a threshold segmentation algorithm to process the region of interest of the target plant to determine the foreground area corresponding to the region of interest; extract and process the depth images of multiple plants based on the foreground area corresponding to the region of interest to obtain the depth image corresponding to the foreground area; perform point cloud recovery processing on the depth image corresponding to the foreground area to obtain point cloud data of the target plant.

[0016] Based on the same inventive concept, the third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executed by the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0017] Based on the same inventive concept, the fourth aspect of the present application provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect above.

[0018] From the above, it can be seen that the method, device and equipment for extracting point cloud data of a target plant from multiple plants provided in the present application utilize the two-dimensional characteristics of the depth image to simplify the problem of three-dimensional point cloud separation into a two-dimensional image processing task, identify and process the plant positions of the depth images of multiple plants, extract the depth image containing one plant as the depth image of the target plant, and then use the threshold segmentation algorithm to remove non-target areas such as the ground and weeds from the depth image of the target plant, retain the foreground area, and based on this, accurately and efficiently restore the point cloud data of the target plant. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of a method for extracting point cloud data of a target plant from multiple plants according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a point cloud of three soybean plants according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a deep pseudo-color image of a point cloud of three soybean plants according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a depth grayscale image of a point cloud of three soybean plants according to an embodiment of the present application;

[0024] Figure 5 A schematic diagram of an anchor frame of each plant obtained by performing target detection on a depth map in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of obtaining a depth map of each plant according to an anchor frame range according to an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of a single plant depth map according to an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a binary mask image generated from a depth image using the Otsu threshold method and the Gaussian filter method according to an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of obtaining pixel points in the depth map (i.e., the target plant depth map) corresponding to white pixel points in the mask map according to a one-to-one correspondence between pixels in the depth map and the mask map according to an embodiment of the present application;

[0029] Fig.10 A schematic diagram of a top view of a plant point cloud restored from an updated depth map according to an embodiment of the present application;

[0030] Fig.11 A schematic diagram of a processing flow of obtaining a point cloud of a single rice plant, taking field rice as an example point cloud in an embodiment of the present application;

[0031] Fig.12 A structural block diagram of a device for extracting point cloud data of a target plant from multiple plants in an embodiment of the present application;

[0032] Fig.13 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0035] It is understandable that before using the technical solutions of each embodiment of the present application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0036] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.

[0037] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of this application. Other methods that meet relevant laws and regulations may also be applied to the implementation method of this application.

[0039] As a three-dimensional expression of plant structure information, point cloud can accurately reflect the morphological characteristics of plants, such as plant height, crown width, and biomass. These characteristics play an important guiding role in the fields of plant breeding, plant monitoring, and crop management.

[0040] Based on the above situation, the existing technology often uses a three-dimensional point cloud segmentation method, but this method is difficult to meet the needs of separating multiple plants, and thus cannot accurately and efficiently extract the point cloud of the target plant from multiple plant point clouds in a complex environment.

[0041] The present application found that the method of 3D point cloud segmentation is difficult to meet the needs of separating multiple plants, and thus cannot accurately and efficiently extract the point cloud of the target plant from multiple plant point clouds in a complex environment. One of the problems is that, for example, there is interference from weeds, the ground, and the influence of noise and outliers. These problems not only increase the difficulty of point cloud processing, but also affect the accuracy and processing efficiency of the final data.

[0042] Traditional 3D point cloud segmentation methods often rely on complex algorithms and manual adjustments, with low processing efficiency and difficulty in meeting the needs of separating multiple plants. Therefore, how to accurately and efficiently extract the point cloud of a single target plant from multiple plant point clouds in a complex environment without increasing the complexity of data processing has become one of the current focuses in the field of plant phenotyping.

[0043] In response to the above problems, this application proposes a point cloud separation method based on depth information and target detection (3D-DepthYOLO). This method uses the two-dimensional characteristics of the depth map to simplify the problem of three-dimensional point cloud separation into a two-dimensional image processing task. A high-quality data set is generated by normalizing the depth map, and then combined with a deep learning target detection model to achieve accurate segmentation of individual plants. Then, threshold segmentation and masking techniques are used to remove non-target areas such as the ground and weeds, and restore the point cloud model of a single plant. This innovative method not only improves the efficiency and accuracy of plant point cloud separation, but also effectively solves the impact of interference between flower pots and the ground on point cloud processing, providing strong technical support for the field of plant phenotyping.

[0044] An embodiment of the present application provides a method for extracting point cloud data of a target plant from multiple plants. The two-dimensional characteristics of the depth image are utilized to simplify the problem of three-dimensional point cloud separation into a two-dimensional image processing task. The plant positions in the depth images of multiple plants are identified and processed, and the depth image containing one plant is extracted as the depth image of the target plant. Then, a threshold segmentation algorithm is used to remove non-target areas such as the ground and weeds from the depth image of the target plant, and the foreground area is retained. Based on this, the point cloud data of the target plant can be accurately and efficiently restored.

[0045] like Figure 1 As shown, the method of this embodiment includes:

[0046] Step 101, obtaining depth images of multiple plants.

[0047] In this step, the depth image, also known as the range image, refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value. It directly reflects the geometry of the visible surface of multiple plants and is the projection of three-dimensional spatial information on a two-dimensional plane. Each pixel in the depth image represents a specific distance value, which is the straight-line distance from the image collector (such as a depth sensor) to the corresponding point in the scene.

[0048] Methods for obtaining depth images of multiple plants include but are not limited to the following methods:

[0049] LiDAR depth imaging method, computer stereo vision imaging, coordinate measuring machine method, moiré fringe method and structured light method, etc.

[0050] The point cloud data of multiple plants can also be converted into depth images of multiple plants through the point cloud data of multiple plants.

[0051] Step 102, identifying and processing the position of each target plant in the depth image of multiple plants to obtain a position identification result of the target plant, and generating a region of interest of the target plant based on the position identification result of the target plant and a preset region of interest size.

[0052] In this step, the depth image of multiple plants means a group or a series of depth images including multiple plants.

[0053] The present application does not directly process three-dimensional point cloud data, but processes two-dimensional depth images of multiple plants, thereby selecting or segmenting at least one plant's depth image from the two-dimensional depth images of multiple plants to achieve the purpose of separating multiple plants.

[0054] The target plant represents a specific individual plant of interest, or an individual plant that requires special attention under certain conditions (such as growth status, variety, etc.).

[0055] In the depth image of the target plant, an image processing algorithm or machine learning model is applied to identify the location of the target plant. This may involve steps such as edge detection, shape matching, and feature point extraction, aiming to separate the target plant from the depth image of the target plant and determine its precise boundary or center point location.

[0056] After the position recognition process is completed, one or more parameters (such as coordinates, bounding boxes, etc.) will be obtained, which accurately describe the position of the target plant in the depth image. These parameters constitute the position recognition result of the target plant.

[0057] The size of the region of interest (ROI size) is set based on the needs of the actual application. The size may be a fixed value or it may be dynamically adjusted according to the size of the target plant.

[0058] Based on the position recognition result of the target plant and the preset size of the region of interest, a region around the target plant is generated in the depth image. This region is the region of interest (ROI) of the target plant. The region of interest is outlined in the form of a box, circle, ellipse, irregular polygon, etc.

[0059] This allows subsequent analysis or processing to focus on this area, thereby improving efficiency and reducing unnecessary data processing.

[0060] Among them, the region of interest can be reflected in the depth image in the form of an anchor box.

[0061] For each target plant's region of interest, do the following:

[0062] Step 103, using a threshold segmentation algorithm to process the region of interest of the target plant, and determine the foreground region corresponding to the region of interest.

[0063] In this step, the threshold segmentation algorithm distinguishes the foreground and background based on the grayscale or color features of the depth image.

[0064] One or more gray level thresholds are selected for the depth image, and then each pixel in the depth image is compared to these thresholds. Based on the comparison results, the pixel is classified as foreground or background.

[0065] For example, if the grayscale range of the depth image is 0 to 255, you can choose a threshold (such as 128), and then consider all pixels with grayscale values ​​less than 128 as background, and pixels greater than or equal to 128 as foreground.

[0066] When using the threshold segmentation algorithm to process the region of interest of the target plant, the goal is to distinguish the plant part in the ROI from the background (such as soil, sky, and other non-target objects).

[0067] By choosing a suitable threshold, the pixels in the ROI can be classified into foreground (usually plant parts) and background.

[0068] The use of threshold segmentation algorithm helps to accurately extract the foreground area from the depth image with complex background, providing support for subsequent analysis and decision-making.

[0069] The foreground area corresponding to the region of interest may be in the form of a binary mask image, in which the white area represents the target plant itself, and the black area represents the background area.

[0070] Step 104 , extracting and processing the depth images of the multiple plants according to the foreground area corresponding to the area of ​​interest, to obtain a depth image corresponding to the foreground area.

[0071] In this step, the foreground area corresponding to the region of interest of the target plant is used as a mask for the depth image of the original point cloud data (i.e., the depth image of multiple plants). Since the binary mask image (i.e., the foreground area) of a single plant (i.e., the target plant) is consistent with the size of the depth image of the original point cloud data, the corresponding coordinates in the depth image of the original point cloud data are extracted according to the coordinates of the non-black pixels in the binary mask image, and the depth image corresponding to the two-dimensional mask image of the plant (i.e., the depth image corresponding to the foreground area) can be obtained.

[0072] The extraction process can accurately separate the part corresponding to the foreground area from the depth image of the original point cloud data. The extraction process involves image segmentation techniques such as threshold segmentation, edge detection, region growing, graph-based segmentation, etc., or semantic segmentation using deep learning methods to accurately separate the foreground area from the background.

[0073] The depth image corresponding to the foreground area only contains the depth information of the foreground area, so that more attention can be paid to processing the plants in the foreground area.

[0074] Step 105 , performing point cloud recovery processing on the depth image corresponding to the foreground area to obtain point cloud data of the target plant.

[0075] In this step, the point cloud data is a collection of points with position information in three-dimensional space. These points can represent the shape and position of the surface of an object. The process of recovering a point cloud from a depth image is to convert each pixel (and its depth value) in the depth image into a point in three-dimensional space, thereby constructing a three-dimensional point cloud representing the surface of a scene or object. This process involves converting two-dimensional image coordinates and depth values ​​into three-dimensional world coordinates.

[0076] After point cloud recovery processing from the depth image corresponding to the foreground area, the three-dimensional point cloud data of the target plant is obtained accurately and efficiently. The point cloud data provides an accurate three-dimensional representation of the plant surface and can be used in a variety of applications, such as three-dimensional reconstruction, plant morphology analysis, growth monitoring, etc.

[0077] In this way, the final extracted point cloud data only contains the target plants, removing unnecessary background (ground) and noise point clouds.

[0078] Through the above scheme, the two-dimensional characteristics of the depth image are utilized to simplify the problem of three-dimensional point cloud separation into a two-dimensional image processing task, the plant positions of the depth images of multiple plants are identified and processed, and the depth image containing one plant is extracted as the depth image of the target plant. Then, the threshold segmentation algorithm is used to remove non-target areas such as the ground and weeds from the depth image of the target plant, and the foreground area is retained. Based on this, the point cloud data of the target plant can be accurately and efficiently restored.

[0079] In some embodiments, the position recognition result of the target plant and the region of interest of the target plant are obtained using a trained target detection model. The target detection model is trained using a total loss function. The process of obtaining the total loss function includes:

[0080] Step A1, obtain the true label and predicted recognition result for training.

[0081] Step A2, determine the classification loss function according to the actual label used for training and the predicted recognition result by the following formula:

[0082]

[0083] in, represents the classification loss function, represents the true label used for training, Represents the predicted recognition result, represents the number of depth images of the target plant used for training, Represents the order of depth images of target plants used for training.

[0084] Step A3, obtaining the real coordinates of the region of interest and the predicted coordinates of the region of interest for training.

[0085] Step A4, determining the regression loss function according to the actual coordinates of the region of interest used for training and the predicted coordinates of the region of interest by the following formula:

[0086]

[0087] in, represents the regression loss function, represents the real region of interest coordinates used for training, represents the predicted coordinates of the region of interest, represents the number of coordinates of the region of interest, Indicates the order of the coordinates of the region of interest, represents the smoothed L1 loss.

[0088] Step A5, obtain the first The real existence of the region of interest, and the The predicted existence of a region of interest.

[0089] Step A6, according to the first The real existence of the region of interest, and the The predicted existence of the region of interest is determined by the following formula to determine the confidence loss function:

[0090]

[0091] in, represents the confidence loss function, represents the first The real existence of the region of interest, Indicates The predicted existence of regions of interest, Indicates the number of regions of interest.

[0092] Step A7, based on the classification loss function, the regression loss function and the confidence loss function, the total loss function is determined by the following formula:

[0093]

[0094] in, represents the total loss function, represents the classification loss function, represents the regression loss function, represents the confidence loss function.

[0095] In the above scheme, the target detection model is a target detection model based on YOLOv8.

[0096] The classification loss function is used to evaluate the accuracy of model classification, and cross entropy loss is commonly used.

[0097] The regression loss function is used to evaluate the gap between the predicted anchor box (predicted region of interest) and the true anchor box (true region of interest), using loss.

[0098] The confidence loss is used to evaluate the accuracy of the model's prediction of the object existence of each anchor box (the predicted existence of the region of interest), and the measurement method is binary cross entropy.

[0099] During the training process, data enhancement methods are used to improve the generalization ability of the model, including: random scaling, scaling the image to generate samples of various scales; random cropping, randomly cropping a part from the image to increase the diversity of samples; random rotation, rotating the image at random angles.

[0100] In some embodiments, the target detection model is obtained by training by back-propagation updating model parameters, and the process of obtaining the back-propagation updating model parameters includes:

[0101] Step B1, obtaining the parameters of the target detection model of the previous iteration process of the current iteration process, the preset learning rate and the preset gradient of the total loss function.

[0102] Step B2, based on the parameters of the target detection model of the previous iteration process of the current iteration process, the preset learning rate and the preset gradient of the total loss function, the parameters of the target detection model of the current iteration process are determined by the following formula, and the parameters of the target detection model of the current iteration process are used as the back propagation update model parameters:

[0103]

[0104] in, Represents the parameters of the target detection model in the current iteration process, Represents the parameters of the target detection model in the previous iteration of the current iteration. represents the preset learning rate, Represents the preset gradient of the total loss function.

[0105] In the above scheme, when training the object detection model, multiple iterations (or training rounds) are usually performed.

[0106] The model's parameters (such as weights and biases) are updated in each iteration in the hope of reducing the loss in the next iteration.

[0107] Before starting the current iteration process, you first need to obtain the parameters of the model at the end of the previous iteration process.

[0108] These parameters are what the model learned after the previous round of training and will serve as the basis for the current iteration.

[0109] The learning rate is a hyperparameter that controls the magnitude of parameter updates.

[0110] The gradient of the total loss function refers to the derivative of the loss function with respect to the model parameters, which indicates the direction in which the total loss function decreases fastest under the current parameter values.

[0111] The model parameters of the current iteration process are the model parameters of the previous iteration process of the current iteration process minus the learning rate multiplied by the gradient of the total loss function.

[0112] This process is repeated until the model's performance on the validation set no longer improves significantly, or the preset number of iterations is reached.

[0113] In some embodiments, step 103 includes:

[0114] Step C1, obtaining the coordinates of the region of interest of the target plant.

[0115] Step C2, based on the coordinates of the region of interest of the target plant, determine the coordinates of the first vertex angle and the coordinates of the second vertex angle of the region of interest of the target plant by the following formula;

[0116]

[0117] in, represents the abscissa of the first vertex, represents the ordinate of the first vertex, represents the abscissa of the second vertex, represents the ordinate of the second vertex, The horizontal coordinate represents the center position of the region of interest. The ordinate represents the center position of the region of interest, represents the width of the center position of the region of interest, Indicates the height of the center of the area of ​​interest.

[0118] Step C3, using the coordinates of the first vertex and the second vertex of the region of interest of the target plant, the depth image of the multiple plants is intercepted and processed by the following formula to obtain the depth image corresponding to the region of interest:

[0119]

[0120] in, represents the depth value in the depth image corresponding to the region of interest, represents the depth value in the depth image of multiple plants, Represents the first dimension direction coordinate of the pixel point in the plane rectangular coordinate system. Represents the second dimension coordinate of the pixel point in the plane rectangular coordinate system. represents the abscissa of the first vertex, represents the abscissa of the second vertex, represents the ordinate of the second vertex, Indicates the ordinate of the first vertex.

[0121] Step C4, using the Otsu threshold segmentation algorithm, processing the depth image corresponding to the region of interest to determine the foreground region corresponding to the region of interest.

[0122] In the above scheme, the bounding box coordinates of the region of interest can be obtained through image recognition algorithms, etc., to determine the approximate position of the target plant in the depth image.

[0123] The region of interest is represented by a rectangular box (i.e., anchor box). Based on the center position (represented by the horizontal coordinate and the vertical coordinate), the width and the height of the region of interest, the coordinates of the two diagonal vertices (i.e., the first vertex and the second vertex) of the rectangular box are calculated.

[0124] Among them, taking the center position of the region of interest as the origin direction, the first vertex angle is the upper left vertex angle relative to the center position of the region of interest as the origin direction, and the second vertex angle is the lower right vertex angle relative to the center position of the region of interest as the origin direction.

[0125] Using the coordinates of the two diagonal vertices calculated in the previous step, a sub-image is cut out from the depth image of multiple plants. This sub-image is the depth image of the area of ​​interest of the target plant.

[0126] This step traverses the pixels in the depth image and decides whether to retain the depth value of the pixel based on whether the coordinates of the pixel fall within the region of interest.

[0127] Otsu's method is a method for automatically determining the image binary segmentation threshold. It searches for a threshold based on the grayscale histogram of the image so that the intra-class variance between the segmented foreground and background is minimized (or equivalently, the inter-class variance is maximized).

[0128] In this step, the Otsu threshold segmentation algorithm is applied to the depth image of the region of interest to distinguish the foreground (target plant) from the background. By setting a threshold, the depth image can be converted into a binary mask map, where the foreground area is white (or 1) and the background area is black (or 0), which will provide the required area information for subsequent processing.

[0129] In some embodiments, step C4 comprises:

[0130] Step D1, counting the number of pixels of each gray level in the depth image corresponding to the region of interest of the target plant.

[0131] Step D2, constructing a grayscale histogram based on the grayscale levels and the corresponding pixel numbers.

[0132] Step D3, determining a first probability that each grayscale value in the grayscale histogram is less than or equal to a preset threshold to be selected, and determining a second probability that each grayscale value in the grayscale histogram is greater than the preset threshold to be selected.

[0133] Step D4, according to the grayscale histogram, counting the first grayscale values ​​less than or equal to the preset threshold value to be selected among the grayscale values, and according to the grayscale histogram, counting the second grayscale values ​​greater than the preset threshold value to be selected among the grayscale values.

[0134] Step D5, averaging the first grayscale values ​​whose grayscale values ​​are less than or equal to the preset threshold value to obtain a first average grayscale value, and averaging the second grayscale values ​​whose grayscale values ​​are greater than the preset threshold value to obtain a second average grayscale value.

[0135] Step D6, based on the first probability, the second probability, the first average gray value and the second average gray value, determine the inter-class variance corresponding to the preset candidate threshold value by the following formula:

[0136]

[0137] in, represents the between-class variance, represents the preset candidate threshold, represents the first probability, represents the second probability, represents the first average gray value, represents the second average gray value;

[0138] Step D7, determining the largest inter-class variance from the inter-class variances corresponding to the preset candidate thresholds, and selecting the preset candidate threshold corresponding to the largest inter-class variance as the target threshold.

[0139] Step D8, taking the pixel points greater than the target threshold among the pixels in the depth image corresponding to the region of interest as target pixels, and performing combination processing based on the target pixels to obtain an initial foreground region corresponding to the region of interest.

[0140] Step D9, performing Gaussian filtering on the initial foreground area corresponding to the region of interest to obtain the foreground area corresponding to the region of interest.

[0141] In the above scheme, the depth image corresponding to the region of interest (ROI) of the target plant is processed, and the number of pixels at each gray level (0 to 255, assuming an 8-bit grayscale image) is counted.

[0142] Based on the above statistical results, a grayscale histogram is constructed, in which the horizontal axis represents the grayscale level and the vertical axis represents the number of pixels corresponding to the grayscale level.

[0143] For each preset candidate threshold (from 0 to 255), calculate the second probability that the gray value is greater than the threshold The first probability that the gray value is less than or equal to the threshold .

[0144] According to the grayscale histogram, the number of pixels corresponding to the first grayscale value less than or equal to the threshold to be selected, and the number of pixels corresponding to the second grayscale value greater than the threshold to be selected, among all grayscale values, are counted.

[0145] The average of all the first grayscale values ​​less than or equal to the selected threshold is calculated to obtain the first average grayscale value. .

[0146] The second grayscale value greater than the selected threshold is averaged to obtain the second average grayscale value. .

[0147] The inter-class variance corresponding to each candidate threshold is determined by the following formula ( ):

[0148]

[0149] This formula takes into account the first and second probabilities ( and ) and the first and second average gray values ​​( and ). The larger the inter-class variance, the higher the contrast between the foreground and the background, and the better the segmentation effect.

[0150] From all candidate thresholds, select the threshold that maximizes the inter-class variance as the target threshold.

[0151] In the depth image of the region of interest, pixels greater than the target threshold are marked as target pixels, and these target pixels are combined to form the initial foreground region.

[0152] The initial foreground area is processed by Gaussian filtering to smooth the image and reduce noise, and the final foreground area is obtained. Gaussian filtering is a commonly used image smoothing technology that can effectively remove high-frequency noise in the image.

[0153] This process determines the optimal segmentation threshold by maximizing the inter-class variance, which can effectively extract the foreground area of ​​the target plant's region of interest from the depth image while maintaining the maximum contrast between the foreground and background.

[0154] In some embodiments, step 101 includes:

[0155] Step E1, obtaining point cloud data of multiple plants.

[0156] Step E2, determining the three-dimensional coordinates of each data point in the point cloud data of multiple plants, the three-dimensional coordinates of each data point including the coordinates in the first dimension direction, the coordinates in the second dimension direction and the coordinates in the third dimension direction in the three-dimensional space.

[0157] Step E3, summing up the square values ​​of the coordinates in the first dimension direction, the square values ​​of the coordinates in the second dimension direction, and the square values ​​of the coordinates in the third dimension direction corresponding to each data point to obtain the summing results corresponding to each data point.

[0158] Step E4, performing square root processing on the summation processing results corresponding to each data point to obtain the depth value corresponding to each data point.

[0159] Step E5, counting the maximum depth value among the depth values ​​corresponding to the data points, and counting the minimum depth value among the depth values ​​corresponding to the data points.

[0160] Step E6, based on the maximum depth value, the minimum depth value and the depth value corresponding to each data point, the depth value corresponding to each data point is subjected to depth normalization processing by the following formula to obtain the normalization processing result corresponding to the depth value of each data point:

[0161]

[0162] in, represents the normalized processing result, Indicates the depth value corresponding to each data point, Represents the first dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. Represents the second dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. Indicates the minimum depth value, Indicates the maximum depth value.

[0163] Step E7, determining a depth mean based on the depth value of each data point, and determining a depth standard deviation based on the depth value of each data point.

[0164] Step E8, using the depth mean, depth standard deviation, and the normalized processing results corresponding to the depth values ​​of each data point to be processed through a screening algorithm to obtain screened normalized processing results, and performing combination processing based on the screened normalized processing results to obtain depth images of multiple plants.

[0165] In the above scheme, the point cloud data is a set of three-dimensional coordinates of the plant surface obtained by some means (such as laser scanning, stereo vision, etc.).

[0166] Each data point in the point cloud data has three coordinate values ​​in three-dimensional space: the first dimension direction (usually the X-axis), the second dimension direction (usually the Y-axis), and the third dimension direction (usually the Z-axis, representing depth or height).

[0167] The depth value of the data point (i.e., the Euclidean distance from the point to the origin) is obtained by squaring the three coordinate values ​​of each data point and adding them together, then taking the square root of the result.

[0168] Find the maximum and minimum depth values ​​among all data points.

[0169] The depth value of each data point is normalized to the range of 0 to 1 using the following formula:

[0170]

[0171] By scaling all depth values ​​to a uniform range, we ensure that the depth maps of different plants have the same depth distribution. The normalized depth maps can quickly filter out high-quality images as the basis for building a depth image dataset of related plants.

[0172] The depth mean is the average of the depth values ​​of all data points, and the depth standard deviation is the degree of dispersion of these depth values ​​relative to the mean.

[0173] By using the depth mean, depth standard deviation and normalized depth value, a certain screening algorithm (such as threshold screening, cluster analysis, etc.) is used to remove outliers or noise to obtain smoother depth data.

[0174] The filtered normalized processing results are combined to form one or more images representing the depth information of multiple plants.

[0175] In some embodiments, in step E8, the depth mean, depth standard deviation, and the normalized processing results corresponding to the depth values ​​of each data point are processed by a screening algorithm to obtain a screened normalized processing result, including:

[0176] Step F1, using the depth mean, depth standard deviation, preset constant parameters, and the normalization processing results corresponding to the depth values ​​of each data point, the normalization index corresponding to the normalization processing results of the depth values ​​of each data point is determined by the following formula:

[0177]

[0178] in, represents the normalized index, represents the normalized processing result, Represents the first dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. Represents the second dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. represents the depth mean, represents the depth standard deviation, Indicates a preset constant parameter.

[0179] Step F2, from the normalization indicators corresponding to the normalization processing results of the depth values ​​of each data point, select the normalization processing results whose normalization indicators are greater than the preset indicator threshold, and use the normalization processing results whose normalization indicators are greater than the preset indicator threshold as the screened normalization processing results.

[0180] In the above scheme, the depth mean is the average of all depth values, and the depth standard deviation It measures the dispersion of these depth values ​​relative to the mean.

[0181] Preset constant parameters , is a fixed value set according to actual needs or experience, used to play a regulatory role in the calculation of normalized indicators, and is usually set to 2.

[0182] The depth mean, depth standard deviation, preset constant parameters, and the normalized processing results corresponding to the depth values ​​of each data point are used to determine the normalized index corresponding to the normalized processing results of the depth values ​​of each data point through the following formula:

[0183]

[0184] The formula comprehensively considers the normalized results of the depth values, the position information and the depth statistical characteristics, as well as the adjustment parameters, so that the normalized index can be used to measure the quality of the depth values ​​of the data points.

[0185] After calculating the normalized index of each data point, by comparing these indexes with a preset index threshold, the depth values ​​of high-quality data points that meet specific conditions (i.e., the normalized index is greater than the preset threshold) can be quickly screened out (i.e., the filtered normalized processing results).

[0186] In some embodiments, the present application discloses a method for removing outliers and separating multiple plant point clouds based on target detection based on depth information, which is applicable to the field of plant phenotype. It includes: obtaining corresponding depth maps from multiple plant point clouds, performing depth normalization on the depth maps, and quickly screening to obtain high-quality depth maps to produce a data set of relevant plant depth maps; using a target detection model based on deep learning to iteratively train on the data set, and inferring the plant point cloud depth map to obtain the anchor frame coordinates of each plant; in order to ensure that the depth map of a single plant has the same scale as the original depth map, the content in the anchor frame is intercepted according to the coordinates, and the outside of the content is filled with black; the threshold segmentation method is used for the depth map of a single plant to generate a mask map to remove outliers and the ground; the mask map is applied to the original depth map to obtain the corresponding single plant depth map after removing the outliers, and finally the point cloud is restored from the depth map to obtain the final single plant point cloud. The present application does not directly process the three-dimensional point cloud, but converts the three-dimensional problem into two dimensions to achieve the purpose of separating multiple plant point clouds. This method can also effectively remove the ground and outliers, significantly improving the work efficiency of the multi-plant segmentation task.

[0187] Specifically, (1) First, import multiple plant point cloud data. The scene contains multiple complex point cloud data:

[0188] Scene description. The scene processed is point cloud data of multiple plants, such as Figure 2 The figure shows the point cloud data of multiple soybean plants. The main task is to accurately extract the point cloud of each plant from these complex point cloud data, while removing the point cloud of the ground, surrounding environment and other impurities. This not only helps in the subsequent analysis of the growth status of each plant, but also improves the processing efficiency of the subsequent algorithm.

[0189] (2) Next, the point cloud of each plant is converted into a corresponding depth map, such as Figure 3 and Figure 4 To ensure that the depth map can be consistent between different plants, the depth map is normalized and all depth values ​​are scaled to a uniform range. This step quickly selects high-quality images, removes noise and distortion, and constructs a high-quality depth map dataset as the basic data for subsequent analysis:

[0190] Generate and filter the plant point cloud depth map. Convert the point cloud into the corresponding depth map, and then normalize the depth map, that is, scale all depth values ​​to a uniform range to ensure that the depth maps of different plants have the same depth distribution. The normalized depth map can quickly filter out high-quality images as the basis for building a relevant plant depth map dataset.

[0191] a-1. Convert point cloud to depth map: Convert point cloud to depth map. The depth map is a two-dimensional matrix in which the value of each pixel is Indicates the distance to the line scan radar. Depth value ,in are the pixel coordinates of the image.

[0192] a-2. Depth normalization: The purpose of depth normalization is to adjust the depth values ​​of the depth map to a uniform range. The normalization formula is as follows:

[0193]

[0194] in, and are the minimum and maximum depth values ​​in the depth map, respectively.

[0195] a-3. Filter depth maps: Use depth distribution statistics to filter high-quality depth maps, calculate the mean and standard deviation of the depth values, and filter according to the following conditions:

[0196]

[0197] in is the mean value of the depth map, is the standard deviation, is a constant, usually taking the value of 2.

[0198] a-4. Constructing a dataset: Use the filtered depth map to construct a dataset. The depth map grid is , is the image width, is the image height, is the number of feature channels. Anchor Box coordinates, in the format of ,in is the center coordinate of the anchor box, and are the width and height of the anchor box respectively.

[0199] (3) After obtaining the depth map dataset, the object detection model based on YOLOv8 is used for training. YOLOv8 accurately identifies and infers the position of each plant in the point cloud. The model can accurately describe the range of each plant in the depth map by calculating the anchor box of each plant, such as Figure 5 As shown:

[0200] Object detection and anchor box extraction based on the YOLOv8 model. In order to automatically separate and detect each plant point cloud, the object detection model based on YOLOv8 is used to iteratively train on the depth map dataset generated in the previous step. By continuously adjusting the model parameters and optimizing the training process, the YOLOv8 model can accurately infer the anchor box of each plant in the plant point cloud depth map.

[0201] Among them, the target detection model based on YOLOv8 is used to perform iterative training on the dataset, and the plant point cloud depth map is inferred to obtain the anchor frame coordinates of each plant, such as Figure 6 shown.

[0202] b-1. Data enhancement: During the training process, data enhancement methods are used to improve the generalization ability of the model, including: random scaling, scaling the image to generate samples of various scales; random cropping, randomly cropping a part from the image to increase the diversity of samples; random rotation, rotating the image at random angles.

[0203] b-2. Loss function definition: Loss function needs to be defined during training , the loss function of YOLOv8 consists of the following three parts:

[0204] Classification Loss : Used to evaluate the accuracy of model classification, cross entropy loss is commonly used:

[0205]

[0206] in, is the true label, is the probability predicted by the model.

[0207] Regression Loss : Used to evaluate the gap between the predicted anchor box and the true anchor box, using loss:

[0208]

[0209] in, is the real anchor box coordinate, are the anchor box coordinates predicted by the model.

[0210] Confidence loss : Evaluate the accuracy of the model's prediction of the object existence for each anchor box, measured in binary cross entropy:

[0211]

[0212] in, Indicates The real existence of anchor boxes, is the predicted value.

[0213] b-3. Training process: In each iteration, the model parameters are updated through back propagation:

[0214]

[0215] in, Represents the parameters of the target detection model in the current iteration process, Represents the parameters of the target detection model in the previous iteration of the current iteration. is the learning rate, is the gradient of the loss function.

[0216] b-4. Reasoning plant point cloud depth map: After training is completed, the new plant point cloud depth map can be reasoned. During the reasoning process, the model accepts the depth map as input and outputs the anchor box of each detected object. and confidence.

[0217] b-5. Remove redundant anchor boxes: After inference is completed, the output of the model needs to be post-processed and the non-maximum suppression (NMS) method is used to remove redundant anchor boxes. The specific steps are as follows:

[0218] 1) Sort by their confidence. Each anchor box has a corresponding confidence score and position coordinates. Sort all anchor boxes in descending order by their confidence scores. Anchor boxes with high scores are more likely to be correct detection results. Select the box with the highest score from the sorted anchor boxes and use it as one of the final detection results.

[0219] 2) Calculate the overlap. For the selected anchor box, calculate the overlap with all other anchor boxes, and use IoU (Intersection over Union) to evaluate the degree of overlap. The calculation formula of IoU is:

[0220]

[0221] 3) Perform threshold comparison. Set an IoU threshold (e.g. 0.5). If the IoU value between the currently selected anchor box and other boxes exceeds the threshold, these overlapping boxes are considered redundant and deleted.

[0222] 4) Repeat the steps. Repeat the selection of the next highest scoring anchor box and calculate the IoU with the remaining boxes again until all boxes have been processed.

[0223] (4) Then, using the anchor frame coordinates, the image content within the anchor frame is cut out, and the area outside the anchor frame is filled with black pixels to ensure that the spatial information of each plant is fully preserved and the irrelevant background area is removed. After that, the depth map is binarized using the Otsu threshold segmentation method, and combined with Gaussian filtering to remove image noise, smooth the image edges, and generate an accurate binary mask image:

[0224] Depth map clipping and mask generation. To ensure that the depth map of each plant is consistent with the scale of the original depth map, the obtained anchor frame coordinates are used to intercept the image content within the anchor frame, and the part outside the anchor frame is filled with black pixels. The spatial information of each plant is retained, and the irrelevant background area is removed. Then, the Otsu threshold segmentation method is used to automatically find the optimal threshold to segment the foreground and background of the intercepted plant depth map, and Gaussian filtering is combined to remove noise and smooth the image, and finally a binary mask map of the plant is generated. The white area in the mask represents the plant, and the black area represents the background.

[0225] At this stage, it is necessary to ensure that the scale of the depth map of each plant is consistent with the original depth map. To this end, the anchor frame coordinates are used for content capture, and the depth map of each plant is threshold segmented and Gaussian filtered to generate a binary mask map, such as Figure 8 As shown, the ground and outliers can be effectively removed. The specific steps are as follows:

[0226] c-1. Plant capture: anchor frame coordinates obtained using YOLOv8 model inference , the area of ​​each plant can be intercepted from the original depth map. The coordinates of the upper left corner and lower right corner of the anchor frame are calculated using the center coordinates of the anchor frame: The intercepted formula is:

[0227]

[0228] in represents the depth value in the original depth map, Represents the captured depth map.

[0229] c-2. Fill black pixels: For pixel values ​​that are not in the anchor box, set them to black (0). This ensures that the area outside the anchor box will not affect the result during subsequent processing.

[0230] c-3. Otsu threshold segmentation: Otsu threshold segmentation is an adaptive threshold selection method, which aims to divide the image into foreground (plants) and background (black pixels). The specific steps are as follows:

[0231] 1) Calculate the grayscale histogram of the image and count the number of pixels at each grayscale level.

[0232] 2) For each possible threshold , calculate the probability of two categories: foreground class (grey value greater than ) ; Background class (grayscale value is less than or equal to )

[0233] 3) Calculate the between-class variance :

[0234]

[0235] in, and are the average grayscale values ​​below and above the threshold, respectively. Traverse all possible thresholds , find The largest , which is the optimal threshold.

[0236] 4) Finally, the image is segmented into foreground and background using the selected threshold. If the gray value of the pixel is greater than the optimal threshold , it is considered as foreground. Otherwise, it is considered as background.

[0237] c-4. Gaussian filtering: After threshold segmentation, in order to reduce noise, Gaussian filtering is applied to smooth the depth map. The formula for Gaussian filtering is:

[0238]

[0239] in, are the coordinates of the nucleus, is the standard deviation, controlling the degree of blur.

[0240] Gaussian Kernel Smoothing is achieved by taking a weighted average of the values ​​of each pixel and its surrounding pixels. Gaussian kernel By giving greater weights to pixels near the center pixel and smaller weights to pixels farther away from the center pixel, a smooth image is achieved and noise is removed.

[0241] c-5. Generate a mask map: After Otsu threshold segmentation and Gaussian filtering, the resulting mask map will show the plant area, with a mask value of 1 and a background of 0. This will provide the required regional information for subsequent processing.

[0242] (5) Finally, the generated mask image is used as the mask of the original point cloud depth image, and the corresponding point cloud coordinate information and depth in the original depth image are extracted according to the coordinates of the non-black pixels in the mask image, thereby reconstructing the 3D point cloud of the plant. The final point cloud model of a single plant is:

[0243] Extract the point cloud of the target plant. Use the mask image of each plant as the mask of the depth map of the original point cloud. Since the mask image of a single plant is consistent with the size of the original point cloud depth map, the corresponding coordinates in the original depth map are extracted according to the coordinates of the non-black pixels in the mask image to obtain the depth map corresponding to the mask image of the plant. Finally, the extracted point cloud only contains the target plant, removing the redundant background (ground) and noise point clouds, and obtaining a point cloud model of a single plant.

[0244] The mask image of each plant is used as a mask for the depth image of the original point cloud. Since the mask image of a single plant is consistent with the depth image of the original point cloud, the corresponding coordinates in the original depth image are extracted according to the coordinates of the non-black pixels in the mask image to obtain the depth image corresponding to the mask image of the plant. Finally, the point cloud is restored from the depth image to obtain the final point cloud of the single plant. The process of this stage is as follows: Figure 7 , Figure 8 , Fig. 9 and Fig.10 As shown:

[0245] d-1. Use the mask image as a mask for the depth map: Since the mask image of each plant is consistent with the original depth map in size, the mask image can be used as a mask for the original depth map to extract the depth value of the target area. Define the mask image is a binary image, where the pixel value of the foreground plant is 1 and the pixel value of the background is 0. The image after defining the depth map mask for:

[0246]

[0247] in is the depth value of the original depth map, is the pixel value of the mask image. If a point is in the foreground area of ​​the mask, its depth value is retained; otherwise, its depth value is set to 0.

[0248] d-2. Recover point cloud: In the depth map, each pixel Two-dimensional (2D) plane coordinates corresponding to the point cloud , and the depth value The height of the corresponding point cloud, that is Since this is a depth map generated from a known point cloud, the resolution and pixel coordinates of the depth map are the same as those in the point cloud. , The coordinates remain consistent. For each non-zero pixel in the depth map , the corresponding three-dimensional (3D) point cloud coordinates are restored as:

[0249]

[0250] in, is the depth value of the target plant area in the depth map. The coordinates directly correspond to the coordinates of each point in the original point cloud .

[0251] This application also uses field rice as an example point cloud, and based on the above processing process, a schematic diagram of the processing flow for obtaining the point cloud of a single rice plant is shown, such as Fig.11 shown.

[0252] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0253] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0254] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a device for extracting point cloud data of a target plant from multiple plants.

[0255] refer to Fig.12 , the device for extracting point cloud data of a target plant from multiple plants comprises:

[0256] The depth image acquisition module 1201 is configured to acquire depth images of multiple plants;

[0257] The region of interest generation module 1202 is configured to identify the position of each target plant in the depth image of multiple plants, obtain a position identification result of the target plant, and generate a region of interest of the target plant based on the position identification result of the target plant and a preset region of interest size;

[0258] The target plant processing module 1203 is configured to perform the following operations for the region of interest of each target plant: use a threshold segmentation algorithm to process the region of interest of the target plant to determine the foreground area corresponding to the region of interest; extract and process the depth images of multiple plants based on the foreground area corresponding to the region of interest to obtain the depth image corresponding to the foreground area; perform point cloud recovery processing on the depth image corresponding to the foreground area to obtain point cloud data of the target plant.

[0259] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0260] The device of the above embodiment is used to implement the method of extracting point cloud data of a target plant from multiple plants in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0261] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting target plant point cloud data from multiple plants as described in any of the above embodiments is implemented.

[0262] Fig.13 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1301, a memory 1302, an input / output interface 1303, a communication interface 1304, and a bus 1305. The processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304 are connected to each other in communication within the device through the bus 1305.

[0263] The processor 1301 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0264] The memory 1302 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1302 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1302 and are called and executed by the processor 1301.

[0265] The input / output interface 1303 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0266] The communication interface 1304 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0267] The bus 305 comprises a path for transmitting information between various components of the device (eg, the processor 1301 , the memory 1302 , the input / output interface 1303 , and the communication interface 1304 ).

[0268] It should be noted that, although the above device only shows the processor 1301, the memory 1302, the input / output interface 1303, the communication interface 1304 and the bus 1305, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0269] The electronic device of the above embodiment is used to implement the method of extracting point cloud data of a target plant from multiple plants in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0270] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method of extracting target plant point cloud data from multiple plants as described in any of the above embodiments.

[0271] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0272] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for extracting point cloud data of a target plant from multiple plants as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0273] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0274] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device may be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0275] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0276] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for extracting point cloud data of a target plant from multiple plants, characterized in that: include: Acquire depth images of multiple plants; Identify and process the position of each target plant in the depth image of multiple plants to obtain a position identification result of the target plant, and generate a region of interest of the target plant based on the position identification result of the target plant and a preset region of interest size; For each target plant's region of interest, do the following: The region of interest of the target plant is processed using a threshold segmentation algorithm to determine the foreground region corresponding to the region of interest; According to the foreground area corresponding to the area of ​​interest, the depth images of the multiple plants are extracted and processed to obtain the depth image corresponding to the foreground area; Perform point cloud recovery processing on the depth image corresponding to the foreground area to obtain point cloud data of the target plant; The position recognition result of the target plant and the region of interest of the target plant are obtained by using a trained target detection model. The target detection model is trained using a total loss function. The process of obtaining the total loss function includes: Obtain the true labels and predicted recognition results for training; The classification loss function is determined by the following formula based on the actual labels used for training and the predicted recognition results: in, represents the classification loss function, represents the true label used for training, Represents the predicted recognition result, represents the number of depth images of the target plant used for training, represents the order of the depth images of the target plants used for training; Obtain the real coordinates of the region of interest and the predicted coordinates of the region of interest for training; The regression loss function is determined by the following formula based on the actual coordinates of the region of interest used for training and the predicted coordinates of the region of interest: in, represents the regression loss function, represents the real region of interest coordinates used for training, represents the predicted coordinates of the region of interest, represents the number of coordinates of the region of interest, Indicates the order of the coordinates of the region of interest, represents smooth L1 loss; Get the first The real existence of the region of interest, and the The predicted existence of regions of interest; According to the training The real existence of the region of interest, and the The predicted existence of the region of interest is determined by the following formula to determine the confidence loss function: in, represents the confidence loss function, represents the first The real existence of the region of interest, Indicates The predicted existence of regions of interest, Indicates the number of regions of interest; Based on the classification loss function, regression loss function and confidence loss function, the total loss function is determined by the following formula: in, represents the total loss function, represents the classification loss function, represents the regression loss function, represents the confidence loss function.

2. The method according to claim 1, characterized in that The target detection model is trained by updating model parameters through back propagation. The process of obtaining the back propagation model parameters includes: Obtain the parameters of the target detection model of the previous iteration process of the current iteration process, the preset learning rate and the preset gradient of the total loss function; Based on the parameters of the target detection model of the previous iteration process of the current iteration process, the preset learning rate and the preset gradient of the total loss function, the parameters of the target detection model of the current iteration process are determined by the following formula, and the parameters of the target detection model of the current iteration process are used as the back propagation update model parameters: in, Represents the parameters of the target detection model in the current iteration process, Represents the parameters of the target detection model in the previous iteration of the current iteration. represents the preset learning rate, Represents the preset gradient of the total loss function.

3. The method according to claim 1, characterized in that: The method of processing the region of interest of the target plant by using a threshold segmentation algorithm to determine a foreground region corresponding to the region of interest includes: Obtain the coordinates of the region of interest of the target plant; Based on the coordinates of the region of interest of the target plant, the coordinates of the first vertex angle and the coordinates of the second vertex angle of the region of interest of the target plant are determined by the following formula; in, represents the abscissa of the first vertex, represents the ordinate of the first vertex, represents the abscissa of the second vertex, represents the ordinate of the second vertex, The horizontal coordinate represents the center position of the region of interest. The ordinate represents the center position of the region of interest, represents the width of the center position of the region of interest, Indicates the height of the center position of the region of interest; Using the coordinates of the first vertex and the second vertex of the region of interest of the target plant, the depth image of multiple plants is intercepted and processed by the following formula to obtain the depth image corresponding to the region of interest: in, represents the depth value in the depth image corresponding to the region of interest, represents the depth value in the depth image of multiple plants, Represents the first dimension direction coordinate of the pixel point in the plane rectangular coordinate system. Represents the second dimension coordinate of the pixel point in the plane rectangular coordinate system. represents the abscissa of the first vertex, represents the abscissa of the second vertex, represents the ordinate of the second vertex, represents the ordinate of the first vertex; The depth image corresponding to the region of interest is processed using the Otsu threshold segmentation algorithm to determine the foreground area corresponding to the region of interest.

4. The method according to claim 3, characterized in that The method of processing the depth image corresponding to the region of interest by using the Otsu threshold segmentation algorithm to determine the foreground region corresponding to the region of interest includes: Count the number of pixels at each gray level in the depth image corresponding to the region of interest of the target plant; Construct a grayscale histogram based on gray levels and corresponding pixel numbers; Determine a first probability that each grayscale value in the grayscale histogram is less than or equal to a preset threshold value to be selected, and determine a second probability that each grayscale value in the grayscale histogram is greater than the preset threshold value to be selected; According to the grayscale histogram, counting the first grayscale values ​​less than or equal to the preset threshold value to be selected among the grayscale values, and according to the grayscale histogram, counting the second grayscale values ​​greater than the preset threshold value to be selected among the grayscale values; The first grayscale values ​​whose grayscale values ​​are less than or equal to the preset threshold value to be selected are averaged to obtain a first average grayscale value, and the second grayscale values ​​whose grayscale values ​​are greater than the preset threshold value to be selected are averaged to obtain a second average grayscale value; Based on the first probability, the second probability, the first average grayscale value, and the second average grayscale value, the inter-class variance corresponding to the preset candidate threshold is determined by the following formula: in, represents the between-class variance, represents the preset candidate threshold, represents the first probability, represents the second probability, represents the first average gray value, represents the second average gray value; Determine the maximum inter-class variance from the inter-class variances corresponding to the preset candidate thresholds, and select the preset candidate threshold corresponding to the maximum inter-class variance as the target threshold; The pixel points greater than the target threshold in each pixel point in the depth image corresponding to the region of interest are taken as target pixel points, and the initial foreground area corresponding to the region of interest is obtained by combining and processing the target pixel points; The initial foreground area corresponding to the region of interest is subjected to Gaussian filtering to obtain the foreground area corresponding to the region of interest.

5. The method according to claim 1, characterized in that The acquiring of depth images of multiple plants comprises: Obtain point cloud data of multiple plants; Determine the three-dimensional coordinates of each data point in the point cloud data of the multiple plants, wherein the three-dimensional coordinates of each data point include coordinates in a first dimension direction, a second dimension direction, and a third dimension direction in the three-dimensional space; The square value of the coordinate in the first dimension direction, the square value of the coordinate in the second dimension direction, and the square value of the coordinate in the third dimension direction corresponding to each data point are summed up to obtain the summation result corresponding to each data point; Perform square root processing on the summation results corresponding to each data point to obtain the depth value corresponding to each data point; Count the maximum depth value among the depth values ​​corresponding to each data point, and count the minimum depth value among the depth values ​​corresponding to each data point; Based on the maximum depth value, the minimum depth value and the depth value corresponding to each data point, the depth value corresponding to each data point is depth normalized using the following formula to obtain the normalized processing result corresponding to the depth value of each data point: in, represents the normalized processing result, Indicates the depth value corresponding to each data point, Represents the first dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. Represents the second dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. Indicates the minimum depth value, Indicates the maximum depth value; Determine a depth mean based on the depth value of each data point, and determine a depth standard deviation based on the depth value of each data point; The depth mean, depth standard deviation, and normalized processing results corresponding to the depth values ​​of each data point are processed through a screening algorithm to obtain screened normalized processing results, and combined processing is performed based on the screened normalized processing results to obtain depth images of multiple plants.

6. The method according to claim 5, characterized in that The depth mean, depth standard deviation, and the normalized processing results corresponding to the depth values ​​of each data point are processed by a screening algorithm to obtain a screened normalized processing result, including: The depth mean, depth standard deviation, preset constant parameters, and the normalized processing results corresponding to the depth values ​​of each data point are used to determine the normalized index corresponding to the normalized processing results of the depth values ​​of each data point through the following formula: in, represents the normalized index, represents the normalized processing result, Represents the first dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. Represents the second dimension coordinate of the pixel point corresponding to the data point in the plane rectangular coordinate system. represents the depth mean, represents the depth standard deviation, Indicates preset constant parameters; From the normalization indicators corresponding to the normalization processing results of the depth values ​​of each data point, the normalization processing results whose normalization indicators are greater than the preset indicator threshold are selected, and the normalization processing results whose normalization indicators are greater than the preset indicator threshold are used as the screened normalization processing results.

7. A device for extracting point cloud data of a target plant from multiple plants, characterized in that: include: A depth image acquisition module is configured to acquire depth images of multiple plants; The region of interest generation module is configured to identify and process the position of each target plant in the depth image of multiple plants, obtain the position identification result of the target plant, and generate the region of interest of the target plant based on the position identification result of the target plant and the preset region of interest size; the position identification result of the target plant and the region of interest of the target plant are obtained by using a trained target detection model, and the target detection model is trained using a total loss function. The process of obtaining the total loss function includes: obtaining the real label and the predicted identification result for training; determining the classification loss function according to the real label and the predicted identification result for training by the following formula: in, represents the classification loss function, represents the true label used for training, Represents the predicted recognition result, represents the number of depth images of the target plant used for training, represents the order of the depth images of the target plants used for training; Obtain the real coordinates of the region of interest and the predicted coordinates of the region of interest for training; The regression loss function is determined by the following formula based on the actual coordinates of the region of interest used for training and the predicted coordinates of the region of interest: in, represents the regression loss function, represents the real region of interest coordinates used for training, represents the predicted coordinates of the region of interest, represents the number of coordinates of the region of interest, Indicates the order of the coordinates of the region of interest, represents smooth L1 loss; Get the first The real existence of the region of interest, and the The predicted existence of regions of interest; According to the training The real existence of the region of interest, and the The predicted existence of the region of interest is determined by the following formula to determine the confidence loss function: in, represents the confidence loss function, represents the first The real existence of the region of interest, Indicates The predicted existence of regions of interest, Indicates the number of regions of interest; Based on the classification loss function, regression loss function and confidence loss function, the total loss function is determined by the following formula: in, represents the total loss function, represents the classification loss function, represents the regression loss function, represents the confidence loss function; The target plant processing module is configured to perform the following operations for the region of interest of each target plant: use a threshold segmentation algorithm to process the region of interest of the target plant to determine the foreground area corresponding to the region of interest; extract and process the depth images of multiple plants based on the foreground area corresponding to the region of interest to obtain the depth image corresponding to the foreground area; perform point cloud recovery processing on the depth image corresponding to the foreground area to obtain point cloud data of the target plant.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images