Deep learning-based root hair segmentation method, system and equipment

Abstract

The invention discloses a deep learning-based root hair segmentation method, system and device. The method comprises the steps of identifying root hair trunk information in an image; carrying out image preprocessing on the plurality of independent sub-region patches; the patch is input into a CAE module, and the appearance characteristics of the root hair are learned; inputting the image processed by the CAE module into a root hair instance segmentation model to detect and segment the root hair; a preset depth estimation module is utilized to predict the depth of root hairs, and an accurate depth map is generated by analyzing depth information in an image, so that the root hairs which are overlapped before and after are accurately distinguished and processed during image segmentation; keeping a detection segmentation result in a set range in the bounding box; and recording a root hair segmentation result and the density of the root hair in the region. According to the method, the depth of the overlapped root hair is extracted by using a depth estimation method, and the obtained depth information is added into an MP-Former encoder as a feature, so that the model fully learns the feature of the root hair during segmentation, and the accuracy of the segmentation of the overlapped root hair is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and specifically relates to a root hair segmentation method, system and device based on deep learning. Background Art

[0002] Timely and accurate acquisition of the number and density of plant root hairs is crucial to understanding the phenotypic characteristics of plants. However, since plant roots are usually buried in opaque soil, there are many challenges in directly observing and studying their growth status, especially without destroying the roots and their surrounding environment. Although traditional evaluation methods such as manual counting, X-ray scanning, and magnetic resonance imaging are feasible, these methods not only consume a lot of resources, but are also still insufficient in providing detailed root structure information.

[0003] In recent years, with the development of deep learning technology, especially the successful application of instance segmentation technology in multiple fields, new possibilities have been provided for solving the above problems. Nevertheless, the differences between different application scenarios make it difficult for a single technical solution to be universally applicable. In view of this, the present invention aims to combine the knowledge of instance segmentation technology, image enhancement and denoising processing to develop a new method that can effectively perform root hair segmentation in a variety of complex environments, and will elaborate on the design and implementation of this method in detail. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a root hair segmentation method, system and device based on deep learning, which improves the accuracy of segmentation through image denoising, image enhancement, depth information, etc., and uses the segmentation results to calculate the root hair density per unit area.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a root hair segmentation method based on deep learning, comprising the following steps:

[0007] Processing the input image, identifying the root hair trunk information in the image, taking the geometric center of the identified root hair trunk as the center point, and dividing the entire image into a plurality of independent sub-region patches based on the center point;

[0008] Perform image preprocessing on several independent sub-region patches;

[0009] The preprocessed image is input into the root hair instance segmentation model to detect and segment the root hairs; the root hair instance segmentation model adopts an improved MP-Former model, and the improved MP-Former model includes a CAE module and a depth estimation module; when detecting and segmenting the root hairs, the preprocessed patch is first input into the preset CAE module to learn the appearance features of the root hairs. In the CAE module, the input patch is segmented into visible patches and mask patches, the encoder is used to process the visible patches to generate potential representations, the regressor is used to predict the potential representations of the mask patches, and finally the decoder is used to reconstruct the occluded patches; then the Depth-Anything model is applied to extract the depth information of all the root hairs to identify the front and back relationship of the root hairs; the improved MP-Former model focuses on the depth information in the current frame to improve the segmentation success rate of staggered root hairs; the depth estimation module uses ConvNeXt-T as a shared backbone network to extract multi-scale features of the root hair image, and the root hair image is divided into single view and multi-view branches to generate depth information of the root hair image respectively;

[0010] After the improved MP-Former model outputs the segmentation results, a low threshold L and a high threshold H are set, and the detection results with confidence higher than the high threshold H are retained. For the detection results with confidence between the high threshold H and the low threshold L, only the detection segmentation results within the set range in the bounding box are retained;

[0011] Record the root hair segmentation results. After completing the segmentation of the entire area, record the number of root hairs in each patch and calculate the density of root hairs in the area.

[0012] As a preferred technical solution, the input image is processed to identify the root hair trunk information in the image, and the geometric center of the identified root hair trunk is used as the center point. Based on the center point, the entire image is divided into several independent sub-region patches, specifically:

[0013] Each image is input into the preset SAM model to generate image embedding, the hint information is input into the hint encoder, and the image embedding and hint information are combined through the lightweight mask decoder to predict the image segmentation mask;

[0014] Set the number of image segmentations in the vertical direction, the overlapping ratio of the sliding window, read the annotation information of the root hair trunk to generate a mask, and find the outline of the mask. Calculate the height of each segmented area based on the image height and the number of segmentations, calculate the center point of each segmented area based on the height of each segmented area, calculate the size of the window according to the predefined aspect ratio, and generate the coordinates of the window according to the outline points. Finally, crop the window image and record the information of each patch.

[0015] As a preferred technical solution, the image preprocessing of several independent sub-region patches is specifically as follows:

[0016] Convert the currently input patch into a grayscale image, use the Canny edge detection algorithm to identify the edges in the grayscale image, obtain an edge intensity map, and then calculate the average intensity value of the edge intensity map. Compare the average intensity value with the set threshold. If it is less than the threshold, the current patch needs to be sharpened.

[0017] The grayscale image is judged, the signal-to-noise ratio is calculated based on the mean and standard deviation of the image, and the signal-to-noise ratio is compared with the set threshold. If it is less than the threshold, the current patch is denoised to obtain the final preprocessed image;

[0018] The preprocessed images were expanded using the “copy and paste” method, and the bounding box area of ​​the root hairs in a certain original image in the dataset was copied to another random original image to generate new samples.

[0019] As a preferred technical solution, the pre-processed patch is first input into a preset CAE module to learn the appearance characteristics of the root hairs, specifically:

[0020] Divide the image into several patches and set the size of the patches;

[0021] According to the random block occlusion strategy in the context information, the occlusion rate is set to obtain the visible patch mask patch;

[0022] The encoder first receives the input visible image patches, embeds each image patch into a patch embedding through linear projection, and adds the position embedding to generate an initial embedding vector; then, the initial embedding vector is input into a series of Transformer blocks based on the self-attention mechanism to gradually generate a latent representation;

[0023] The regressor uses the representation of the visible patch and the position information of the mask patch to process it through a series of cross-attention-based Transformer blocks to generate a potential representation of the mask patch. Through the prediction alignment mechanism, the mask patch representation Z output by the regressor is ensured to be m Aligned in the same representation space as the representations computed directly by the encoder;

[0024] The decoder receives the mask patch representation and the position information of the mask patch from the regressor, gradually extracts and refines the root hair features through multiple Transformer blocks, and then predicts the final mask image patch through the linear layer.

[0025] As a preferred technical solution, it also includes using a preset depth estimation module to predict the depth of the root hairs, specifically:

[0026] In the depth hypothesis sampling stage, a set number of depth hypotheses are uniformly sampled in the logarithmic space, and then a 4D cost volume is constructed, and the number of channels is reduced to 1 through a 3D convolutional layer;

[0027] Subsequently, the cost volume is regularized using a stacked 2D hourglass network, the depth feature map is obtained by combining the backbone features, and the depth probability volume and confidence map are generated through the Softmax operation; the depth estimation results of the single view and multi-view are fused through the soft-weighted sum method to calculate the final single view depth;

[0028] Finally, the CRF technique is used to refine the depth map, and the model performance is improved through end-to-end training and parameter adjustment.

[0029] As a preferred technical solution, the depth estimation module adopts a single-target depth estimation model depthanything, generates an accurate depth map by analyzing the depth information in the image, and inputs it into MP-Former as feature information, so that when performing image segmentation, the overlapping root hairs can be accurately distinguished and processed.

[0030] As a preferred technical solution, the root hair segmentation result is recorded, and after the segmentation of the entire area is completed, the number of root hairs in each patch is recorded, and the density of root hairs in the area is calculated, specifically:

[0031] After the root hair segmentation of each patch is completed, the previously saved patch information is read, and each patch is pieced back into the original image. During the restoration process, the context information is combined to ensure the consistency of the same instance in the overlapping area. Finally, the total number of root hairs n is recorded, and the density of root hairs per unit area is calculated based on the area of ​​the original image and the total number of root hairs.

[0032] In a second aspect, the present invention provides a root hair segmentation system based on deep learning, which is applied to the root hair segmentation method based on deep learning, including an image segmentation module, an image preprocessing module, a root hair detection and segmentation module, a detection result selection module, and a root hair density calculation module;

[0033] The image segmentation module is used to process the input image, identify the root hair trunk information in the image, use the identified geometric center of the root hair trunk as the center point, and divide the entire image into a plurality of independent sub-region patches based on the center point;

[0034] The image preprocessing module is used to perform image preprocessing on a plurality of independent sub-region patches;

[0035] The root hair detection and segmentation module is used to input the preprocessed image into the root hair instance segmentation model to detect and segment the root hairs; the root hair instance segmentation model adopts the improved MP-Former model, and the improved MP-Former model includes a CAE module and a depth estimation module; when detecting and segmenting the root hairs, the preprocessed patch is first input into the preset CAE module to learn the appearance characteristics of the root hairs. In the CAE module, the input patch is segmented into visible patches and mask patches, the encoder is used to process the visible patches to generate potential representations, the regressor is used to predict the potential representations of the mask patches, and finally the decoder is used to reconstruct the occluded patches; then the Depth-Anything model is applied to extract the depth information of all the root hairs to identify the front and back relationship of the root hairs; the improved MP-Former model focuses on the depth information in the current frame to improve the segmentation success rate of staggered root hairs; the depth estimation module uses ConvNeXt-T as a shared backbone network to extract multi-scale features of the root hair image, and the root hair image is divided into single view and multi-view branches to generate depth information of the root hair image respectively;

[0036] The detection result selection and screening module is used to set a low threshold L and a high threshold H after the improved MP-Former model outputs the segmentation result, retain the detection results with a confidence level higher than the high threshold H, and for the detection results with a confidence level between the high threshold H and the low threshold L, only retain the detection segmentation results within the set range in the bounding box;

[0037] The root hair density calculation module is used to record the root hair segmentation results. After completing the segmentation of the entire area, the number of root hairs in each patch is recorded and the density of root hairs in the area is calculated.

[0038] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0039] at least one processor; and,

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the deep learning-based root hair segmentation method.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the root hair segmentation method based on deep learning.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The present invention adopts an image preprocessing module, and uses an image diagnosis module to judge the current patch, sharpen the current patch that needs to be sharpened, and remove noise from the current patch that needs to be denoised. After the image is enhanced, it is passed to the next module. This diversified processing target makes the system more flexible and adaptable to a wider range, and can better meet the image processing needs in different scenarios.

[0045] 2. The present invention uses the CAE module to pre-train the unlabeled images. The CAE module predicts the representation of the masked parts and ensures that these representations are in the same encoding representation space, making the feature representation richer and more efficient. This is especially important for segmentation tasks such as root hairs, as the model needs to capture the details and texture features of the root hairs.

[0046] 3. The present invention proposes to add depth information to MP-Former. Depth information can provide more 3D structural details, making the segmentation results more accurate and refined. Secondly, by combining depth information, the model can better distinguish root hairs at different depth levels, thereby improving the overall accuracy of segmentation. In addition, depth information can also reduce errors caused by illumination changes and background noise, thereby enhancing the robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 is a flow chart of a root hair instance segmentation method based on deep learning according to an embodiment of the present invention;

[0049] Figure 2 is a flowchart of an algorithm of an image preprocessing module according to an embodiment of the present invention;

[0050] Figure 3 is an unsegmented root hair image according to an embodiment of the present invention;

[0051] Figure 4 is a root hair image segmented after being processed by the SAM model in an embodiment of the present invention;

[0052] Figure 5 is an image segment obtained by sliding window processing according to an embodiment of the present invention;

[0053] Figure 6 is a diagram of an improved MP-Former model according to an embodiment of the present invention;

[0054] Figure 7 is a result diagram of the segmentation of the root hair instance according to the embodiment of the present invention;

[0055] Figure 8 is a block diagram of a root hair instance segmentation system based on deep learning according to an embodiment of the present invention;

[0056] Fig. 9 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0058] See also Figure 1 In one embodiment of the present application, a root hair instance segmentation method based on deep learning is provided, comprising the following steps:

[0059] (1) Please refer to Figure 3 , Figure 4 , Figure 5 The input image is processed through the preset SAM model, the backbone information in the image is automatically identified, and the geometric center of the identified backbone is used as the center point. Based on the center point, the entire image is divided into several independent sub-regions (patches).

[0060] It can be understood that this division method can effectively enhance the ability to extract local features of the image and ensure accurate capture of details in complex scenes; further, step (1) is specifically as follows:

[0061] (1-1) Each image is input into the Segment Anything Model (SAM), and the model's powerful image encoder is used to generate image embedding. Then, the prompt encoder is used to process the input prompts (such as foreground points, background points, boxes, etc.), and the image segmentation mask is predicted by combining the image embedding and prompt information through a lightweight mask decoder. Specifically, for each image, the model will segment the trunk part, calculate the center point position of the trunk, record it as point A, and then write the trunk information into a json file.

[0062] (1-2) Set the number of image segmentations in the vertical direction to num_split = 8, the overlap ratio of the sliding window to 0.5, read the annotation information of the backbone to generate a mask, and find the outline of the mask. Calculate the height of each segmented area based on the image height and the number of segmentations. The specific formula is as follows:

[0063]

[0064] Then calculate the center point of each segmented area. The specific formula is as follows:

[0065]

[0066] The size of the window is calculated according to the predefined aspect ratio, and the coordinates of the window are generated according to the contour points. Finally, the window image is cropped and the information of each patch is recorded.

[0067] (2) Please refer to Figure 2 , input each patch into the image preprocessing module, which includes: image diagnosis module, sharpening enhancement module, noise suppression module and data expansion module, specifically:

[0068] (2-1), the image diagnosis module first performs the current input patchf k Convert it into a grayscale image, use the Canny edge detection algorithm to identify the edges in the image, obtain the edge intensity map, and then calculate the average intensity value of the edge intensity map and compare it with the set threshold. Here, the threshold is set to 50; if it is less than the threshold, the current patch needs to be k Perform image sharpening.

[0069] (2-2) The image diagnosis module judges the grayscale image by first calculating the mean μ and standard deviation σ of the image, and then calculating the signal-to-noise ratio. The specific formula is as follows:

[0070]

[0071] Compare it with the set threshold, here the threshold is set to 15; if it is less than the threshold, the current patchf k Perform image denoising to obtain the final preprocessed image α.

[0072] (2-3) Use the “copy and paste” method to expand the image α, and copy the bounding box area where the root hairs are located in a certain original image in the dataset to another random original image to generate a new sample.

[0073] (3) Please refer to Figure 6, the preprocessed image is input into the root hair instance segmentation model to detect and segment the root hair; the root hair instance segmentation model adopts an improved MP-Former model, and the improved MP-Former model includes a CAE module and a depth estimation module. When detecting and segmenting the root hair, the preprocessed patch is first input into the preset CAE module to learn the appearance characteristics of the root hair; the image processed by the CAE module is input into the root hair instance segmentation model to detect and segment the root hair; in the CAE module, the input patch is divided into visible patches and mask patches, the encoder is used to process the visible patch to generate a potential representation, the potential representation of the mask patch is predicted by the regressor, and finally the decoder is used to reconstruct the occluded patch; the root hair instance segmentation model uses an improved MP-Former, and in view of the problem that the root hair is interlaced and the recognition rate is low and a large number of unlabeled images are not fully utilized, this embodiment adopts a convolutional autoencoder (CAE) model, which is pre-trained by unlabeled images to further extract the root hair features; in addition, this embodiment uses the Depth-Anything model to extract the depth information of all root hairs to accurately identify the front and back relationship of the root hairs. Through this method, we can capture the spatial relationship between root hairs in more detail and improve the model's ability to parse complex root hair structures, thereby improving overall recognition accuracy and robustness. The improved MP-Former focuses on the depth information in the current frame to improve the segmentation success rate of interlaced root hairs.

[0074] The depth processing module is used to predict the depth of the root hair using a preset depth estimation module, and uses ConvNeXt-T as a shared backbone network to extract features of four scales, which are divided into single view and multi-view branches to generate initial depth estimates respectively. In the depth hypothesis sampling stage, a certain number of depth hypotheses (such as 128) are uniformly sampled in the logarithmic space, and then a 4D cost volume is constructed, and the number of channels is reduced to 1 through a 3D convolution layer. Subsequently, the cost volume is regularized using a stacked 2D hourglass network, a depth feature map is obtained in combination with the backbone features, and a depth probability volume and a confidence map are generated through a Softmax operation. The depth estimation results of the single view and multi-view are fused by the soft-weighted sum method to calculate the final single view depth and alleviate the error caused by pose noise. Finally, the depth map is refined using technologies such as CRF, and the model performance is improved through end-to-end training and parameter adjustment. The depth estimation module generates an accurate depth map by analyzing the depth information in the image, so that the overlapping root hairs before and after can be accurately distinguished and processed when performing image segmentation.

[0075] It can be understood that this embodiment uses the improved MP-Former instance segmentation model to detect and segment root hairs because the root hairs are small and interlaced, and the improved MP-Former instance segmentation model is easier to focus on the depth information in the current frame, thereby improving the segmentation success rate of interlaced root hairs. Specifically:

[0076] (3-1) The image resolution is 2000×1000. For efficient image processing and analysis, we divide the image into several small blocks (patches), each of which has a size of ρ×ρ, where ρ is set to 100 pixels. Therefore, the size of each patch is 100×100 pixels. This means that the entire image will be divided into patches of 20 rows and 10 columns, for a total of 200 patches.

[0077] (3-2) According to the random block occlusion strategy in the context information, set the occlusion rate α = 60%, visiblepatch visible patch X v The number is 120, masked patch mask patch X m The number is 80.

[0078] (3-3), the encoder first receives the input visible image patch X v , embed each image patch as a patch embedding via linear projection and add the position embedding P v , generate the initial embedding vector u v Then, the initial embedding vector u v Input into a series of Transformer blocks based on the self-attention mechanism to gradually generate the potential representation Z v .

[0079] (3-4), the regressor uses the representation Z of the visible patch v and the position information of the mask patch are processed through a series of cross-attention based Transformer blocks to generate the potential representation Z of the mask patch m , through the prediction alignment mechanism, ensure that the mask patch representation Z output by the regressor m The representations computed directly with the encoder are aligned in the same representation space.

[0080] (3-5) The decoder accepts the mask patch representation Z from the regressor m The root hair features are gradually extracted and refined through multiple Transformer blocks, and then the final mask image patch Y is predicted through the linear layer. m .

[0081] (4) The depth of the root hairs is predicted using the depth estimation module. This embodiment uses the most advanced single-target depth estimation model depth anything, which generates an accurate depth map by analyzing the depth information in the image and inputs it into MP-Former as feature information, so that when performing image segmentation, the overlapping root hairs can be accurately distinguished and processed. This reduces the segmentation errors caused by the crossing of root hairs and improves the reliability and accuracy of the overall segmentation effect.

[0082] (5) Please refer to Figure 7 , set a low threshold L and a high threshold H, retain the detection results with confidence higher than the high threshold H, and for the detection results with confidence between the high threshold H and the low threshold L, only retain the detection and segmentation results with 100 or more pixels in the bounding box;

[0083] (6) After the root hair segmentation of each patch is completed, the previously saved patch information is read and each patch is pieced back into the original image. In the restoration process, the context information is combined to ensure the consistency of the same instance in the overlapping area, and the density of root hairs per unit area is calculated, laying the foundation for subsequent plant phenotypic research.

[0084] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0085] Based on the same idea as the root hair segmentation method based on deep learning in the above-mentioned embodiment, the present invention also provides a root hair segmentation system based on deep learning, which can be used to execute the above-mentioned root hair segmentation method based on deep learning. For ease of explanation, the structural schematic diagram of the embodiment of the root hair segmentation system based on deep learning only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0086] See also Figure 8 In another embodiment of the present application, a root hair segmentation system 100 based on deep learning is provided, the system comprising an image segmentation module 101, an image preprocessing module 102, a root hair detection and segmentation module 103, a detection result selection and screening module 104, and a root hair density calculation module 105;

[0087] The image segmentation module 101 is used to process the input image, identify the root hair trunk information in the image, use the identified geometric center of the root hair trunk as the center point, and divide the entire image into a plurality of independent sub-region patches based on the center point;

[0088] The image preprocessing module 102 is used to perform image preprocessing on a plurality of independent sub-region patches;

[0089] The root hair detection and segmentation module 103 is used to input the preprocessed patch into a preset CAE module to learn the appearance features of the root hairs; the image processed by the CAE module is input into the root hair instance segmentation model to detect and segment the root hairs; in the CAE module, the input patch is segmented into visible patches and mask patches, the encoder is used to process the visible patches to generate potential representations, the regressor is used to predict the potential representations of the mask patches, and finally the decoder is used to reconstruct the occluded patches; the root hair instance segmentation model uses an improved MP-Former. To address the problems of low recognition rate caused by interlaced root hairs and under-utilization of a large number of unlabeled images, we use a convolutional autoencoder (CAE) model and pre-train with unlabeled images to further extract root hair features; in addition, we apply the Depth-Anything model to extract the depth information of all root hairs to accurately identify the front and back relationship of the root hairs. Through this method, we can capture the spatial relationship between root hairs in more detail and enhance the model's ability to parse complex root hair structures, thereby improving overall recognition accuracy and robustness. The improved MP-Former focuses on the depth information in the current frame to improve the segmentation success rate of interlaced root hairs.

[0090] The detection result selection and screening module 104 is used to set a low threshold L and a high threshold H after the modified MP-Former model outputs the segmentation result, retain the detection results with a confidence level higher than the high threshold H, and for the detection results with a confidence level between the high threshold H and the low threshold L, only retain the detection segmentation results within the set range in the bounding box;

[0091] The root hair density calculation module 106 is used to record the root hair segmentation result, and after completing the segmentation of the entire area, record the number of root hairs in each patch and calculate the density of root hairs in the area.

[0092] It should be noted that the deep learning-based root hair segmentation system of the present invention corresponds one-to-one to the deep learning-based root hair segmentation method of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the deep learning-based root hair segmentation method are all applicable to the embodiment of the deep learning-based root hair segmentation. For specific contents, please refer to the description in the embodiment of the method of the present invention. It will not be repeated here. This is hereby declared.

[0093] In addition, in the implementation of the deep learning-based root hair segmentation system in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the deep learning-based root hair segmentation system is divided into different program modules to complete all or part of the functions described above.

[0094] See also Fig. 9 In one embodiment, an electronic device for implementing a root hair segmentation method based on deep learning is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a root hair segmentation program 203 based on deep learning.

[0095] Wherein, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the root hair segmentation program 203 based on deep learning, but also can be used to temporarily store data that has been output or is to be output.

[0096] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0097] Fig. 9 Only an electronic device with components is shown, and those skilled in the art will understand that Fig. 9 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0098] The root hair segmentation program 203 based on deep learning stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions, and when running in the first processor 201, it can achieve:

[0099] Processing the input image, identifying the root hair trunk information in the image, taking the geometric center of the identified root hair trunk as the center point, and dividing the entire image into a plurality of independent sub-region patches based on the center point;

[0100] Perform image preprocessing on several independent sub-region patches;

[0101] The preprocessed image is input into the root hair instance segmentation model to detect and segment the root hairs; the root hair instance segmentation model adopts an improved MP-Former model, and the improved MP-Former model includes a CAE module and a depth estimation module; when detecting and segmenting the root hairs, the preprocessed patch is first input into the preset CAE module to learn the appearance features of the root hairs. In the CAE module, the input patch is segmented into visible patches and mask patches, the encoder is used to process the visible patches to generate potential representations, the regressor is used to predict the potential representations of the mask patches, and finally the decoder is used to reconstruct the occluded patches; then the Depth-Anything model is applied to extract the depth information of all the root hairs to identify the front and back relationship of the root hairs; the improved MP-Former model focuses on the depth information in the current frame to improve the segmentation success rate of staggered root hairs; the depth estimation module uses ConvNeXt-T as a shared backbone network to extract multi-scale features of the root hair image, and the root hair image is divided into single view and multi-view branches to generate depth information of the root hair image respectively;

[0102] After the improved MP-Former model outputs the segmentation results, a low threshold L and a high threshold H are set, and the detection results with confidence higher than the high threshold H are retained. For the detection results with confidence between the high threshold H and the low threshold L, only the detection segmentation results within the set range in the bounding box are retained;

[0103] Record the root hair segmentation results. After completing the segmentation of the entire area, record the number of root hairs in each patch and calculate the density of root hairs in the area.

[0104] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A root hair segmentation method based on deep learning, characterized in that: The steps include: Processing the input image, identifying the root hair trunk information in the image, taking the geometric center of the identified root hair trunk as the center point, and dividing the entire image into a plurality of independent sub-region patches based on the center point; Perform image preprocessing on several independent sub-region patches; The preprocessed image is input into the root hair instance segmentation model to detect and segment the root hairs; the root hair instance segmentation model adopts an improved MP-Former model, and the improved MP-Former model includes a CAE module and a depth estimation module; when detecting and segmenting the root hairs, the preprocessed patch is first input into the preset CAE module to learn the appearance features of the root hairs. In the CAE module, the input patch is segmented into visible patches and mask patches, the encoder is used to process the visible patches to generate potential representations, the regressor is used to predict the potential representations of the mask patches, and finally the decoder is used to reconstruct the occluded patches; then the Depth-Anything model is applied to extract the depth information of all the root hairs to identify the front and back relationship of the root hairs; the improved MP-Former model focuses on the depth information in the current frame to improve the segmentation success rate of staggered root hairs; the depth estimation module uses ConvNeXt-T as a shared backbone network to extract multi-scale features of the root hair image, and the root hair image is divided into single view and multi-view branches to generate depth information of the root hair image respectively; After the improved MP-Former model outputs the segmentation results, a low threshold L and a high threshold H are set, and the detection results with confidence higher than the high threshold H are retained. For the detection results with confidence between the high threshold H and the low threshold L, only the detection segmentation results within the set range in the bounding box are retained; Record the root hair segmentation results. After completing the segmentation of the entire area, record the number of root hairs in each patch and calculate the density of root hairs in the area.

2. The root hair segmentation method based on deep learning according to claim 1, characterized in that: The input image is processed to identify the root hair trunk information in the image, and the geometric center of the identified root hair trunk is used as the center point. Based on the center point, the entire image is divided into a plurality of independent sub-region patches, specifically: Each image is input into the preset SAM model to generate image embedding, the hint information is input into the hint encoder, and the image embedding and hint information are combined through the lightweight mask decoder to predict the image segmentation mask; Set the number of image segmentations in the vertical direction, the overlapping ratio of the sliding window, read the annotation information of the root hair trunk to generate a mask, and find the outline of the mask. Calculate the height of each segmented area based on the image height and the number of segmentations, calculate the center point of each segmented area based on the height of each segmented area, calculate the size of the window according to the predefined aspect ratio, and generate the coordinates of the window according to the outline points. Finally, crop the window image and record the information of each patch.

3. The root hair segmentation method based on deep learning according to claim 1, characterized in that: The image preprocessing of several independent sub-region patches is specifically as follows: Convert the currently input patch into a grayscale image, use the Canny edge detection algorithm to identify the edges in the grayscale image, obtain an edge intensity map, and then calculate the average intensity value of the edge intensity map. Compare the average intensity value with the set threshold. If it is less than the threshold, the current patch needs to be sharpened. The grayscale image is judged, the signal-to-noise ratio is calculated based on the mean and standard deviation of the image, and the signal-to-noise ratio is compared with the set threshold. If it is less than the threshold, the current patch is denoised to obtain the final preprocessed image; The preprocessed images were expanded using the "copy and paste" method, and the bounding box area of ​​the root hairs in a certain original image in the dataset was copied to another random original image to generate new samples.

4. The root hair segmentation method based on deep learning according to claim 1, characterized in that: First, the preprocessed patch is input into the preset CAE module to learn the appearance characteristics of the root hair, specifically: Divide the image into several patches and set the size of the patches; According to the random block occlusion strategy in the context information, the occlusion rate is set to obtain the visible patch mask patch; The encoder first receives the input visible image patches, embeds each image patch into a patch embedding through linear projection, and adds the position embedding to generate an initial embedding vector; Then, the initial embedding vector is input into a series of Transformer blocks based on the self-attention mechanism to gradually generate latent representations; The regressor uses the representation of the visible patch and the position information of the mask patch to process it through a series of cross-attention-based Transformer blocks to generate a potential representation of the mask patch. Through the prediction alignment mechanism, the mask patch representation Z output by the regressor is ensured to be m Aligned in the same representation space as the representations computed directly by the encoder; The decoder receives the mask patch representation and the position information of the mask patch from the regressor, gradually extracts and refines the root hair features through multiple Transformer blocks, and then predicts the final mask image patch through the linear layer.

5. The root hair segmentation method based on deep learning according to claim 1, characterized in that: It also includes predicting the depth of the root hair using a preset depth estimation module, specifically: In the depth hypothesis sampling stage, a set number of depth hypotheses are uniformly sampled in the logarithmic space, and then a 4D cost volume is constructed, and the number of channels is reduced to 1 through a 3D convolutional layer; Subsequently, the cost volume is regularized using a stacked 2D hourglass network, the depth feature map is obtained by combining the backbone features, and the depth probability volume and confidence map are generated through the Softmax operation; the depth estimation results of the single view and multi-view are fused through the soft-weighted sum method to calculate the final single view depth; Finally, the CRF technique is used to refine the depth map, and the model performance is improved through end-to-end training and parameter adjustment.

6. The root hair segmentation method based on deep learning according to claim 1 or 5, characterized in that: The depth estimation module adopts a single-target depth estimation model depth anything, and generates an accurate depth map by analyzing the depth information in the image, which is input into MP-Former as feature information, so that the overlapping root hairs can be accurately distinguished and processed during image segmentation.

7. The root hair segmentation method based on deep learning according to claim 1, characterized in that: The root hair segmentation result is recorded. After the segmentation of the entire area is completed, the number of root hairs in each patch is recorded, and the density of root hairs in the area is calculated, specifically: After the root hair segmentation of each patch is completed, the previously saved patch information is read, and each patch is pieced back into the original image. During the restoration process, the context information is combined to ensure the consistency of the same instance in the overlapping area. Finally, the total number of root hairs n is recorded, and the density of root hairs per unit area is calculated based on the area of ​​the original image and the total number of root hairs.

8. A root hair segmentation system based on deep learning, characterized in that: A root hair segmentation method based on deep learning applied to any one of claims 1-7, comprising an image segmentation module, an image preprocessing module, a root hair detection and segmentation module, a detection result selection module, and a root hair density calculation module; The image segmentation module is used to process the input image, identify the root hair trunk information in the image, use the identified geometric center of the root hair trunk as the center point, and divide the entire image into a plurality of independent sub-region patches based on the center point; The image preprocessing module is used to perform image preprocessing on a plurality of independent sub-region patches; The root hair detection and segmentation module is used to input the preprocessed image into the root hair instance segmentation model to detect and segment the root hairs; the root hair instance segmentation model adopts the improved MP-Former model, and the improved MP-Former model includes a CAE module and a depth estimation module; when detecting and segmenting the root hairs, the preprocessed patch is first input into the preset CAE module to learn the appearance characteristics of the root hairs. In the CAE module, the input patch is segmented into visible patches and mask patches, the encoder is used to process the visible patches to generate potential representations, the regressor is used to predict the potential representations of the mask patches, and finally the decoder is used to reconstruct the occluded patches; then the Depth-Anything model is applied to extract the depth information of all the root hairs to identify the front and back relationship of the root hairs; the improved MP-Former model focuses on the depth information in the current frame to improve the segmentation success rate of staggered root hairs; the depth estimation module uses ConvNeXt-T as a shared backbone network to extract multi-scale features of the root hair image, and the root hair image is divided into single view and multi-view branches to generate depth information of the root hair image respectively; The detection result selection and screening module is used to set a low threshold L and a high threshold H after the improved MP-Former model outputs the segmentation result, retain the detection results with a confidence level higher than the high threshold H, and for the detection results with a confidence level between the high threshold H and the low threshold L, only retain the detection segmentation results within the set range in the bounding box; The root hair density calculation module is used to record the root hair segmentation results. After completing the segmentation of the entire area, the number of root hairs in each patch is recorded and the density of root hairs in the area is calculated.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based root hair segmentation method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the root hair segmentation method based on deep learning described in any one of claims 1 to 7 is implemented.

Images