Automatic hair follicle identification method and system based on deep learning and hair transplant robot

Abstract

The present invention discloses a method and system for automatic hair follicle identification based on deep learning, as well as a hair transplant robot. The method extracts images of hair follicles and their respective target hair follicles in the hair removal area of ​​a captured image; then constructs a deep learning model and evaluates the target hair follicle images; selects hair follicle identification images that meet preset conditions based on the evaluation results; and finally, obtains the position information of the hair follicle roots in the hair follicle images based on the hair follicle identification images. The method provided by the present invention can automatically perform the hair removal process during a hair transplant operation by identifying hair follicles and their root locations through deep learning. The algorithm can automatically identify, evaluate, and locate the coordinates of the hair follicle roots and plan the insertion path of the hair transplant robot based on the image data transmitted back by the camera. The robot can assist the doctor in performing the operation and perform the hair removal work fully automatically, effectively reducing the labor cost and surgical time of hair transplant surgery, and bringing favorable improvements to the problem of hair loss.

Description

Technical Field

[0001] The present invention relates to the field of image information processing technology, and in particular to a deep learning-based automatic hair follicle recognition method and system and a hair transplant robot. Background Art

[0002] With the development of the internet and rising urbanization, more and more people are moving into office buildings and transitioning to a nine-to-five routine. In recent years, there's been a growing consensus that long hours in front of screens and high workplace pressure are contributing to a growing number of hair loss sufferers. Objectively speaking, hair loss is a combination of innate and acquired factors, not solely due to acquired stress. However, this consensus also reflects a growing emphasis on hair health in public aesthetics, which directly exacerbates the appearance anxiety of those suffering from hair loss. For middle-aged patients with hair loss, hair transplant surgery has become virtually the only permanent cure. Essentially, hair transplant surgery involves transplanting permanent hair follicles from the back of the head to the forehead, where hair loss is most prevalent, to restore the hairline to an aesthetically pleasing level.

[0003] The hair follicle is a continuous pocket of epidermal cells that forms a continuous epithelium. It is the origin of hair and provides nutrients for hair growth and maintenance. The base of the hair follicle is the dermal papilla, which is recessed into the dermis. The center is a hair. One side of the arrector pili muscle is attached obliquely to the follicle wall. Above the attachment point is the short neck where the sebaceous gland enters the hair follicle. The opening of the hair follicle on the skin surface is the follicular pore, from which the hair grows. The density of hair follicles in the epithelial tissue of the human head is 80-140 / cm 2 . A single hair follicle only supports the growth of one hair. In 1984, Headington discovered through scalp cross-section that human hair grows in bundles. The reason is that hair follicles are generally clustered in units of 2-4 hair follicles on the human scalp. Such a group of hair follicles is called a follicular unit (FU) in histology. In a follicular unit, the hair follicles are separated from each other by a small amount of soft tissue, and each hair follicle has independent nerves, blood vessels, sebaceous glands, sweat glands, and arrector pili muscles. The follicular unit is surrounded by a collagen fiber sheath.

[0004] The most popular hair transplant technique currently is follicular unit extraction (FUE). This technique uses a follicle extractor to obtain follicular units. Specifically, this tool drills into the superficial layer of the scalp at the back of the head and cheekbones to isolate individual follicular units. These follicles are then extracted using blunt instruments such as tweezers.

[0005] Currently, hair transplants utilize FUE technology, a technique based on follicular unit extraction (FUE). Two teams of surgeons perform the procedure. One team performs surgical work on the patient's head, primarily harvesting hair from the back of the head and transplanting it to the desired location. The other team processes the follicles removed from the back of the head, inhibiting their post-transplant devitalization and transforming them into a transplantable state.

[0006] Current Development Status of Hair Transplant Equipment: To reduce the workload of hair transplant surgeons, reduce manual labor, improve efficiency, shorten surgical operation time, minimize damage to hair follicles, and lower the cost of hair transplantation, research on hair transplant-assisting robots has begun both domestically and internationally. Since 2013, the ARTAS hair transplant robot from the foreign company RESTORATION ROBOTICS has gradually entered the market. In 2017, some Chinese hair transplant institutions introduced this intelligent hair transplant system for the first time, but its frequency of application remains suboptimal. On the one hand, the price of hair transplantation using hair transplant robots remains high, reducing the potential for the hair transplant surgery market to expand further. On the other hand, the superiority of the ARTAS hair transplant robot's hair transplant results is questionable. Reports indicate that the robot's extraction of hair follicles is slow and imprecise, resulting in a high rate of follicle damage, resulting in a lower-than-expected survival rate for the transplanted hair follicles.

[0007] In general, foreign hair transplant robots started early and have been on the market for many years. Their ARTAS hair transplant robot technology has a monopoly position, but due to its high hair transplant costs and less than ideal hair transplant results, the market still has expectations for efficient and reliable automated hair transplant equipment. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a hair follicle automatic identification method and system and a hair transplant robot based on deep learning, which improves the accuracy of hair follicle identification and the precision of hair follicle root positioning.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] The present invention provides a method for automatic hair follicle identification based on deep learning, comprising the following steps:

[0011] Extracting a hair follicle image in a hair removal area of ​​the acquired image;

[0012] extracting each target hair follicle image from the hair follicle image;

[0013] Build a deep learning model and evaluate target hair follicle images;

[0014] Selecting hair follicle recognition images that meet preset conditions based on the evaluation results;

[0015] The hair follicle root position information in the hair follicle image is obtained according to the hair follicle identification image.

[0016] Furthermore, the hair removal area is identified based on a hair removal area positioning device provided on the hair area, and the hair removal area is determined by identifying the hair removal area positioning device.

[0017] Furthermore, the hair follicle image in the hair removal area is processed by image recognition to obtain a fitting rectangle of each target hair follicle outline.

[0018] Furthermore, the target hair follicle image is evaluated by screening the hair follicles in the target hair follicle image based on their morphological features, wherein the morphological features include any one or more of the following features: bifurcation, excessive thinness, multiplication, and blurring of the hair follicles; specifically, as follows:

[0019] Perform any one or more of the following image processing on the target hair follicle image: perform hair follicle image clarity assessment, perform hair follicle clustering detection, perform hair follicle hair thinning detection; perform hair follicle bifurcation assessment on the image after the above image processing; or

[0020] The hair follicle root position information is achieved through hair follicle root positioning processing, and the specific steps are as follows:

[0021] The image processed by hair follicle bifurcation assessment is input into a two-headed neural network consisting of FCN structure and CNN structure, and the hair follicle root position information is output after being processed by the neural network.

[0022] Furthermore, the FCN structure in the dual-headed neural network adopts a U-Net neural network, and the output data structure is a feature map; the neural network of the CNN structure adopts a depth-separable convolution mechanism, and the output data structure is a feature vector.

[0023] Further, the following steps are included:

[0024] S1: Using the above-mentioned deep learning-based automatic hair follicle recognition method to obtain the hair follicle root location information;

[0025] S2: Calculate the hair transplantation path based on the hair follicle root position information to obtain the shortest traversal path for the hair follicle to be transplanted.

[0026] Furthermore, the calculation of the hair transplantation path is solved using the ant-periphery model.

[0027] The present invention provides a hair follicle automatic identification system based on deep learning, the system includes

[0028] at least one processor;

[0029] at least one memory for storing at least one program;

[0030] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned deep learning-based automatic hair follicle identification method.

[0031] The hair transplant robot provided by the present invention includes a robot body and a controller disposed on the body; the controller receives a captured image containing hair follicles, processes the captured image to obtain hair follicle root position information of a target hair follicle image, and generates a hair transplant path based on the hair follicle root position information; the controller is provided with a hair follicle recognition module, a hair follicle assessment and positioning module, and a hair transplant path planning module;

[0032] The hair follicle identification module is used to extract hair follicle images from the original image data stream one by one and pass them to the next module;

[0033] The hair follicle evaluation and positioning module is used to process the hair follicle images extracted by the previous module one by one, evaluate and screen them, and calculate the coordinates of the hair follicle roots as the cutting position;

[0034] The hair transplantation path planning module is used to calculate the hair transplantation path to obtain the shortest traversal path for the hair follicles to be harvested.

[0035] Furthermore, the hair follicle identification module includes a tensioner identification unit and a hair follicle identification unit; the tensioner identification unit is used to identify the hair removal area positioning device; the hair follicle identification unit is used to identify the hair follicles and extract them one by one; or

[0036] The hair follicle assessment and positioning module includes a hair follicle assessment unit and a hair follicle root positioning unit; the hair follicle assessment unit is used to screen target hair follicle images that meet the requirements based on the bifurcation, excessive thinness, multiplication, and blurring of the hair follicles; the hair follicle root positioning unit is used to locate the hair follicle roots of the screened target hair follicle images and use them as the entry point for hair transplantation.

[0037] The beneficial effects of the present invention are:

[0038] The present invention provides a deep learning-based automatic hair follicle identification method and system, as well as a hair transplant robot. This method uses deep learning to identify hair follicles and their root locations, and is used for the visual control function of the hair transplant robot's automatic control system. The hair transplant robot can automatically perform the hair removal process during the hair transplant surgery. This algorithm can automatically identify and evaluate the hair follicles in the target scalp area, automatically locate the coordinates of the hair follicle roots, and plan the hair transplant robot's cutting path based on the image data transmitted back by the camera. The robot is a fully automatic hair transplant robot that can assist doctors in performing the surgery and fully automatically perform the hair removal process, effectively reducing the labor cost and surgical time of hair transplant surgery, and bringing favorable improvements to the problem of hair loss. The hair transplant robot mainly assists the work of the previous doctor group, specifically assisting the previous doctor group in the hair removal process.

[0039] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration:

[0041] Figure 1 Schematic diagram of the tensioner.

[0042] Figure 2 This is a schematic diagram of the tensioner application.

[0043] Figure 3 Flowchart of the tensioner identification algorithm.

[0044] Figure 4 Schematic diagram of the hair follicle recognition algorithm flow.

[0045] Figure 5 This is the hair follicle identification algorithm process and effect diagram.

[0046] Figure 6 Flowchart of the algorithm for the identification and positioning module for hair follicle assessment.

[0047] Figure 7 Schematic diagram showing the difference between the two neural network structures.

[0048] Figure 8 Schematic diagram of the output of two neural networks for the hair follicle positioning problem.

[0049] Figure 9 Schematic diagram of hair follicle positioning effect.

[0050] Figure 10 Schematic diagram of the hair follicle clarity assessment and hair follicle over-fineness and multi-aggregation detection algorithm.

[0051] Figure 11(a) is a flowchart of the hair follicle assessment algorithm

[0052] Figure 11(b) is a flowchart of the neural network for hair follicle bifurcation assessment in the hair follicle assessment algorithm.

[0053] Figure 12 Schematic diagram of depth-wise separable convolution.

[0054] Figure 13 Schematic diagram of ordinary convolution (top) and depth-wise separable convolution (bottom).

[0055] Figure 14 Flowchart of depth-wise separable convolution.

[0056] Figure 15 Schematic diagram of the network classification accuracy for hair follicle bifurcation evaluation.

[0057] Figure 16 Schematic diagram of the U-Net structure.

[0058] Figure 17-1 Take a screenshot for the SE model.

[0059] Figure 17-2 This is the ECA model structure diagram.

[0060] Figure 18 A diagram of hair follicles and their roots with standard markings.

[0061] Figure 19 This is a graph showing the convergence of the training set and the test set.

[0062] Figure 20 This is the recognition effect of DHNet on the root of the hair follicle during several iterations.

[0063] Figure 21 This is a schematic diagram of the DHNet network structure.

[0064] Figure 22 This is a diagram of the DHNet network.

[0065] Figure 23 Identify a flow chart for the output.

[0066] Figure 24 This is a diagram of the DHNet network processing process.

[0067] Figure 25 Schematic diagram of the accuracy and stability of graph output and vector output, as well as the invalid graph output rate (test set data).

[0068] Figure 26 A diagram showing the effect of positioning the neural network at the root of the hair follicle.

[0069] Figure 27 Schematic diagram of a virtual high-dimensional point.

[0070] Figure 28 Hair transplant path planning block scheme.

[0071] Figure 29 Algorithm iteration process record.

[0072] Figure 30 Demonstration of path planning results.

[0073] Figure 31 Schematic diagram of the tensioner.

[0074] Figure 32 Panoramic view of the hair transplant robot.

[0075] Figure 33 Diagram of the structure of the hair transplant robotic arm. DETAILED DESCRIPTION

[0076] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0077] Example 1

[0078] like Figure 1 As shown, the method for automatic hair follicle identification based on deep learning provided in this embodiment includes the following steps:

[0079] Extracting a hair follicle image in a hair removal area of ​​the acquired image;

[0080] extracting each target hair follicle image from the hair follicle image;

[0081] Build a deep learning model and evaluate target hair follicle images;

[0082] Selecting hair follicle recognition images that meet preset conditions based on the evaluation results;

[0083] The hair follicle root position information in the hair follicle image is obtained according to the hair follicle identification image.

[0084] In this embodiment, the hair removal area is identified based on a hair removal area positioning device provided on the hair area, and the hair removal area is determined by identifying the hair removal area positioning device.

[0085] In this embodiment, the hair follicle image in the hair removal area is processed by image recognition to obtain a fitting rectangle of each target hair follicle outline.

[0086] The target hair follicle image in this embodiment is evaluated by screening the hair follicles in the target hair follicle image based on their morphological features, which may include any one or more of the following: bifurcation, excessive thinness, multiplication, and blurring of the hair follicles; the details are as follows:

[0087] Perform any one or more of the following image processing on the target hair follicle image: perform hair follicle image clarity assessment, perform hair follicle clustering detection, perform hair follicle hair thinning detection; perform hair follicle bifurcation assessment on the image after the above image processing; or

[0088] The hair follicle root position information is achieved through hair follicle root positioning processing, and the specific steps are as follows:

[0089] The image processed by hair follicle bifurcation assessment is input into a two-headed neural network consisting of FCN structure and CNN structure, and the hair follicle root position information is output after being processed by the neural network.

[0090] The FCN structure in the dual-headed neural network described in this embodiment adopts a U-Net neural network, and the output data structure is a feature map; the neural network of the CNN structure adopts a depth-separable convolution mechanism, and the output data structure is a feature vector.

[0091] The present embodiment provides a deep learning-based automatic hair follicle recognition system, which includes a hair follicle recognition module, a hair follicle assessment and positioning module, and a hair transplantation path planning module that is required after hair follicle recognition. The three modules provided in this embodiment work in a sequential manner, and their functional details are briefly described as follows:

[0092] The hair follicle identification module is used to extract hair follicle images from the original image data stream one by one and pass them to the next module;

[0093] The hair follicle identification module includes a tensioner identification unit and a hair follicle identification unit;

[0094] The tensioner identification unit is used to identify the tensioner and correct the viewing angle;

[0095] The hair follicle identification unit is used to identify hair follicles and extract them one by one;

[0096] The hair follicle evaluation and positioning module is used to process the hair follicle images extracted by the previous module one by one, evaluate and screen them, and calculate the coordinates of the hair follicle roots as the cutting position;

[0097] The hair follicle assessment and positioning module includes a hair follicle assessment unit and a hair follicle root positioning unit;

[0098] The hair follicle evaluation unit is used to screen hair follicles suitable for hair removal based on whether the hair follicles are bifurcated, too thin, too aggregated, or blurred and empty;

[0099] The hair follicle root positioning unit is used to locate the roots of the screened hair follicles and obtain the entry point of the robot arm.

[0100] The hair transplantation path planning module is used to calculate the hair transplantation path to obtain the shortest traversal path of the hair follicles to be harvested and optimize the hair transplantation speed.

[0101] The working process of the tensioner identification unit of the hair follicle identification module is as follows:

[0102] like Figure 1 As shown, Figure 1 The tensioner provided in this embodiment is a rectangular frame. Figure 2As shown, the tensioner is attached to the back of the patient's head to tighten the skin and define the hair removal area. The four corners of the tensioner are marked with standardized colors to assist the algorithm in positioning. The tensioner recognition algorithm aims to obtain the position of the tensioner plane and transform the hair follicle image in the original image to a normal perspective using a homography matrix for subsequent algorithm recognition.

[0103] like Figure 2 As shown, Figure 2 The following is a schematic diagram of a tensioner application. The main process of the tensioner recognition algorithm is: inversion and binarization with a fixed threshold of 230; kernel size of 10*10 opening operation; contour extraction and contour rectangle fitting to extract the coordinates of each contour center point; calculation of the homography transformation matrix based on the coordinates of each contour center point, and transformation of the original image perspective to the normal perspective.

[0104] like Figure 3 As shown, Figure 3 This is a flowchart of the tensioner identification algorithm. The following are the specific steps of the process:

[0105] 0) Original image;

[0106] 1) Invert the color and perform binarization with a fixed threshold of 230. This is helpful for capturing the color blocks of the marker points;

[0107] 2) Opening with a kernel size of 10*10 helps eliminate small color patches caused by hair;

[0108] 3) Contour extraction and contour rectangle fitting to extract the coordinates of each contour center point;

[0109] 4) Perform homography transformation on the original image according to the four marker points to transform the original image to the normal perspective.

[0110] The specific process of the homography transformation in step 4) is as follows:

[0111] Homography, also known as projective transformation, is a common linear image transformation method. In real life, the same object will produce different images from different perspectives. The relationship between these images is the homography. For a two-dimensional image composed of several pixels, the homography can be described by the following matrix operation:

[0112]

[0113] In the above formula, (x1,x2,1) represents the coordinates of a pixel in the original image, (x″1,x″2,1) represents the coordinates of that pixel in the transformed image, and h represents the parameters of the transformation matrix. A homography is a reversible mapping that does not alter collinearity. The homography matrix has eight degrees of freedom, and solving it requires at least eight equations, representing at least four point correspondences. Solving the homography matrix based on the four markers of the tensioner transforms the original image to a normal perspective for ease of processing by subsequent algorithms.

[0114] The specific process of the hair follicle recognition unit is as follows: the hair follicle recognition algorithm is used to identify the target hair follicle unit in the original video stream, so as to obtain the hair follicle image for the subsequent evaluation and positioning algorithm. The goal of this algorithm is to extract close-up images of hair follicles one by one from the original scalp image, as shown in the figure below. The algorithm flow chart is as follows:

[0115] like Figure 4 As shown, Figure 4 The figure is a schematic diagram of the hair follicle recognition algorithm process. The hair follicle recognition algorithm process in this embodiment is processed by adaptive binary segmentation, followed by an opening operation with a kernel size of 3*3, contour recognition and rectangle fitting, and screening; finally, the original hair follicle image is intercepted according to the fitted rectangle.

[0116] like Figure 5 As shown, Figure 5 This is the hair follicle identification algorithm process and effect diagram.

[0117] The specific process of adaptive binary segmentation in step 1) is as follows:

[0118] Threshold segmentation is a common technique in image processing. It processes grayscale images and its main concept is to divide pixels into grayscale levels to distinguish between foreground and background. For fixed-threshold image segmentation, the key is to find the optimal threshold T according to certain criteria. This optimal threshold T is applied directly to all pixels. Pixels with grayscale values ​​below T are set to 0, while pixels with grayscale values ​​above T are set to 255. Methods for determining the optimal threshold T include the maximum between-class variance method (OTSU) and the maximum entropy method. Fixed-threshold threshold segmentation methods have significant disadvantages when it comes to hair follicle identification. This is primarily due to their inability to handle images with uneven brightness. The geometric shape of the back of the human head is spherical, and uneven illumination is often seen at the edges. Applying fixed-threshold threshold segmentation methods will result in the submerging of hair follicles at the edges of the illumination.

[0119] Adaptive threshold segmentation is a flexible threshold segmentation method. The threshold of each pixel is determined by the grayscale distribution of pixels within a certain distance around it. Common threshold selection methods include the average method and the Gaussian method. Both methods require the user to pre-set the pixel distribution range that affects the threshold. In the average method, the segmentation threshold T of the pixel at the (i, j) coordinate is (i,j) Determined by the following formula:

[0120]

[0121] Among them, G (x，y) is the grayscale of the pixel at coordinate (x, y), and r is the pixel distribution range that affects the threshold. Generally speaking, smaller r values ​​make adaptive segmentation more sensitive to the high-frequency components of the image, resulting in more detailed representation of light and dark changes at image boundaries and a greater impact of noise. Larger r values ​​make the adaptive segmentation closer to global threshold segmentation, resulting in a coarser representation of light and dark changes in the segmented image and stronger noise suppression. To achieve a compromise, an r value of 25 was selected in this example, which is more reasonable and produces a better binarization effect.

[0122] The working process of the hair follicle assessment, identification and positioning module is as follows:

[0123] After obtaining segmented single follicle images, the follicles are further evaluated and located. This evaluation focuses on structural features such as follicle clustering, bifurcation, thickness, and image blur. These structural features can affect follicle transplantability in the following ways: Inadequate follicle image clarity makes follicle location impossible. Excessively thin or bifurcated follicles indicate low bioavailability and a low transplant survival rate. Clustered follicles hinder implant placement. The goal of location is to pinpoint the precise coordinates of the follicle root. These problems are relatively complex and difficult to solve using traditional algorithms. This module's design objectives are achieved using a combination of traditional algorithms and neural networks.

[0124] like Figure 6 As shown, Figure 6 This is the algorithm flow chart of the hair follicle assessment, identification and positioning module, which includes a hair follicle assessment unit and a hair follicle root positioning unit. The hair follicle assessment unit is used for hair follicle image clarity assessment, hair follicle clustering detection, and hair follicle hair thinning detection, which can be implemented using traditional image algorithms. The hair follicle bifurcation assessment function is implemented using a CNN structure neural network for hair follicle assessment. The hair follicle root positioning part is implemented using a dual-headed neural network that combines the FCN structure and the CNN structure for hair follicle root positioning.

[0125] like Figure 7 As shown, Figure 7 As shown, Figure 7 The following is a diagram showing the difference between the two neural network structures and the reasons for using such an architecture:

[0126] First, hair follicle image clarity can be described using the information entropy of the grayscale histogram, hair follicle clustering can be detected using the number of connected domains in the binarized hair follicle image, and the degree of hair thinning can be described using the area of ​​the connected domain in the binarized hair follicle image. These features are relatively basic, with a low level of abstraction and simple mathematical modeling. Using traditional algorithms saves time and computing power.

[0127] The output of the hair follicle bifurcation assessment neural network is the structural characteristics of the hair follicle, while the output of the hair follicle localization algorithm is the coordinates of the hair follicle root. In the field of computer vision, hair follicle bifurcation assessment is an image-level regression problem (hereinafter referred to as image regression), while hair follicle root localization is a pixel-level regression problem (hereinafter referred to as pixel regression), also known in academia as the semantic segmentation problem. These two types of problems are conceptually relative. Image regression primarily refers to problems that focus on abstract information in an image, such as target or object shape recognition and detection, while pixel regression primarily refers to problems that focus on simple structural information in an image, such as image segmentation and image registration.

[0128] Convolutional neural network (CNN) is a common neural network structure with high applicability in image regression problems. The reason is that the pyramid-structured neural network calculation graph can rely on calculation methods such as pooling and convolution to gradually improve the receptive field of feature layer pixels, so as to pass the abstract information in the image to the deeper neural network and finally output it by the fully connected layer network.

[0129] In the hair follicle bifurcation assessment algorithm, the neural network is designed using a convolutional neural network (CNN) architecture. However, when it comes to locating hair follicle root coordinates, the traditional convolutional neural network (CNN) is not suitable. After convolution, CNNs are connected to several fully connected (FC) layers to map the convolution results into fixed-length feature vectors. However, according to deep learning researchers, the final fully connected (FC) layer in a CNN loses spatial structural information during information transmission. To compensate for this loss of spatial structural information, the network parameters must be increased and the network depth must be increased, which increases the likelihood of overfitting and reduces performance. Furthermore, the output in the form of a feature vector is difficult to interpret, making it difficult for researchers to determine the confidence level or the presence of other suboptimal solutions. These shortcomings make CNNs less suitable for locating hair follicle root coordinates.

[0130] A fully convolutional neural network (FCN) is a type of neural network suitable for pixel-wise regression problems. FCNs accept input images of any size. After the input image is transformed into a high-dimensional feature map through continuous convolution and downsampling, it is not directly fed into the fully connected layer. Instead, it is restored to the same size as the input image through an equal number of deconvolution layers or upsampling. This generates a predicted value for each pixel. The FCN structure preserves the spatial information in the original input image. Leveraging this property, FCNs are effective for semantic-level image segmentation.

[0131] like Figure 8 As shown, Figure 8 This is a schematic diagram of the output of two neural networks for the hair follicle positioning problem. In the design of the visual part of the hair transplant robot control system, the CNN structure neural network directly outputs two-dimensional coordinates, while the FCN structure neural network outputs a hair follicle root positioning marker map. Figure 8 The [ ] in the figure is the original image of the hair follicle and the hair follicle root marker map identified by the neural network. It can be seen that the hair follicle root marker map has a significant high response at the location where the hair follicle appears, and it is possible that several locations in the image that resemble the hair follicle root also have high responses of varying intensities. Compared to the result vector with a fixed length output by CNN, the marker map output by FCN has the following obvious advantages: flexible output format, which can output the optimal result and several better results at the same time; rich output information, and the confidence of each output can be evaluated, which is conducive to the later screening of failed hair follicles. It has higher translation invariance and scaling invariance, such as Figure 9 As shown, Figure 9 Schematic diagram of hair follicle positioning effect.

[0132] The output images for hair follicle localization often present numerous challenges, including complex content, variable structure, and low robustness, which can result in invalid output. FCN neural networks require high parameters, are difficult to train, and have slow computational speeds. The output labeled images cannot be used directly, requiring subsequent processing with other algorithms. Invalid images generated by hair follicle localization suffer from jagged patterns, which compromises localization accuracy and renders them invaluable. FCN neural networks output graphs, while CNN neural networks output feature vectors. Graph outputs that fail to identify valid information are referred to as null outputs. Practical experience shows that the probability of null outputs generally decreases with increasing training depth. Even after algorithm convergence, a null output rate of approximately 10%-12% persists, significantly impacting localization.

[0133] In order to solve the defects of the two types of neural networks and integrate the performance of the two neural networks, this embodiment adopts a dual-head positioning neural network. The network has two branches, which can simultaneously generate a graph output based on the U-Net structure and a feature vector output based on the CNN structure. The graph output has higher accuracy and higher interpretability, but it may produce empty output. The feature vector output has lower accuracy and poor interpretability, but it is highly robust and has stable output. A judgment program is designed at the exit of the neural network. The program will evaluate the quality of the two outputs to ultimately determine which output to use as the positioning result. The main basis of the judgment program is: when the graph output has a high interpretability, the graph output is given priority as the final result, otherwise the feature vector output is used as the final result.

[0134] The hair follicle evaluation unit of this embodiment includes a hair follicle clarity evaluation and hair follicle over-fineness and multiplication detection algorithm, and a hair follicle bifurcation evaluation neural network.

[0135] The main processes of the hair follicle clarity assessment and hair follicle over-clustering detection algorithms are as follows:

[0136] Grayscale the original input hair follicle image;

[0137] Convolve with the Laplacian operator to obtain the grayscale gradient of the image; the Laplacian operator is as follows:

[0138]

[0139] The grayscale gradient histogram is calculated, and the average grayscale of the top 5% of pixels in the grayscale gradient is used as the image clarity index. Hair follicle images with a clarity index less than a set threshold are screened out and marked as blurred images.

[0140] Select the hair follicle image that has passed the clarity evaluation and convolve it with the following convolution kernel:

[0141]

[0142] Invert the original image and use the maximum inter-class difference method to binarize the original image;

[0143] Connected domain screening to remove connected domains with too small an area;

[0144] Perform corrosion operation, the kernel size of the corrosion operation is 4*4;

[0145] The connected domain area of ​​the eroded image is calculated. The hair follicle images with too small connected domain area are filtered out and marked as too fine hair follicles; the hair follicle images with more than 2 connected domains are filtered out and marked as polycystic hair follicles; the remaining hair follicles are marked as normal hair follicles and output.

[0146] like Figure 10 As shown, Figure 10This is a schematic diagram of the effects of the hair follicle clarity assessment and hair follicle over-fineness and multi-clustering detection algorithms. The following is an example of the processing effects of each process:

[0147] The algorithms for hair follicle clarity assessment and hair follicle over-fineness and multi-aggregation detection are as follows:

[0148] 1. The principle of using the Laplacian operator to evaluate image emotion in steps 2) and 3) is as follows: The Laplacian operator describes the grayscale gradient of a pixel in an image. Convolving it with the original image calculates the second-order grayscale gradient of all pixels. A sharp image exhibits more dramatic grayscale transformations than a blurred image, resulting in a higher overall grayscale gradient for a sharp image. For ease of calculation, the average grayscale gradient of the top 5% of pixels with the largest grayscale gradients in an image is used as the image clarity metric.

[0149] 2. The significance of using a special convolution kernel in step "4"): The special convolution kernel can cut out the background and increase the grayscale range between the foreground and background of the image. The next step of this step is the image maximum inter-class difference binarization algorithm. A larger grayscale range helps the binarization algorithm obtain more accurate binarization results.

[0150] 3. The significance of removing smaller connected domains and performing image erosion in steps 6 and 7: Removing smaller connected domains helps remove image noise, while erosion can disconnect connected domains belonging to different hair follicles, helping the algorithm accurately count the number of hair follicles in the image and more accurately distinguish between clustered and overly fine follicles.

[0151] The following are the values ​​of the fixed values ​​in the algorithm:

[0152]

[0153] With these values, the algorithm achieved the following accuracy and confusion matrix results on a manually labeled image classification dataset:

[0154]

[0155]

[0156] As shown in Figure 11(a), the hair follicle assessment algorithm flow chart uses a convolutional neural network structure to design the hair follicle bifurcation assessment neural network. To reduce computational complexity and improve efficiency, a depthwise separable convolution mechanism is employed. First, the input hair follicle image is processed using the hair follicle image clarity assessment algorithm, the hair follicle clustering detection algorithm, and the hair follicle excessively thin detection algorithm. The hair follicle bifurcation assessment neural network is then processed to output the optimal hair follicle image for positioning.

[0157] The hair follicle image clarity assessment algorithm process is as follows: first, the image is grayscaled, convolved with the Laplace gradient operator, and the image grayscale gradient histogram is calculated. The average grayscale gradient of the top 5% of pixels is calculated as the image clarity index. Images with clarity indexes less than a threshold are removed and marked as blurred images. Hair follicle images that pass the clarity assessment are selected and convolved with a preset convolution kernel, then color inverted, binarized using the maximum inter-class difference method, connected domain screening is performed to screen connected domains with areas less than a threshold, connected domain screening is performed to remove connected domains with areas less than the threshold, and then erosion is performed with a kernel size of 4*4. The maximum area of ​​the remaining connected domains is used as the feature value for detecting excessively thin hair follicles. Images less than the threshold are removed and marked as excessively thin hair follicles. The number of connected domains in the remaining connected domains is used as the feature value for detecting hair follicle clustering. Hair follicles with a number of connected domains greater than or equal to 2 are removed and marked as clustered hair follicles. The remaining hair follicle images are marked as normal hair follicles and output as preferred hair follicles.

[0158] As shown in Figure 11(b), the hair follicle bifurcation evaluation neural network is performed according to the following process: input the preferred hair follicle image, calculate the block processing: input channel is 3, output channel is 6, expansion channel is 12, step number is 1; input channel is 8, output channel is 16, expansion channel is 64, step number is 2; input channel is 16, output channel is 24, expansion channel is 112, step number is 1; then maximum pooling: 2; then calculate the block processing: input channel is 24, output channel is 32, expansion channel is 128, step number is 2; input channel is 32, output channel is 64, expansion channel is 2 56, the step number is 1; the input channel is 64, the output channel is 64, the expansion channel is 256, and the step number is 2; then the maximum pooling is 2; the third time is processed by three calculation blocks: the input channel is 64, the output channel is 64, the expansion channel is 256, and the step number is 2; then the maximum pooling is 2; again processed by two calculation blocks: the input channel is 64, the output channel is 64, the expansion channel is 256, and the step number is 2; then the maximum pooling is 4; finally, the fully connected layer is processed with an input channel of 64 and an output channel of 1, the sigmoid activation function is applied, and the hair follicle bifurcation assessment score is output.

[0159] The computational block processing is performed as follows: input image, number of channels is X, convolution processing is performed separately, kernel size is 1, number of channels is Z; batch normalization processing, relu function calculation, convolution operation: kernel size is 3, number of channels is Z, group is 3, number of steps is S; batch normalization again, relu function calculation; ECA model processing, number of channels is Z; convolution: kernel size is 1, number of channels is Y, batch normalization, relu function calculation, and finally superposition with the input image after convolution operation, outputting an image with Y number of channels.

[0160] like Figure 12 、 Figure 13 、 Figure 14 As shown, Figure 12 This is a schematic diagram of depth-wise separable convolution. Figure 13 Schematic diagram of ordinary convolution (top) and depth-separable convolution (bottom). Figure 14 This is a flowchart for depthwise separable convolution. Depthwise convolution is a decomposable convolution operation that can be broken down into two smaller operations: spatial convolution (depthwise convolution) and channel-wise convolution (pointwise convolution). Take, for example, the process of mapping the number of channels in a feature layer from a to b. In a standard k*k two-dimensional convolution, this process requires b sets of convolution kernels, each with a number of channels, and each channel is a k*k two-dimensional convolution kernel. During the convolution operation, each set of convolution kernels independently performs a one-to-one two-dimensional convolution operation with the feature layer, and the results are then superimposed into a single channel, resulting in a total of b channels in the feature layer output. In k*k depthwise separable convolution, this process is divided into two sequential steps. The spatial convolution step requires only one set of convolution kernels, each with a number of channels, and each channel is also a k*k two-dimensional convolution kernel. During convolution, a channel convolution kernel performs a one-to-one two-dimensional convolution operation with a feature layer. The results are not summed up, but the resulting a channels are directly fed into the next stage. The channel convolution stage is equivalent to a regular 1x1 two-dimensional convolution. Its significance lies in increasing the number of channels by combining the spatial convolution results with different weights. The channel convolution stage requires b groups of convolution kernels, each with a channels, and each channel is a fixed 1x1 two-dimensional convolution kernel. During convolution, each group of convolution kernels independently performs a one-to-one two-dimensional convolution operation with the feature layer. The results are then summed up to form a channel, resulting in a total of b channels of feature layer output.

[0161] Generally speaking, the number of calculations C for a normal convolution is given by the following formula:

[0162] C=K S 2 ×H×W×C in ×C out

[0163] The parameter P of the ordinary convolutional layer is given by the following formula:

[0164] P=K S 2 ×C in ×C out

[0165] The symbols are defined as follows K S : convolution kernel width, H: input feature map height; W: input feature map width;

[0166] C in Number of input feature map channels; C out The number of output feature map channels.

[0167] The number of computations of the depthwise separable convolution is given by:

[0168] C=K S 2 ×H×W×C in +C in ×C out

[0169] The parameter P of the depth-wise separable convolution is given by:

[0170] P=K S 2 ×C in +C in ×C out

[0171] The computational ratio of the two convolutions is:

[0172]

[0173] The parameter ratio of the two convolutions is:

[0174]

[0175] It can be seen that the computational complexity and parameter amount of depthwise separable convolution are much smaller than those of ordinary convolution. According to the experiments conducted by the proposers of MobileNet on the Imagenet classification task, the performance of depthwise separable convolution is close to that of ordinary convolution.

[0176] In addition, to accelerate the convergence of the neural network and improve the expressiveness of the neural network parameters, the ECA attention mechanism is added to the computational graph of the neural network, which has the ability to improve the generalization ability of the model and improve the accuracy of the model.

[0177] Among them, the hair follicle bifurcation assessment network has a higher assessment accuracy. The specific experimental data are as follows:

[0178] Figure 15 Schematic diagram of network classification accuracy for hair follicle bifurcation evaluation

[0179]

[0180] The neural network used in the hair follicle localization algorithm is named DoubleHeadNet (DH-Net). DH-Net takes into account the advantages of both structures and consists of two main parts: the main network (MainNet) and the auxiliary network (AssistNet). The main network is a neural network with an FCN structure, whose structural design is based on U-Net, and the output data structure is a feature map; the auxiliary network is a neural network with a CNN structure, whose structural design is based on the depthwise separable convolution mechanism, and the output data structure is a feature vector. Figure 16 As shown, Figure 16 This is a schematic diagram of the U-Net structure, where the basic process of U-Net is as follows:

[0181] U-Net is a typical FCN structure neural network, which consists of two parts: the contracting path and the expansive path.

[0182] The compression path can be regarded as an encoder, which consists of four computing blocks. Each block uses 3 convolution layers and 1 maximum pooling layer for downsampling, which can gradually reduce the size of the input feature map and increase the number of channels of the input feature map. The compression path can gradually extract abstract information from the original input image to describe the abstract structure of the original image. The expansion path can be regarded as a decoder, which consists of four computing blocks. Before the start of each block, the size of the feature map is doubled by upsampling to achieve the same size as the symmetrical compression path on the left. It is then merged with the feature map of the symmetrical compression path on the left and merged into a skip connection. The merged feature map will pass through 3 ordinary convolutions. The expansion path can gradually restore spatial information from the high-dimensional feature map to achieve pixel-scale segmentation calculations.

[0183] The jump connection is the most revolutionary design in the UNet structure. In the process of network propagation, as the depth increases, the receptive field of the corresponding feature map will increase, the size will decrease, and the contained detail information will also decrease, which is not conducive to the accuracy of the final output label map. The jump connection structure uses the concatenation layer (Concat) to combine the feature map extracted by downsampling in the compression path with the new feature map obtained by upsampling in the expansion path. Figure 1The channel-wise splicing is performed one by one. The rich, detailed features retained by the upstream convolutional layer are directly introduced into the downstream convolutional layer through this path alone. This increases the scale information and hierarchical structure of the output, improves information utilization, and achieves a more refined segmentation effect. Furthermore, skip connections increase the error backpropagation path. Each layer on the expansion path can directly transfer the gradient to the corresponding layer on the compression path, which accelerates the convergence of the neural network and reduces the training time.

[0184] To improve the performance of the original U-Net, an attention mechanism was added to the original UNet network structure. The essence of the attention mechanism in neural networks is to enhance the model's expressiveness by allocating weights to increase the influence of key structures on the output, given limited computing resources.

[0185] Neural networks are composed of parameters. Given a fixed number of parameters, the model's expressiveness is closely related to the efficiency of these parameters. The attention mechanism can focus on the information that is most critical to the current task among a large amount of input information, reducing attention to other information, enhancing the spatial encoding quality of feature information, and achieving the generalization capabilities that require several layers of convolution with fewer parameters, thereby improving the efficiency and accuracy of neural networks.

[0186] Attention structures have different specific implications for computer vision (CV), natural language processing (NIP), speech recognition, and other problems. In the field of computer vision, widely used attention mechanisms include SE-Net, CBAM, DANet, and Non-local Neural Networks. SE-Net is the most representative. The core concept of the SE-Net model is to improve the quality of the feature maps generated by the network by modeling the mutual importance of feature channels through a fully connected structure. Specifically, it automatically learns the importance weight of each feature channel and then uses this weight to enhance useful features and suppress unhelpful ones.

[0187] As shown in Figure 17, Figure 17-1 The following is a diagram of the SE-Net process. The main features of the SE-Net structure are:

[0188] 1. SE-Block assigns a weight to each channel, allowing different channels to have different effects on the results.

[0189] 2. This SE module can be easily added to the current mainstream neural network.

[0190] However, according to relevant practical experience, SE-Net networks also have the following problems:

[0191] 1. Large amount of calculation and slow calculation speed

[0192] 2. Too many parameters and redundancy

[0193] In order to improve this type of problem, the ECA attention mechanism is used as Figure 17-2 As shown, Figure 17-2 The ECA model architecture is shown in Figure 2. The ECA attention mechanism is designed to deliver significant performance improvements without increasing model complexity. It employs a local cross-channel interaction strategy without dimensionality reduction, which can be effectively implemented using one-dimensional convolution. Appropriate cross-channel interaction significantly reduces model complexity while maintaining performance.

[0194] like Figure 18 As shown, Figure 18 For the hair follicle map and its root standard label map, the grayscale values ​​of several pixels around the corresponding position of the hair follicle root in the completely black image are set to 255, and the hair follicle root standard label map is constructed, and this is used as the training data of the neural network structure, such as Figure 19 As shown, Figure 19 This is a graph showing the convergence of the training set and the test set.

[0195]

[0196] The experimental data above shows that both ECA and SE attention mechanisms can slightly improve neural network positioning accuracy and significantly enhance positioning stability. Based on data from the 60th training epoch, they achieved 15.6% and 6% higher positioning accuracy, and 63.7% and 74.6% higher positioning stability, respectively, compared to neural networks without attention mechanisms.

[0197] In a comparison of the ECA and SE attention mechanisms, based on data from the 60th training epoch, ECA's localization accuracy improved by 7.52%, while its localization stability decreased by 43.1%. However, ECA only added 0.003 megabytes of file size to the backbone model, a 94.92% reduction compared to SE; and only 0.04 megabytes of file size to the auxiliary network model, a 99.93% reduction compared to SE. In summary, the ECA attention mechanism is superior to the SE attention mechanism in this scenario.

[0198] The following is a demonstration of DHNet's recognition of the hair follicle root during several iterations. It can be observed that the DHNet network gradually learns to locate the hair follicle end. Figure 20 As shown, Figure 20 This is the recognition effect of DHNet on the root of the hair follicle during several iterations.

[0199] like Figure 21 As shown, Figure 21 Schematic diagram of the DHNet network structure, including input unit, backbone network, auxiliary network and output unit. The input unit sets the number of channels and size. The backbone network is provided with an encoding block. The number of input channels in the encoding block is 3, and the number of output channels is 60, so the number of channels is 60. Then downsampling, the number of input channels is 60, the number of output channels is 100, and the size is reduced, and the original 32*32 is changed to 16*16. The next downsampling block has 100 input channels, 100 output channels, and the size is changed to 8*8. Then upsampling, the size is changed to 16*16, and the channel number is merged with the channel number of 100 and the size of 16*16 to obtain a channel number of 200 and a size of 16*16. Repeat the upsampling block, and merge the channel number with the initial channel number of 60 and the size of 32*32 to obtain a channel number of 120 and a size of 64*64. The decoding block converts the input channel number of 32 into an output channel number of 64. Finally, the output channel number is 1 and the size of 32*32. The auxiliary network block in the auxiliary network receives the image with 100 channels and size 8*8 in the backbone network, and outputs an image with 100 channels, 800 expanded channels, and a kernel size of 3 through several auxiliary network blocks. Then, an image with 100 channels and size 8*8 is obtained. After maximum pooling and average pooling, an image with 32 channels and size 64*64 is obtained, and the coordinates are output through the fully connected layer and sigmoid function.

[0200] like Figure 22 As shown, Figure 22 This is a schematic diagram of the DHNet network. The figure describes the working process of each block, including upsampling block, downsampling block, residual block, encoding block, decoding block, ECA model and auxiliary network block. Each block includes input unit, intermediate processing unit and output unit.

[0201] like Figure 23 As shown, Figure 23 In order to output the recognition flowchart, the hair follicle root marker map output by DHNet needs to be interpreted to finally obtain the hair follicle root coordinates. In the interpretation process, it is necessary to determine whether the current map output is valid, so that when necessary, the map output can be discarded and the feature vector output can be accepted to improve the robustness of the algorithm. The processing flow is as follows:

[0202] Perform global threshold segmentation with a threshold of 135.

[0203] Connected domain count.

[0204] If the connected domain is not unique, calculate the grayscale centroid of the maximum contour and output it.

[0205] The final output value is the coordinate of the hair follicle root.

[0206] like Figure 24 As shown, Figure 24 This is an illustration of the DHNet network processing process. The graph output and feature vector output generated by DHNet are used in the learning process of locating the hair follicle root.

[0207] Figure 25 Schematic diagram of the accuracy and stability of graph output and vector output, as well as the invalid graph output rate (test set data)

[0208] The statistical values ​​in practice are listed as follows:

[0209]

[0210] It is easy to observe that the graph output is generally more accurate and more stable than the vector output. The DH-Net network can achieve an average error of about 2.13 pixels in a single hair follicle image of 32*32 size, which has high application value. Figure 26 As shown, Figure 26 A diagram showing the effect of positioning the neural network at the root of the hair follicle.

[0211] The path planning algorithm in the hair transplant path planning module in this embodiment is as follows:

[0212] After hair follicle localization is complete, a point cloud map is formed, ultimately requiring path planning for hair transplantation. Planning a hair transplant path is mathematically equivalent to the Traveling Salesman Problem (TSP), which does not require a return to the starting point. The TSP is a well-known mathematical problem, proven to be NP-Complete (or NPC). The main idea is that a traveling salesman plans a route to visit n cities. The route is restricted to visiting each city only once and ultimately returning to the original starting point. The shortest possible path must be found.

[0213] To date, no efficient algorithm has been found for any of these problems. The academic community tends to accept the conjecture that there are no efficient algorithms for NP-Complete (NPC) and NP-Hard (NPH) problems. It holds that large instances of these problems cannot be solved with exact algorithms, and that efficient approximate algorithms must be sought for them.

[0214] Generating simulated follicle maps and simulated path planning: The Ant Colony Algorithm (AG) is a leading heuristic algorithm for solving the Traveling Salesman (TSP) problem. It simulates the path planning behavior of a colony of ants while foraging. Invented by Italian scholars Dorigo, Maniezzo, and others in the early 1990s, it is a key heuristic search algorithm for combinatorial optimization, following other heuristic search algorithms such as simulated annealing, genetic algorithms, and tabu search. Dorigo and others applied the AG to classic optimization problems, including the Traveling Salesman (TSP) problem and the quadratic assignment problem, achieving excellent results.

[0215] The main ideas of the ant colony algorithm are as follows:

[0216] Several artificial ants are randomly placed on the path graph. In each iteration, the ants will traverse the entire map.

[0217] When the artificial ants encounter an intersection or a fork in the road, they comprehensively consider the pheromone concentration and the specific details of each fork and the next path point to determine the next path point to visit.

[0218] After an iteration, each ant will update the pheromone on the path of that iteration, and the old pheromone will evaporate at a certain ratio.

[0219] Based on the above principles, after several iterations, ants will naturally find the optimal traversal path.

[0220] The main formula of the ant colony algorithm is as follows:

[0221] The probability of an ant choosing a path is determined by the following formula:

[0222]

[0223] The symbols in the above formula are defined as follows: d ij is the distance from path point i to path point j, τ ij The pheromone concentration on the path from path point i to path point j, The probability that ant k will choose the next path point j when it is on the path point i, allowedx is the set of unvisited path points, α is the heuristic pheromone importance factor, and β is the heuristic distance importance factor.

[0224] The update formula for path pheromones is defined differently in different variations of the ant colony algorithm, specifically three variations: the Ant-Cycle model, the Ant-Quantity model, and the Ant-Density model. The Ant-Cycle model utilizes global information, meaning that ants update pheromones on all paths only after completing a cycle, and the updated value is inversely proportional to the total length of the ant's traversed path. The Ant-Density model and the Ant-Quantity model utilize local information, meaning that ants update pheromones on the path immediately after completing each step. The Ant-Cycle model maintains a constant pheromone update quantity, while the Ant-Quantity model updates a value inversely proportional to the path length.

[0225] The update of path pheromone is determined by the following formula:

[0226]

[0227]

[0228] Among them, τ ij (t+1) is the pheromone concentration between path point i and path point j at time t+1, τ ij (t) The pheromone concentration between path point i and path point j, ρ is the heuristic pheromone volatility factor, Q is the system constant, l k is the moving distance of ant k traversing all path points, Z ij is the set of all ants whose paths contain paths from i to j, d ij is the distance from path point i to path point j.

[0229] Compared with the ant density model and the ant quantity model, the ant-perimeter model pays more attention to the globality of path planning. In the hair transplant path planning algorithm, the ant-perimeter model is used to solve the hair transplant path.

[0230] When solving the multi-point TSP problem, the ant colony algorithm is faster than conventional algorithms and can produce more stable and superior results. However, it is important to note that hair transplant path planning is not equivalent to the TSP problem. In this problem, the path traversing the hair follicles does not need to return to the starting point, but rather to reach a predetermined endpoint. By processing the input data, the ant colony algorithm program designed for solving the TSP problem can be reused for hair transplant path planning.

[0231] The key to this technique is to preprocess the undirected distance graph that is imported into the ant colony algorithm program. A high-dimensional virtual point can be added to the original undirected graph. The distance between this point and the path points in the graph is set as follows: the distance between this high-dimensional virtual point and only the starting and ending points approaches zero; the distance between this point and all other points is infinite.

[0232] Figure 27The virtual high-dimensional point is shown. After adding the virtual high-dimensional point, when using the ordinary ant colony algorithm to solve the optimal traversal path of the undirected graph after adding the high-dimensional point, since the distance between the virtual high-dimensional point and all points except the starting and ending points is infinite, the algorithm needs to traverse all path points including the virtual high-dimensional point. The solution result must be an optimal traversal path that includes a route from the starting point to the high-dimensional virtual point and then to the end point. After removing the virtual high-dimensional point from the obtained optimal traversal path, the remaining path is the available hair transplant path.

[0233] In addition, the time complexity of the ant colony algorithm also needs to be considered. For the ant colony algorithm, when each heuristic factor is fixed, the time complexity of a path planning task containing n path points is O(n^4). Hair transplant path planning requires high real-time performance, and the number of path points to be planned in the hair transplant path planning task is often as many as 300-400, and the computational overhead is intolerable. In order to speed up the calculation speed and reduce the calculation time, the hair transplant area needs to be divided into blocks. Simplifying a large overall path optimization task into path optimization tasks within several blocks can greatly speed up the calculation speed and reduce the calculation time.

[0234] Figure 28 The hair transplantation path planning block plan is as follows:

[0235] The tensioner block is divided into six parts, and the hair follicles closest to the corresponding corners are searched in each block as the starting point and the end point. Path planning is performed independently in each block. The six blocks are coordinated to plan the paths as shown in the figure above, which can greatly improve the speed of hair transplant path planning.

[0236] The key to designing an ant colony algorithm is to correctly set three heuristic factors: the heuristic pheromone importance factor α, the heuristic distance importance factor β, and the heuristic pheromone volatility factor ρ. When these factors are not properly set, the adverse effects they produce are summarized as follows:

[0237]

[0238] According to relevant literature, the optimal parameter settings are as follows:

[0239]

[0240] Under the above parameter settings, the ant-week model shows high application value in hair transplant path planning algorithm.

[0241] As shown in 29, Figure 29 Algorithm iteration process record, Figure 30 Schematic diagram of the path planning results demonstration.

[0242] Example 2

[0243] This embodiment of the hair transplant robot comprises a robot body and a controller mounted on the body. The body is equipped with a hair transplant robotic arm, a laser positioning device, a depth camera, a macro camera, and a structural frame. During surgery, the patient wears a tensioner on the back of the head. The tensioner is a rigid frame that tensions the patient's scalp to facilitate insertion of the robotic arm. Furthermore, the tensioner limits the retrieval range; hair removal occurs only on the scalp within the tensioner.

[0244] The controller receives a captured image containing hair follicles, processes the captured image to obtain hair follicle root position information of the target hair follicle image, and generates a hair transplant path based on the hair follicle root position information; the controller is provided with a hair follicle recognition module, a hair follicle assessment and positioning module, and a hair transplant path planning module;

[0245] The hair follicle identification module is used to extract hair follicle images from the original image data stream one by one and pass them to the next module;

[0246] The hair follicle evaluation and positioning module is used to process the hair follicle images extracted by the previous module one by one, evaluate and screen them, and calculate the coordinates of the hair follicle roots as the cutting position;

[0247] The hair transplantation path planning module is used to calculate the hair transplantation path to obtain the shortest traversal path of the hair follicles to be harvested and optimize the hair transplantation speed.

[0248] The hair follicle identification module includes a tensioner identification unit and a hair follicle identification unit; the tensioner identification unit is used to identify the tensioner and correct the viewing angle; the hair follicle identification unit is used to identify the hair follicles and extract them one by one; or

[0249] The hair follicle assessment and positioning module includes a hair follicle assessment unit and a hair follicle root positioning unit; the hair follicle assessment unit is used to screen hair follicles suitable for hair removal based on whether the hair follicles are bifurcated, too thin, multi-clustered, or fuzzy; the hair follicle root positioning unit is used to locate the roots of the screened hair follicles and obtain the entry point of the robotic arm.

[0250] When the hair transplant robot provided in this embodiment is operating, when the system is performing a hair removal operation, the macro camera transmits a data stream of images of the patient's occipital region to the computer. This data stream is then fed into the system's visual algorithm to automatically identify, evaluate, and locate hair follicle images, calculate the entry point, and plan the optimal hair removal path. The depth camera then measures the precise three-dimensional coordinates of the entry point of each hair follicle along the hair removal path and transmits these coordinates to the hair removal robot controller to complete the hair removal.

[0251] The visual part of the algorithm needs to implement the following functions:

[0252] According to the original image information, the hair follicle objects in the image are automatically identified.

[0253] Evaluate the automatically identified hair follicles to select the ones suitable for hair removal

[0254] The filtered hair follicle images are processed to calculate the entry point of the robotic arm.

[0255] Path planning is performed on the hair follicle image for which entry point calculation has been completed to obtain the shortest traversal path.

[0256] in, Figure 31 Schematic diagram of the tensioner. Figure 32 Panoramic view of hair transplant robot, Figure 33 Diagram of the hair transplant robotic arm; the tensioner is a rectangular frame that is attached to the back of the patient's head, tensioning the skin and defining the hair removal area. The four corners of the tensioner are marked with standardized colors to assist the algorithm in positioning. The tensioner recognition algorithm aims to obtain the position of the tensioner plane and transform the hair follicle image in the original image to a normal perspective using a homography matrix for subsequent algorithm recognition.

[0257] In the tensioner recognition algorithm, a fixed threshold of 230 is selected and then binarized; the kernel size is 10*10; contour extraction and contour rectangle fitting are performed to extract the coordinates of each contour center point; the homography transformation matrix is ​​calculated based on the coordinates of each contour center point, and the original image perspective is transformed to the normal perspective.

[0258] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A method for automatic hair follicle identification based on deep learning, characterized by: The following steps are involved: Extracting a hair follicle image in a hair removal area of ​​the acquired image; extracting each target hair follicle image from the hair follicle image; Build a deep learning model and evaluate target hair follicle images; Selecting hair follicle recognition images that meet preset conditions based on the evaluation results; Acquiring hair follicle root position information in the hair follicle image according to the hair follicle identification image; The target hair follicle image is evaluated by screening the hair follicles in the target hair follicle image based on their morphological features, which may include any one or more of the following: bifurcation, excessive thinness, multiplication, and blurring of the hair follicles; specifically, as follows: Perform any one or more of the following image processing on the target hair follicle image: perform hair follicle image clarity assessment, perform hair follicle clustering detection, perform hair follicle hair thinning detection; perform hair follicle bifurcation assessment on the image after the above image processing; or The hair follicle root position information is achieved through hair follicle root positioning processing, and the specific steps are as follows: The image processed by hair follicle bifurcation assessment is input into a two-headed neural network including an FCN structure and a CNN structure. After being processed by the two-headed neural network, the hair follicle root position information is output; The FCN structure in the dual-head neural network adopts a U-Net neural network, and the output data structure is a feature map; The neural network of the CNN structure adopts a depth-separable convolution mechanism, and the output data structure is a feature vector; The dual-headed neural network has two branches that can simultaneously generate a graph output based on a U-Net structure and a feature vector output based on a CNN structure. A judgment program is set at the output of the neural network to evaluate the quality of the two outputs to ultimately determine which output to use as the positioning result. The main basis of the judgment program is: when the graph output has high interpretability, the graph output is given priority as the final result, otherwise the feature vector output is used as the final result. The dual-headed neural network includes an input unit, a backbone network, an auxiliary network, and an output unit; the input unit is configured with a set number of channels and a set size; the backbone network is configured with a set encoding block, a downsampling block, an upsampling block, and then channel merging, and finally a decoding block to obtain an image with a set number of channels and a set size; The auxiliary network receives the minimum size image from the downsampling block of the backbone network, performs maximum pooling and average pooling on the minimum size image through several auxiliary network blocks, and outputs the coordinates of the obtained image through a fully connected layer and a sigmoid function.

2. The deep learning-based automatic hair follicle identification method according to claim 1, wherein: The hair removal area is identified based on a hair removal area positioning device provided on the hair area, and the hair removal area is determined by identifying the hair removal area positioning device.

3. The method for automatic hair follicle identification based on deep learning according to claim 1, wherein: The hair follicle image in the hair removal area is processed by image recognition to obtain a fitting rectangle of each target hair follicle outline.

4. A hair transplant path planning method, characterized by: The following steps are involved: S1: Obtaining hair follicle root location information using the deep learning-based automatic hair follicle recognition method according to any one of claims 1 to 3; S2: Calculate the hair transplantation path based on the hair follicle root position information to obtain the shortest traversal path for the hair follicle to be transplanted.

5. The hair transplant path planning method according to claim 4, wherein: The calculation of the hair transplantation path is solved using the ant-periphery model.

6. Deep learning-based automatic hair follicle recognition system, characterized by: The system includes at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the deep learning-based automatic hair follicle identification method as described in any one of claims 1 to 3.

7. A hair transplant robot implemented using the deep learning-based automatic hair follicle recognition method according to any one of claims 1 to 3, characterized in that: The robot comprises a main body and a controller disposed on the main body; the controller receives a captured image containing hair follicles, processes the captured image to obtain hair follicle root position information of the target hair follicle image, and generates a hair transplant path based on the hair follicle root position information; the controller is provided with a hair follicle recognition module, a hair follicle assessment and positioning module, and a hair transplant path planning module; The hair follicle identification module is used to extract hair follicle images from the original image data stream one by one and pass them to the next module; The hair follicle evaluation and positioning module is used to process the hair follicle images extracted by the previous module one by one, evaluate and screen them, and calculate the coordinates of the hair follicle roots as the cutting position; The hair transplantation path planning module is used to calculate the hair transplantation path to obtain the shortest traversal path for the hair follicles to be harvested.

8. The hair transplant robot according to claim 7, wherein: The hair follicle identification module includes a tensioner identification unit and a hair follicle identification unit; the tensioner identification unit is used to identify the hair removal area positioning device; the hair follicle identification unit is used to identify the hair follicles and extract them one by one; or The hair follicle assessment and positioning module includes a hair follicle assessment unit and a hair follicle root positioning unit; the hair follicle assessment unit is used to screen target hair follicle images that meet the requirements based on the bifurcation, excessive thinness, multiplication, and blurring of the hair follicles; the hair follicle root positioning unit is used to locate the hair follicle roots of the screened target hair follicle images and use them as the entry point for hair transplantation.

Images