Hair transplantation methods, devices, systems, storage media, and computer program products

Abstract

The application relates to a hair planting method, device, system, storage medium and computer program product. The method comprises the following steps: obtaining a stored three-dimensional model and a real-time three-dimensional model of a hair-planting object; obtaining an actual hairline through key point detection based on a stored image; obtaining a hair-planting facial point cloud through three-dimensional reconstruction based on the actual hairline, a stored hairline and the stored image; registering a model facial point cloud obtained based on the stored three-dimensional model and the hair-planting facial point cloud to obtain a first registration matrix; determining a model point cloud of a hair-planting area based on the first registration matrix, the stored three-dimensional model and the hair-planting facial point cloud; the model point cloud of the hair-planting area is used for determining a hair-planting planning path; registering the model point cloud of the hair-planting area and the real-time three-dimensional model to obtain a second registration matrix; and performing hair planting on the hair-planting object according to the second registration matrix and the hair-planting planning path. The method can improve the hair-planting efficiency.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a hair transplantation method, apparatus, system, storage medium, and computer program product. Background Technology

[0002] In existing hair transplant methods, the condition and density distribution of hair follicles are obtained through hair follicle testing before the transplant, and then the hairline, donor area and transplant area are planned. The hair transplant operator extracts and separates the hair follicles in the donor area based on his own experience, and then implants the extracted hair follicles into the transplant area. This method of hair transplantation, which mainly relies on human experience, has the problem of low hair transplantation efficiency. Summary of the Invention

[0003] This application provides a hair transplantation method, apparatus, system, computer-readable storage medium, and computer program product that can improve hair transplantation efficiency, thereby solving the problem of low efficiency in existing hair transplantation methods.

[0004] Firstly, this application provides a hair transplantation method. The method includes:

[0005] The existing 3D model and the real-time 3D model of the object to be transplanted are obtained. The existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted.

[0006] Based on the existing image, the actual hairline is obtained through key point detection; based on the actual hairline, the existing hairline, and the existing image, the point cloud of the hair transplant face is obtained through three-dimensional reconstruction.

[0007] The model facial point cloud obtained by segmenting the existing 3D model is registered with the hair transplant facial point cloud to obtain the first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0008] The point cloud of the area to be transplanted is registered with the real-time 3D model to obtain the second registration matrix; the hair transplant is performed on the subject according to the second registration matrix and the hair transplant planning path.

[0009] Secondly, this application also provides a hair transplant device. The device includes:

[0010] The acquisition module is used to acquire the existing 3D model and the real-time 3D model of the object to be transplanted. The existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted.

[0011] The facial point cloud acquisition module is used to obtain the actual hairline based on the existing image through key point detection; and to obtain the facial point cloud for hair transplantation through three-dimensional reconstruction based on the actual hairline, the existing hairline, and the existing image.

[0012] The model point cloud determination module is used to register the model facial point cloud obtained by segmenting the existing 3D model with the hair transplant facial point cloud to obtain the first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0013] The hair transplant module is used to register the model point cloud of the area to be transplanted with the real-time 3D model to obtain the second registration matrix; and to perform hair transplantation on the object to be transplanted according to the second registration matrix and the hair transplantation planning path.

[0014] Thirdly, this application also provides a hair transplant system. The system includes: a robotic arm comprising multiple joints, with an end effector mounted at the end of the robotic arm; the rotation angles of the multiple joints allow the robotic arm to perform multi-degree-of-freedom movements, and the end effector is used to perform hair transplantation at the desired location; and a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0015] The existing 3D model and the real-time 3D model of the object to be transplanted are obtained. The existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted.

[0016] Based on the existing image, the actual hairline is obtained through key point detection; based on the actual hairline, the existing hairline, and the existing image, the point cloud of the hair transplant face is obtained through three-dimensional reconstruction.

[0017] The model facial point cloud obtained by segmenting the existing 3D model is registered with the hair transplant facial point cloud to obtain the first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0018] The point cloud of the area to be transplanted is registered with the real-time 3D model to obtain the second registration matrix; the hair transplant is performed on the subject according to the second registration matrix and the hair transplant planning path.

[0019] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0020] Obtain the existing 3D model and the real-time 3D model of the object to be transplanted, wherein the existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted;

[0021] Based on the existing image, the actual hairline is obtained through key point detection; based on the actual hairline, the existing hairline, and the existing image, the point cloud of the hair transplant face is obtained through three-dimensional reconstruction.

[0022] The model facial point cloud obtained by segmenting the existing 3D model is registered with the hair transplant facial point cloud to obtain the first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0023] The point cloud of the area to be transplanted is registered with the real-time 3D model to obtain the second registration matrix; the hair transplant is performed on the subject according to the second registration matrix and the hair transplant planning path.

[0024] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0025] The existing 3D model and the real-time 3D model of the object to be transplanted are obtained. The existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted.

[0026] Based on the existing image, the actual hairline is obtained through key point detection; based on the actual hairline, the existing hairline, and the existing image, the point cloud of the hair transplant face is obtained through three-dimensional reconstruction.

[0027] The model facial point cloud obtained by segmenting the existing 3D model is registered with the hair transplant facial point cloud to obtain the first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0028] The point cloud of the area to be transplanted is registered with the real-time 3D model to obtain the second registration matrix; the hair transplant is performed on the subject according to the second registration matrix and the hair transplant planning path.

[0029] The aforementioned hair transplant method, device, system, storage medium, and computer program product acquire both a pre-existing 3D model and a real-time 3D model of the subject to be transplanted. Based on the pre-existing image, key point detection is used to obtain the actual hairline. Based on the actual hairline, the pre-existing hairline, and the pre-existing image, 3D reconstruction is performed to obtain a point cloud of the transplanted face. The point cloud of the model face obtained by segmenting the pre-existing 3D model is registered with the point cloud of the transplanted face to obtain a first registration matrix. Based on the first registration matrix, the pre-existing 3D model, and the point cloud of the transplanted face, the point cloud of the model of the area to be transplanted is determined. This method involves segmenting the pre-existing 3D model of the subject to be transplanted... The proposed method for registering facial point clouds of the target area with those of the hair transplantation model can determine the model point cloud of the hair transplantation area on an existing 3D model through facial point cloud registration. Based on the model point cloud of the target area, a second registration matrix is ​​obtained by registering it with a real-time 3D model. According to the second registration matrix and the hair transplantation planning path, hair transplantation is performed on the target object. The hair transplantation planning area is determined by the model point cloud of the target area. By registering the model point cloud of the target area with the real-time 3D model, the hair transplantation planning path can be mapped onto the real-time 3D model, thereby performing hair transplantation on the target object according to the hair transplantation planning path, improving hair transplantation efficiency. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating a hair transplantation method in one embodiment;

[0031] Figure 2 This is a schematic diagram of a sub-process of an embodiment S104;

[0032] Figure 3 This is a schematic diagram of a method for obtaining a target facial image in one embodiment;

[0033] Figure 4 This is a schematic diagram of a sub-process of another embodiment S104;

[0034] Figure 5 This is a schematic diagram of a method for obtaining two-dimensional hair transplant images in one embodiment;

[0035] Figure 6 This is a schematic diagram of a sub-process of S402 in one embodiment;

[0036] Figure 7 This is a schematic diagram of a sub-process of S106 in one embodiment;

[0037] Figure 8 This is a schematic diagram of a sub-process of S108 in one embodiment;

[0038] Figure 9 This is a flowchart illustrating a hair transplant method in another embodiment;

[0039] Figure 10This is a schematic diagram of the composition of a hair transplant system in one embodiment;

[0040] Figure 11 This is a schematic diagram of the send / fetch actuator performing send / fetch at the location to be sent / fetched in one embodiment;

[0041] Figure 12 This is a schematic diagram of a hair transplant actuator performing hair transplantation at the location to be transplanted, as shown in one embodiment.

[0042] Figure 13 This is a schematic diagram of the overall process of a hair transplantation method in one embodiment;

[0043] Figure 14 This is a flowchart illustrating the process of obtaining a stored 3D model in one embodiment;

[0044] Figure 15 This is a schematic diagram of the process flow before hair transplantation in one embodiment;

[0045] Figure 16 This is a schematic diagram of a method for obtaining a pre-existing hairline in one embodiment;

[0046] Figure 17 This is a schematic diagram of the process of obtaining the model point cloud of the area to be transplanted in one embodiment;

[0047] Figure 18 This is a schematic diagram of point cloud reconstruction of the back of the head of a subject undergoing hair transplantation in one embodiment.

[0048] Figure 19 This is a schematic diagram illustrating the determination of a hair transplant planning path in one embodiment;

[0049] Figure 20 This is a schematic diagram of real-time 3D model reconstruction in one embodiment;

[0050] Figure 21 This is a schematic diagram of the hair transplant process in one embodiment;

[0051] Figure 22 This is a schematic diagram of a robotic arm implanting hair follicles in one embodiment;

[0052] Figure 23 This is a schematic diagram of the process for generating a 3D hairstyle model in one embodiment;

[0053] Figure 24 This is a structural block diagram of a hair transplant device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The hair transplant method provided in this application embodiment can be applied to a hair transplant system. The system includes a robotic arm and a computer device. The robotic arm includes multiple joints, and an end effector is installed at the end of the robotic arm. The multiple joints allow the robotic arm to perform multi-degree-of-freedom movements, and the end effector is used to perform hair transplantation at the desired location. The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the hair transplant method. The computer device can be either a terminal or a server. The hair transplant method provided in this application embodiment can be executed by the terminal or the server alone, or by the terminal and the server collaboratively. The terminal communicates with the server via a network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The computer equipment acquires both a pre-existing 3D model and a real-time 3D model of the subject to be transplanted. The pre-existing 3D model is obtained through 3D reconstruction based on existing images of the subject. Based on the pre-existing images, the actual hairline is obtained through key point detection. Based on the actual hairline, the pre-existing hairline, and the pre-existing images, a point cloud of the face to be transplanted is obtained through 3D reconstruction. The point cloud of the face to be transplanted is registered with the point cloud of the face to be transplanted to obtain a first registration matrix. Based on the first registration matrix, the pre-existing 3D model, and the point cloud of the face to be transplanted, the point cloud of the area to be transplanted is determined. The point cloud of the area to be transplanted is used to determine the hair transplant planning path. The point cloud of the area to be transplanted is registered with the real-time 3D model to obtain a second registration matrix. Based on the second registration matrix and the hair transplant planning path, hair transplantation is performed on the subject. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Servers can be implemented using standalone servers or server clusters consisting of multiple servers.

[0056] In one embodiment, such as Figure 1 As shown, a hair transplant method is provided. Taking the application of this method to a computer device as an example, the method includes the following steps:

[0057] S102, obtain the existing 3D model and the real-time 3D model of the object to be transplanted. The existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted.

[0058] The computer equipment acquires both stored and real-time images of the recipient of hair transplantation through image acquisition devices. The stored images are acquired before the transplantation procedure, while the real-time images are acquired during the transplantation process.

[0059] The computer equipment uses existing images of the hair transplant recipient to reconstruct a 3D model of the recipient. Specifically, an image acquisition device captures multiple images of the recipient from various angles. The computer acquires these images and extracts feature points from each. Feature point matching is performed between adjacent images to obtain a sparse spatial point cloud. The computer acquires the intrinsic parameter matrix of the image acquisition device and performs stereo correction on adjacent images to satisfy epipolar constraints. Under these constraints, feature matching is used to find corresponding points between adjacent images, resulting in a dense spatial point cloud. The dense point clouds from different viewpoints are registered and stitched together to obtain a denser spatial point cloud. The computer performs Poisson reconstruction on the dense point cloud to obtain a triangular mesh model of the recipient. Texture mapping is then applied to this triangular mesh model to obtain the existing 3D model of the recipient.

[0060] The method of obtaining a stored 3D model from an existing image through 3D reconstruction is adopted. The computer equipment obtains a real-time 3D model of the object to be transplanted through 3D reconstruction from real-time images.

[0061] S104: Based on the existing image, the actual hairline is obtained through key point detection; based on the actual hairline, the existing hairline, and the existing image, the point cloud of the hair transplanted face is obtained through three-dimensional reconstruction.

[0062] The purpose of keypoint detection is to identify key points of the hairline in existing images. The computer device performs keypoint detection on the existing images to obtain the actual hairline of the recipient. The existing hairline refers to the hairline generated through deep learning methods before the hair transplant is performed. In other words, the existing hairline is generated by a deep learning model. The computer device inputs the existing images of the recipient into the hairline recognition model to obtain the recipient's existing hairline. The training method for the hairline recognition model includes: acquiring multiple existing images of the recipient; inputting these images into the deep learning model; using the existing images with drawn hairlines as labels for model training; adjusting model parameters; and stopping model training when a preset stopping condition is met, thus obtaining the hairline recognition model.

[0063] The computer device uses the area bounded by the actual hairline and the existing hairline as the two-dimensional hair transplant area. This two-dimensional hair transplant area is the region to be transplanted in the existing image. The computer device then performs three-dimensional reconstruction on the existing image of the hair transplant area to obtain a point cloud of the face for hair transplantation. This point cloud includes the region to be transplanted.

[0064] S106, the model facial point cloud obtained by segmentation based on the existing 3D model is registered with the hair transplant facial point cloud to obtain the first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0065] The process involves using computer equipment to segment an existing 3D model to obtain a facial point cloud. Point cloud segmentation algorithms can be employed. These algorithms divide the point cloud model based on spatial, geometric, and textural feature points. The computer equipment segments the existing 3D model using facial feature points to obtain the facial point cloud. The computer equipment then registers the facial point cloud obtained from the segmentation with the hair transplant facial point cloud to obtain a first registration matrix. This first registration matrix represents the transformation relationship between the facial point cloud of the existing 3D model and the hair transplant facial point cloud.

[0066] The computer equipment multiplies the facial point cloud for hair transplantation by a first registration matrix to obtain a transformed facial point cloud. This transformed point cloud includes the area to be transplanted, thus mapping the area to be transplanted into an existing 3D model, resulting in a model point cloud of the area to be transplanted. Based on this model point cloud, the computer equipment determines the planned hair transplantation path.

[0067] S108: Based on the model point cloud of the area to be transplanted, the model is registered with the real-time 3D model to obtain the second registration matrix; according to the second registration matrix and the hair transplant planning path, the hair transplant is performed on the object to be transplanted.

[0068] The computer equipment registers the point cloud of the area to be transplanted with the real-time 3D model to obtain a second registration matrix. This second registration matrix represents the transformation relationship between the point cloud of the area to be transplanted and the real-time 3D model. The computer equipment then multiplies the hair transplant planning path by the second registration matrix to obtain the transformed hair transplant planning path. This maps the hair transplant planning path to the real-time 3D model, and the computer equipment performs hair transplantation on the recipient according to the transformed hair transplant planning path.

[0069] In the aforementioned hair transplantation method, an existing 3D model and a real-time 3D model of the subject to be transplanted are acquired. Based on the existing image, the actual hairline is obtained through key point detection. Based on the actual hairline, the existing hairline, and the existing image, a 3D reconstruction is performed to obtain a point cloud of the transplanted face. The point cloud of the model face obtained by segmenting from the existing 3D model is registered with the point cloud of the transplanted face to obtain a first registration matrix. Based on the first registration matrix, the existing 3D model, and the point cloud of the transplanted face, the point cloud of the model of the area to be transplanted is determined. This method involves segmenting the point cloud of the model face from the existing 3D model of the subject to be transplanted and then registering it with the point cloud of the transplanted face to obtain a first registration matrix. The facial point cloud registration method can determine the model point cloud of the area to be transplanted on an existing 3D model through facial point cloud registration. Based on the model point cloud of the area to be transplanted, a second registration matrix is ​​obtained by registering it with the real-time 3D model. According to the second registration matrix and the hair transplant planning path, hair transplantation is performed on the subject. The hair transplant planning area is determined by the model point cloud of the area to be transplanted. By registering the model point cloud of the area to be transplanted with the real-time 3D model, the hair transplant planning path can be mapped into the real-time 3D model, thereby performing hair transplantation on the subject according to the hair transplant planning path, improving hair transplantation efficiency.

[0070] In one embodiment, such as Figure 2 As shown, based on the existing image, the actual hairline is obtained through key point detection, including:

[0071] S202, the stored image is classified using the first classification model to obtain the target facial image.

[0072] The first classification model is a deep learning classification model. Since the stored images were acquired from multiple angles of the subject to be transplanted, they include images of the subject from multiple angles. The computer equipment classifies the stored images using the first classification model to obtain the target facial image. The target facial image is the image representing the face of the subject to be transplanted from the stored images. The training steps of the first classification model include: the computer equipment acquires sample images of the subject from multiple angles, and uses a sample image of the subject's face viewed directly by the image acquisition device as a reference image. The sample images are input into the deep learning model for model training, and the reference image is used as the label for model training. Model parameters are adjusted until a preset stopping condition is met, at which point model training stops, resulting in the first classification model. Figure 3 The diagram shows a method for obtaining a target facial image.

[0073] S204, perform key point detection on the target facial image to obtain the actual hairline.

[0074] The computer equipment uses a key point detection algorithm to detect key points in the target facial image, identifying the hairline key points. These hairline key points are then fitted into a curve to obtain the actual hairline of the recipient.

[0075] In this embodiment, the existing images are classified using a first classification model to obtain the target facial image. The actual hairline in the target facial image is obtained through key point detection. This classification method using the first classification model can improve the recognition efficiency of the target facial image. Key point detection can quickly and accurately identify the actual hairline of the subject to be transplanted, which is beneficial to improving the efficiency of hair transplantation.

[0076] In one embodiment, such as Figure 4 As shown, based on the actual hairline, existing hairline data, and existing images, a point cloud of the face used for hair transplantation is obtained through 3D reconstruction, including:

[0077] S402, based on the actual hairline and the existing hairline, uses a segmentation model to segment the target facial image to obtain a two-dimensional hair transplant image.

[0078] The area enclosed by the actual hairline and the existing hairline is the region to be transplanted. The segmentation model is a deep learning model; for example, it can be a U-net (U-shaped network structure) model. Sample images are input into the deep learning model for training. The binary image of the region to be transplanted, composed of the actual hairline and the existing hairline, is used as a label to adjust the model parameters, resulting in the trained deep learning model, which is the segmentation model. The computer device inputs the target facial image into the segmentation model to obtain a two-dimensional hair transplant image, which is a binary image containing the region to be transplanted. For example... Figure 5 The diagram shows a method for obtaining two-dimensional hair transplant images.

[0079] S404, based on two-dimensional hair transplant images and target facial images, obtains the hair transplant facial point cloud through three-dimensional reconstruction.

[0080] The computer equipment identifies the area to be transplanted in the target facial image based on a two-dimensional hair transplant image, thus obtaining a target facial image of the area to be transplanted. The computer equipment then performs three-dimensional reconstruction on the target facial image of the area to be transplanted, obtaining a point cloud of the hair transplant face. This point cloud includes the area to be transplanted.

[0081] In this embodiment, the target facial image is segmented using a segmentation model based on the actual hairline and the existing hairline to obtain a two-dimensional hair transplant image. Then, a three-dimensional reconstruction is performed to obtain a hair transplant facial point cloud. This method of segmenting the area to be transplanted using a segmentation model can improve the segmentation efficiency of the area to be transplanted. The hair transplant facial point cloud obtained through three-dimensional reconstruction includes the area to be transplanted, which is beneficial to improving the hair transplant efficiency.

[0082] In one embodiment, such as Figure 6 As shown, based on the actual hairline and the existing hairline, a segmentation model is used to segment the target facial image to obtain a two-dimensional hair transplant image, including:

[0083] S602, determine whether the actual hairline consists of only one curve.

[0084] In some embodiments, the actual hairline in the target facial image includes only one line. In other embodiments, the actual hairline is discontinuous; for example, only part of the hairline extends from the face to the top of the head, with the remaining part at the back of the head. The actual hairline detected from the target facial image via key point detection includes at least two lines. A computer device determines whether the actual hairline consists of only one curve.

[0085] S604, when the actual hairline includes at least two curves, connect the different curves of the at least two curves end to end in sequence to obtain the connected hairline; based on the connected hairline and the existing hairline, use a segmentation model to segment the target facial image to obtain a two-dimensional hair transplant image.

[0086] The actual hairline consists of at least two curves. This means that the actual hairline detected in the target facial image is discontinuous. The computer device sequentially connects the different curves from these two curves end-to-end to obtain a connected hairline. Sequential connection means that if the actual hairline in the target facial image is not connected at the top of the head, at least two lines at the top of the head are connected sequentially to fit a single, connected hairline.

[0087] The computer device segments the target facial image using a segmentation model based on the concatenated hairline and the existing hairline, obtaining a two-dimensional hair transplant image. Specifically, the computer device uses the region enclosed by the concatenated hairline and the existing hairline as the area to be transplanted. The segmentation model is a deep learning model; for example, it can be the Unet model. Sample images are input into the deep learning model for training. The binary image of the area to be transplanted, composed of the concatenated hairline and the existing hairline, is used as a label to adjust the model parameters, resulting in the trained deep learning model, which is the segmentation model. The computer device inputs the target facial image into the segmentation model to obtain a two-dimensional hair transplant image, which is a binary image containing the area to be transplanted.

[0088] S606, when the actual hairline consists of only one curve, a segmentation model is used to segment the target facial image based on the actual hairline and the existing hairline to obtain a two-dimensional hair transplant image.

[0089] The actual hairline consists of only one curve, meaning the detected actual hairline in the target facial image is unbroken. The computer equipment uses a segmentation model to segment the target facial image based on the actual hairline and the existing hairline, obtaining a two-dimensional hair transplant image. The area enclosed by the actual hairline and the existing hairline is designated as the area to be transplanted. A segmentation model is trained using deep learning to segment the target facial image, resulting in a two-dimensional hair transplant image. The two-dimensional hair transplant image is a binary image containing the area to be transplanted.

[0090] In this embodiment, by determining whether the actual hairline includes only one curve, if it includes at least two curves, the disconnected actual hairlines are connected sequentially, and the connected hairline and the existing hairline are used as the area to be transplanted. If it includes only one curve, the actual hairline and the existing hairline are directly used as the area to be transplanted, which can obtain a more accurate area to be transplanted. The two-dimensional hair transplant image is obtained by using the segmentation model method, which is beneficial to improving the efficiency of hair transplantation.

[0091] In one embodiment, such as Figure 7 As shown, based on the first registration matrix, the existing 3D model, and the point cloud of the face to be transplanted, the model point cloud of the area to be transplanted is determined, including:

[0092] S702, based on the first registration matrix, the area to be transplanted in the hair transplant facial point cloud is transformed to obtain the transformed hair transplant area point cloud.

[0093] The computer equipment transforms the area to be transplanted in the facial point cloud based on the first registration matrix, obtaining a transformed hair transplant area point cloud. Specifically, the coordinates of the area to be transplanted in the facial point cloud are multiplied by the first registration matrix to obtain the transformed point cloud coordinates. The position points corresponding to the transformed point cloud coordinates constitute the transformed hair transplant area point cloud.

[0094] S704: Based on the converted point cloud of the hair transplant area and the existing 3D model, determine the model point cloud of the area to be transplanted.

[0095] In this process, the converted hair transplant area point cloud and the point cloud corresponding to the existing 3D model are located in the same coordinate space. The computer device uses the point cloud formed by the converted hair transplant area point cloud and the point cloud corresponding to the existing 3D model as the model point cloud of the area to be transplanted.

[0096] In this embodiment, the area to be transplanted in the hair transplant face point cloud is transformed by the first registration matrix to obtain the transformed hair transplant area point cloud. Based on the transformed hair transplant area point cloud and the existing 3D model, the area to be transplanted in the hair transplant face point cloud can be mapped to the existing 3D model, thereby obtaining the model point cloud of the area to be transplanted. Hair transplantation based on the model point cloud of the area to be transplanted is beneficial to improving hair transplantation efficiency.

[0097] In one embodiment, such as Figure 8 As shown, based on the second registration matrix and the hair transplant planning path, hair transplantation is performed on the recipient, including:

[0098] S802, based on the second registration matrix, transform the hair transplant planning path to obtain the transformed hair transplant path.

[0099] In this process, the computer equipment multiplies each location point in the hair transplant planning path by the second registration matrix to obtain the transformed location points, and determines the transformed hair transplant path composed of the transformed location points.

[0100] S804, based on the converted hair transplant path, determines each hair transplant location and the order of each hair transplant location in the area to be transplanted.

[0101] In this process, each point in the converted hair transplant path is a hair transplant position in the area to be transplanted. The order of each hair transplant position in the area to be transplanted is determined according to the arrangement of each point in the converted hair transplant path.

[0102] S806 determines the rotation angles of multiple joints of the robotic arm for any given hair transplant location, controls the movement of the robotic arm based on the rotation angles of the multiple joints, and controls the end effector of the robotic arm to perform hair transplantation at the location to be transplanted.

[0103] The robotic arm's end effector performs hair transplantation at various locations within the transplantation area. Each location within the transplantation area is the position the end effector needs to reach. For any given location, inverse kinematics is used to obtain the rotation angles of multiple joints in the robotic arm. Each joint of the robotic arm moves according to its corresponding rotation angle, thereby controlling the end effector to move to the desired location for hair transplantation.

[0104] In this embodiment, the hair transplant planning path is transformed using a second registration matrix to obtain the transformed hair transplant path. This transformed hair transplant planning path is then mapped onto a real-time 3D model. Based on the various hair transplant positions and their order in the transformed hair transplant path, the end effector of the robotic arm is controlled to move to each hair transplant position to perform the transplant. This mapping of the pre-transplant planning path to the real-time hair transplant scene facilitates hair transplantation according to the pre-transplant planning effect, thereby improving hair transplantation efficiency.

[0105] In one embodiment, the step of determining the hair transplant planning path includes: dividing the hair transplant area in the model point cloud of the area to be transplanted into at least one block; determining the hair transplant path composed of each hair transplant position in each block; and connecting the hair transplant paths corresponding to each block in sequence according to the order of each block to obtain the hair transplant planning path.

[0106] The computer equipment divides the hair transplant area in the model point cloud into at least one block, and determines the hair transplant path composed of each hair transplant position in each block. Following the order of the blocks, the corresponding hair transplant paths for each block are connected sequentially. Specifically, according to the order of the blocks, the next path planning point after the last planned path point in the current block becomes the first path planning point in the next block, until the path planning for all blocks is completed, resulting in the planned hair transplant path.

[0107] In this embodiment, dividing the area to be transplanted into blocks and using the block-based path planning method helps to improve the efficiency of path planning.

[0108] In one embodiment, before performing hair transplantation on the subject based on the second registration matrix and the hair transplantation planning path, the method further includes: acquiring a real-time 3D model of the subject to be transplanted; identifying hair follicle points in existing images and determining the area to be transplanted in existing images through density clustering; obtaining a posterior head hair transplantation point cloud through 3D reconstruction based on the area to be transplanted in existing images and existing images; registering the posterior head point cloud obtained by segmenting the existing 3D model with the posterior head hair transplantation point cloud to obtain a third registration matrix; and performing hair transplantation based on the third registration matrix and the existing 3D model. In addition, a point cloud of hair extraction is generated from the back of the head to determine the model point cloud of the area to be extracted; the model point cloud of the area to be extracted is used to determine the hair extraction planning path; the model point cloud of the area to be extracted is registered with the real-time hair extraction 3D model to obtain the fourth registration matrix; according to the fourth registration matrix and the hair extraction planning path, hair is extracted from the object to be extracted, and the extracted hair follicles are obtained; correspondingly, according to the second registration matrix and the hair transplant planning path, hair transplantation is performed on the object to be transplanted, including: according to the second registration matrix and the hair transplant planning path, the extracted hair follicles are used to transplant hair onto the object to be transplanted.

[0109] Before performing hair transplantation, hair follicles need to be extracted from the recipient. These extracted follicles are then used for the transplantation. Specifically, the follicle extraction method involves a computer acquiring a real-time 3D model of the recipient. This real-time 3D model is the actual 3D model of the recipient during the extraction process. The computer uses image recognition algorithms to identify hair follicle points in the stored image. Density clustering is then used to determine the regions in the stored image to be extracted from the follicle points. Density clustering groups the hair follicle points in the stored image according to their density, and the regions with higher density in the clustering results are selected as the areas to be extracted.

[0110] The computer device performs 3D reconstruction on an existing image containing the region to be captured, obtaining a point cloud for capturing the back of the head. This point cloud includes the region to be captured. Based on the feature points of the back of the head in the existing 3D model, the computer device segments the existing 3D model, obtaining a point cloud for capturing the back of the head. The segmentation method can employ a point cloud segmentation algorithm. The computer device then registers the point cloud for capturing the back of the head obtained from the segmentation of the existing 3D model with the point cloud for capturing the back of the head, obtaining a third registration matrix. This third registration matrix characterizes the transformation relationship between the point cloud for capturing the back of the head and the point cloud for capturing the back of the head.

[0111] The computer device multiplies the point cloud of the back of the head by a third registration matrix to obtain a transformed point cloud of the back of the head. This transformed point cloud includes the region to be retrieved, thus mapping the region to be retrieved into the existing 3D model, resulting in a model point cloud of the region to be retrieved. The computer device then determines the planned retrieval path from the model point cloud of the region to be retrieved.

[0112] The computer equipment registers the point cloud of the region to be extracted with the real-time 3D model, obtaining a fourth registration matrix. The extraction planning path is then multiplied by this fourth registration matrix to obtain a transformed extraction planning path, thus mapping the extraction planning path to the real-time 3D model. The computer equipment then extracts hair follicles from the target object according to the transformed extraction planning path.

[0113] Accordingly, the computer equipment uses the extracted hair follicles to perform hair transplantation on the recipient based on the second registration matrix and the hair transplantation planning path.

[0114] In this embodiment, the method of registering the point cloud of the posterior head of the existing 3D model of the object to be transplanted with the point cloud of the hair extraction follicle can determine the model point cloud of the hair extraction area on the existing 3D model through the registration of the posterior head point cloud. The hair extraction planning path is determined by the model point cloud of the hair extraction area. By registering the model point cloud of the hair extraction area with the real-time hair extraction 3D model, the hair extraction planning path can be mapped to the real-time hair extraction 3D model, thereby performing hair extraction on the object to be transplanted according to the hair extraction planning path, which improves the hair extraction efficiency and facilitates the rapid extraction of hair follicles. Using the extracted hair follicles to transplant the object to be transplanted is beneficial to improving the hair transplant efficiency.

[0115] In one embodiment, determining the hair follicle region to be extracted in an existing image by density clustering includes: clustering hair follicle points in the existing image according to density to obtain multiple clustered regions; and determining the hair follicle region to be extracted in the existing image based on the hair follicle point density of each clustered region.

[0116] The computer equipment clusters hair follicles according to their density in the stored image, resulting in multiple clustered regions. The hair follicle density of each clustered region is then obtained, and the region to be extracted from the stored image is determined based on the density of the hair follicles in each clustered region.

[0117] In this embodiment, the clustering method based on the density of hair follicle points in the existing image can determine the density of hair follicle point distribution in the existing image. The hair follicle point density of each cluster region is used to determine the hair extraction area, which is beneficial for extracting hair in areas with high hair follicle density and improving hair extraction efficiency.

[0118] In one embodiment, such as Figure 9 As shown, based on the hair follicle density of each cluster region, the region to be extracted from the stored image is determined, including:

[0119] S902, the region with the highest hair follicle density in each cluster region is taken as the target hair extraction region.

[0120] The computer equipment identifies the region with the highest hair follicle density within each cluster and uses this region as the target region. This facilitates hair follicle retrieval in areas with high density, improving retrieval efficiency.

[0121] S904, obtain the number of hair follicles in the target hair extraction area.

[0122] Among them, the computer equipment obtains the number of hair follicles in the target hair extraction area.

[0123] S906, if the number of hair follicle points is less than the preset extraction quantity, for the remaining hair follicle points in the stored image after removing the target extraction region, the density of the remaining hair follicle points is re-clustered to obtain an updated clustered region; based on the hair follicle point density of the updated clustered region, the target extraction region is re-determined, and the step of obtaining the number of hair follicle points in the target extraction region is returned to continue execution until the number of hair follicle points in the target extraction region is not less than the preset extraction quantity, and the target extraction region is used as the extraction region in the stored image.

[0124] If the number of hair follicles is less than the preset extraction count, it indicates that the number of hair follicles in the target area cannot meet the extraction count requirement. For the remaining hair follicles in the stored image after removing the target extraction area, the computer device re-clusters the remaining hair follicles by density, obtaining an updated clustered region. Based on the hair follicle density of the updated clustered region, the target extraction area is redefined. The redefined target extraction area is the region with the highest hair follicle density in the updated clustered region. Based on the redefined target extraction area, the step of obtaining the number of hair follicles in the target extraction area is returned and execution continues until the number of hair follicles in the target extraction area is not less than the preset extraction count. That is, the number of hair follicles in the target extraction area can meet the extraction count requirement. The computer device uses the target extraction area as the extraction area in the stored image.

[0125] In this embodiment, the region with the highest hair follicle density in the clustered region is taken as the target extraction region. When the target extraction region does not meet the extraction quantity requirement, the remaining hair follicles are used to continue density clustering and update the target extraction region until the number of hair follicles in all target extraction regions is not less than the preset extraction quantity. This determines the extraction region in the stored image, which can use the region with a high hair follicle density in the stored image as the extraction region, which is beneficial to improving extraction efficiency.

[0126] In one embodiment, obtaining a head retrieval point cloud by three-dimensional reconstruction based on the retrieval region in an existing image and the existing image includes: classifying the existing image using a second classification model to obtain a target head retrieval image; segmenting the target head retrieval image using a segmentation model based on the retrieval region in the target head retrieval image to obtain a two-dimensional retrieval image; and obtaining a head retrieval point cloud by three-dimensional reconstruction based on the two-dimensional retrieval image and the target head retrieval image.

[0127] The second classification model is a deep learning classification model. The stored images include images of the subject to be transplanted from multiple angles. The computer device classifies these images using the second classification model to obtain the target posterior head image. The target posterior head image is the image representing the posterior head of the subject from the stored images. The training steps of the second classification model include: the computer device acquiring sample images of the subject from multiple angles, using a sample image of the subject facing the image acquisition device at the posterior head as a reference image, inputting the sample images into the deep learning model for training, using the reference image as the model's training label, adjusting the model parameters until a preset stopping condition is met, and then stopping the model training to obtain the second classification model.

[0128] The computer equipment inputs the target head rear image, which identifies the region to be captured, into the segmentation model to obtain a two-dimensional capture image. The two-dimensional capture image is a binary image containing the region to be captured. The computer equipment then performs three-dimensional reconstruction using the two-dimensional capture image and the target head rear image to obtain a head rear capture point cloud.

[0129] In this embodiment, the method of segmenting the region to be retrieved using a segmentation model can improve the segmentation efficiency of the region to be retrieved. Through three-dimensional reconstruction, the retrieval point cloud of the head includes the region to be retrieved, which is beneficial to improving the retrieval efficiency.

[0130] In one embodiment, the hair transplantation method further includes: determining a desired hairstyle image; generating a three-dimensional hairstyle model using a deep learning model based on a two-dimensional hair transplantation image, a two-dimensional hair extraction image, and the desired hairstyle image; and adjusting the three-dimensional hairstyle model based on the model point cloud of the area to be extracted and the model point cloud of the area to be transplanted, to obtain an adjusted hairstyle model.

[0131] The process involves a computer device acquiring an image of the desired hairstyle for the recipient of hair transplantation. The two-dimensional (2D) transplantation image, the 2D extraction image, and the desired hairstyle image are input into a deep learning model to generate a 2D hairstyle image. This 2D hairstyle image is then processed by an encoder-decoder deep learning model to obtain a 3D hairstyle model. This 3D hairstyle model is used to demonstrate the effects after extraction and transplantation. The computer device can also adjust the 3D hairstyle model based on the point clouds of the extraction and transplantation areas to obtain an adjusted hairstyle model.

[0132] In this embodiment, the effects of hair extraction and hair transplantation are displayed through a 3D hairstyle model using a deep learning model. This helps to predict the hairstyle effect before performing hair extraction and hair transplantation operations, facilitates the adjustment of the hair extraction and hair transplantation path, and improves the efficiency of hair extraction and hair transplantation.

[0133] To illustrate the hair transplantation method and its effects in this solution in detail, the following is a detailed example:

[0134] In scenarios where a robotic arm performs hair transplants, a hair transplant system comprises a robotic arm and computer equipment. For example... Figure 10 The diagram shows the structural composition of a hair transplant system. The computer equipment can be located within the robot's base. The robotic arm includes multiple joints, and an end effector is mounted at its end. The rotation angles of the multiple joints allow the robotic arm to perform multi-degree-of-freedom movements. The end effector includes a hair-retrieving actuator and a hair-transplanting actuator. The hair-retrieving actuator is used to retrieve hair from the desired location, and the hair-transplanting actuator is used to transplant hair to the desired location. Figure 11 The diagram shows the actuator performing the fetching and sending operation at the desired location. Figure 12The diagram illustrates a hair transplant actuator performing a hair transplant at the desired location. A computer device, including a memory and a processor, is used. The memory stores a computer program, and the processor executes the program to implement the hair transplant method. A display device is used to display the area to be harvested and the area to be transplanted in real time. Figure 13 The diagram illustrates the overall process of hair transplantation. Computer equipment acquires real-time images from multiple angles surrounding the recipient using image acquisition devices, automatically plans the transplantation path beforehand, and visualizes the post-transplantation effect. It determines whether path planning needs adjustment; if so, the adjustment steps are executed; otherwise, the planned transplantation path is determined. The existing 3D model before transplantation is registered with the real-time 3D model during the transplantation process, allowing the hair transplantation operation to proceed according to the planned path. For any transplantation moment *ti* during the process, if the transplantation operation is complete, the real-time 3D model at moment *ti* is reconstructed. This model is then registered with the real-time 3D model mapped to the planned transplantation path, and the transplantation operation continues according to the registered path until the transplantation is complete.

[0135] Before the robotic arm performs the hair transplant procedure on the recipient, the computer system acquires a pre-existing 3D model of the recipient. For example... Figure 14 The diagram shows a flowchart for obtaining a stored 3D model. Figure 15 The diagram illustrates the procedure before the hair transplant surgery. Computer equipment acquires multiple pre-existing images of the recipient from various angles. These images are then used for 3D reconstruction to create a 3D model. The optimal facial and posterior head images are automatically identified, and a 3D point cloud of the transplant area is automatically reconstructed. The 3D point cloud of hair follicles is then automatically reconstructed, and the density distribution of the entire point cloud is analyzed. Pre-transplant path planning is automatically performed, including hairline design, hair follicle density in the transplant area, and the transplant sequence. The post-transplant effect is displayed to monitor the results and allow for adjustments to the planned path to achieve satisfactory results.

[0136] Specifically, the computer device classifies the existing image using a first classification model to obtain the target facial image. Key point detection is then performed on the target facial image to obtain the actual hairline. Based on the actual hairline and the existing hairline, the computer device uses a segmentation model to segment the target facial image, obtaining a two-dimensional hair transplant image. For example... Figure 16 The diagram illustrates a method for obtaining pre-existing hairline data. A computer inputs multiple target facial images of subjects to be transplanted into a deep learning model. Multiple target facial images with pre-drawn hairlines are used as labels for model training. Model parameters are adjusted until a preset stopping condition is met, at which point model training stops, resulting in a hairline recognition model. The target facial image of the subject to be transplanted is then input into the hairline recognition model to obtain a facial image with pre-existing hairlines.

[0137] like Figure 17 The diagram illustrates the process of obtaining the point cloud model of the area to be transplanted. The computer determines whether the actual hairline consists of only one curve. If the actual hairline consists of at least two curves, the different curves from these two curves are connected end-to-end to obtain the connected hairline. Based on the connected hairline and the existing hairline, a segmentation model is used to segment the target facial image, resulting in a two-dimensional hair transplant image. If the actual hairline consists of only one curve, the segmentation model is used to segment the target facial image based on the actual hairline and the existing hairline, resulting in a two-dimensional hair transplant image. The computer then uses the two-dimensional hair transplant image and the target facial image to perform three-dimensional reconstruction, obtaining the point cloud of the transplanted face.

[0138] In some embodiments, hair transplantation can also be performed on the back of the head of the recipient. For example... Figure 18 The diagram illustrates point cloud reconstruction of the posterior scalp of a candidate for hair transplantation. The computer system detects the actual hairline in the selected posterior scalp image. If the actual hairline exists and is discontinuous, it is connected sequentially to form a closed curve. The area enclosed by this closed curve is designated as the transplantation area. If the actual hairline is absent, hair follicles in the posterior scalp are detected, and the extraction area is automatically planned. The health status of the hair follicles in the extraction area is assessed; those that are too fine are not extracted, thus identifying the extraction areas with identified follicles. The point cloud of the posterior scalp, including the transplantation and extraction areas, is reconstructed. Point cloud segmentation yields the point cloud of the posterior scalp of the existing 3D model. Point cloud registration is performed between the identified point clouds of the transplantation and extraction areas and the existing 3D model's point cloud, resulting in a complete model point cloud of the transplantation and extraction areas. If the area to be transplanted at the back of the head exists, the hair follicle density and extraction order of the area to be transplanted will be planned. If it does not exist, the total number of hair follicles to be extracted, the density of remaining hair follicles, and the extraction order of the hair follicles will be automatically planned.

[0139] The computer equipment registers the facial point cloud obtained from segmenting an existing 3D model with the hair transplant facial point cloud, resulting in a first registration matrix. Based on this first registration matrix, the area to be transplanted in the hair transplant facial point cloud is transformed, yielding a transformed hair transplant area point cloud. Using this transformed point cloud and the existing 3D model, the model point cloud of the area to be transplanted is determined. This model point cloud of the area to be transplanted is used to determine the hair transplant planning path. Figure 19 The diagram shown illustrates the process of determining the hair transplant planning path.

[0140] During the hair transplant procedure performed by the robotic arm, a computer acquires a real-time 3D model of the recipient. For example... Figure 20 The diagram shows a real-time 3D model reconstruction. Based on the parameters of the image acquisition device, the following projection matrix Q is obtained.

[0141]

[0142] Among them, (C) x C y ) represents the coordinates of the principal point on the left view, T is the baseline, f is the focal length, and C is the focal length. * x It is the x-coordinate of the principal point on the right view. If the principal rays intersect at infinity, C x =C * x Given P(x,y) and disparity d, we have:

[0143]

[0144] Then the three-dimensional coordinates of point P in the world coordinate system of the image acquisition device of the first image acquisition device are (X... * ,Y * Z * = (X / W, Y / W, Z / W). By using stereo matching, all corresponding points in the left and right views can be found, and the corresponding 3D spatial points can be calculated to obtain a real-time 3D model.

[0145] like Figure 21 The diagram illustrates the procedure for hair transplantation. Computer equipment acquires a real-time image of the patient at time t0 using an image acquisition device; that is, at the very beginning of the transplantation process, binocular image acquisition is performed on the entire head of the patient. A 3D reconstruction is performed on the real-time image of the patient at time t0 to obtain a real-time 3D model. The real-time 3D model at time t0 is registered with the point cloud model of the area to be transplanted before the procedure to map the planned hair transplant path onto the real-time 3D model. A real-time image of a local area of ​​the patient's head at time ti is acquired, and a 3D intermediate image is performed on the real-time image at time ti to obtain a real-time 3D model at time ti. The real-time 3D model at time ti is registered with the real-time 3D model at time t0 to mitigate the impact of head movement during the transplantation process. Figure 22 The diagram shows a robotic arm implanting hair follicles. Referring to the method used for robotic arm hair follicle implantation, the robotic arm can be controlled to extract hair follicles.

[0146] Specifically, the computer equipment registers the point cloud of the area to be transplanted with the real-time 3D model to obtain a second registration matrix. Based on the second registration matrix, the hair transplant planning path is transformed to obtain the transformed hair transplant path. According to the transformed hair transplant path, each hair transplant position and its order are determined within the area to be transplanted. For any given hair transplant position, the rotation angles of multiple joints of the robotic arm are determined. The movement of the robotic arm is controlled based on these rotation angles, and the end effector of the robotic arm is controlled to perform hair transplantation at the desired position. The calculation method for the coordinate transformation relationship of the robotic arm end effector includes: since the end effector is located at the end of the robotic arm and its position is fixed, the robotic arm can be positioned by calculating its pose. Before the hair transplant operation, hand-eye calibration is used to obtain the transformation relationship between the 3D coordinate system C of the image acquisition device and the coordinate system G of the robotic arm end effector, as well as the rotation matrix of the robotic arm end effector coordinate system G relative to the robotic arm base coordinate system R. R E G (U,V,W) and position vector R P G (X,Y,Z), given a position vector on the end effector coordinate system of the robotic arm. G P can be used to obtain the representation of this vector in the robot arm's base coordinate system R. R P:

[0147] R P = R E G × G P+ R P G

[0148] in, R E G = R T G (W,Y,Z) and (X,Y,Z) correspond to the three translational degrees of freedom of the robotic arm, while (U,V,W) correspond to the rotational degrees of freedom of the robotic arm.

[0149] Given the initial position vector of the robotic arm's end effector G V p ([XYZ)) T ) and the current robotic arm rotation transformation R E G and the location vector of the hair follicle point Calculate the required rotation transformation and position vector The following steps are taken: First, calculate the rotation axis between the two poses. G V axis and rotation angle θ:

[0150]

[0151] Initial pose G V p Position of the hair follicle point The rotational motion can be described as: around The rotation is θ by an angle in the right-handed direction. The rotation angle θ can be represented by a quaternion:

[0152]

[0153] The general expression for quaternions is: q = s + xi + yi + zk, s, x, y, z ∈ R.

[0154] Euler angles can be calculated using quaternions:

[0155]

[0156] By q axis Obtain Euler angle U new V new W new The rotation matrix can be obtained. and The rotation angle (U) can be calculated using the rotation matrix. * V * W * In the known R In the case of P, It can be done R The formula for calculating P is obtained. Using the above calculation method, given the initial pose of the robotic arm's end effector, the rotational transformation matrix and displacement vector from the robotic arm to the hair follicle point can be obtained.

[0157] In some embodiments, before performing hair transplantation on the recipient, it is necessary to extract hair follicles from the donor site. These extracted hair follicles are then used for hair transplantation. The specific method for extracting hair follicles is as follows:

[0158] Obtain the real-time 3D model of the object to be extracted. Identify hair follicle points in the stored image, and cluster the hair follicle points in the stored image according to density to obtain multiple cluster regions. Determine the extraction region in the stored image based on the hair follicle point density of each cluster region. Specifically, the region with the highest hair follicle point density in each cluster region is taken as the target extraction region. Obtain the number of hair follicle points in the target extraction region. If the number of hair follicle points is less than the preset extraction quantity, re-cluster the remaining hair follicle points in the stored image after removing the target extraction region to obtain an updated cluster region. Based on the hair follicle point density of the updated cluster region, redetermine the target extraction region, and return to the step of obtaining the number of hair follicle points in the target extraction region to continue execution until the number of hair follicle points in the target extraction region is not less than the preset extraction quantity. The target extraction region is then taken as the extraction region in the stored image.

[0159] Based on the retrieval region in the existing image and the existing image itself, a 3D reconstruction is performed to obtain the retrieval point cloud of the back of the head. Specifically, the existing image is classified using a second classification model to obtain the target back-of-head image. Based on the retrieval region in the target back-of-head image, a segmentation model is used to segment the target back-of-head image to obtain a 2D retrieval image. Based on the 2D retrieval image and the target back-of-head image, a 3D reconstruction is performed to obtain the retrieval point cloud of the back of the head.

[0160] The point cloud of the posterior head segmented from the existing 3D model is registered with the hair follicle extraction point cloud at the posterior head to obtain a third registration matrix. Based on the third registration matrix, the existing 3D model, and the hair follicle extraction point cloud at the posterior head, the model point cloud of the region to be extracted is determined, and this point cloud is used to determine the hair follicle extraction planning path. The model point cloud of the region to be extracted is registered with the real-time hair follicle extraction 3D model to obtain a fourth registration matrix. According to the fourth registration matrix and the hair follicle extraction planning path, hair follicles are extracted from the object to be extracted. Correspondingly, hair transplantation is performed on the object to be transplanted according to the second registration matrix and the hair transplantation planning path, including: transplanting hair follicles into the object using the extracted hair follicles according to the second registration matrix and the hair transplantation planning path.

[0161] In addition, such as Figure 23 The diagram illustrates the process of generating a 3D hairstyle model. First, the desired hairstyle image is determined. Based on the 2D hair transplant image, the 2D hair extraction image, and the desired hairstyle image, a 3D hairstyle model is generated using a deep learning model. Then, the 3D hairstyle model is adjusted based on the point clouds of the hair extraction area and the hair transplant area to obtain the adjusted hairstyle model.

[0162] The aforementioned hair transplant method involves acquiring both an existing 3D model and a real-time 3D model of the subject to be transplanted. Based on the existing image, key point detection is used to obtain the actual hairline. Then, based on the actual hairline, the existing hairline, and the existing image, 3D reconstruction is performed to obtain a point cloud of the face to be transplanted. The point cloud of the model face, segmented from the existing 3D model, is registered with the point cloud of the face to be transplanted to obtain a first registration matrix. Based on the first registration matrix, the existing 3D model, and the point cloud of the face to be transplanted, the point cloud of the model for the area to be transplanted is determined. This method involves segmenting the point cloud of the model face from the existing 3D model of the subject to be transplanted and then... The facial point cloud registration method can determine the model point cloud of the area to be transplanted on an existing 3D model by registering the facial point cloud. Based on the model point cloud of the area to be transplanted, a second registration matrix is ​​obtained by registering it with the real-time 3D model. According to the second registration matrix and the hair transplant planning path, hair transplantation is performed on the subject. The hair transplant planning area is determined by the model point cloud of the area to be transplanted. By registering the model point cloud of the area to be transplanted with the real-time 3D model, the hair transplant planning path can be mapped into the real-time 3D model, thereby performing hair transplantation on the subject according to the hair transplant planning path, improving hair transplantation efficiency.

[0163] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0164] Based on the same inventive concept, this application also provides a hair transplant device for implementing the hair transplant method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more hair transplant device embodiments provided below can be found in the limitations of the hair transplant method described above, and will not be repeated here.

[0165] In one embodiment, such as Figure 24 As shown, a hair transplant device 100 is provided, including: an acquisition module 120, a facial point cloud acquisition module 140, a model point cloud determination module 160, and a hair transplant module 180, wherein:

[0166] The acquisition module 120 is used to acquire the existing 3D model and the real-time 3D model of the object to be transplanted. The existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted.

[0167] The facial point cloud acquisition module 140 is used to obtain the actual hairline based on the existing image through key point detection; and to obtain the hair transplant facial point cloud through three-dimensional reconstruction based on the actual hairline, the existing hairline, and the existing image.

[0168] The model point cloud determination module 160 is used to register the model facial point cloud obtained by segmenting the existing 3D model with the hair transplant facial point cloud to obtain a first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path.

[0169] The hair transplant module 180 is used to register the model point cloud of the area to be transplanted with the real-time 3D model to obtain the second registration matrix; and to perform hair transplantation on the object to be transplanted according to the second registration matrix and the hair transplantation planning path.

[0170] The aforementioned hair transplant device acquires both a pre-existing 3D model and a real-time 3D model of the subject to be transplanted. Based on the pre-existing image, it obtains the actual hairline through key point detection. Based on the actual hairline, the pre-existing hairline, and the pre-existing image, it performs 3D reconstruction to obtain a point cloud of the transplanted face. It then registers the point cloud of the transplanted face with the point cloud obtained from segmenting the pre-existing 3D model to obtain a first registration matrix. Based on the first registration matrix, the pre-existing 3D model, and the point cloud of the transplanted face, it determines the point cloud of the area to be transplanted. This method involves segmenting the point cloud of the pre-existing 3D model of the subject and then registering it with the point cloud of the transplanted face to obtain a first registration matrix. The facial point cloud registration method can determine the model point cloud of the area to be transplanted on an existing 3D model by registering the facial point cloud. Based on the model point cloud of the area to be transplanted, a second registration matrix is ​​obtained by registering it with the real-time 3D model. According to the second registration matrix and the hair transplant planning path, hair transplantation is performed on the subject. The hair transplant planning area is determined by the model point cloud of the area to be transplanted. By registering the model point cloud of the area to be transplanted with the real-time 3D model, the hair transplant planning path can be mapped into the real-time 3D model, thereby performing hair transplantation on the subject according to the hair transplant planning path, improving hair transplantation efficiency.

[0171] In one embodiment, in obtaining the actual hairline based on an existing image through key point detection, the facial point cloud acquisition module 140 is further configured to: classify the existing image through a first classification model to obtain a target facial image; and perform key point detection on the target facial image to obtain the actual hairline.

[0172] In one embodiment, in obtaining a hair transplant facial point cloud through three-dimensional reconstruction based on the actual hairline, the existing hairline, and the existing image, the facial point cloud acquisition module 140 is further configured to: segment the target facial image using a segmentation model based on the actual hairline and the existing hairline to obtain a two-dimensional hair transplant image; and obtain a hair transplant facial point cloud through three-dimensional reconstruction based on the two-dimensional hair transplant image and the target facial image.

[0173] In one embodiment, in segmenting the target facial image using a segmentation model based on the actual hairline and the existing hairline to obtain a two-dimensional hair transplant image, the facial point cloud acquisition module 140 is further configured to: determine whether the actual hairline includes only one curve; if the actual hairline includes at least two curves, sequentially connect the different curves of the at least two curves end to end to obtain a connected hairline; segment the target facial image using a segmentation model based on the connected hairline and the existing hairline to obtain a two-dimensional hair transplant image; and if the actual hairline includes only one curve, segment the target facial image using a segmentation model based on the actual hairline and the existing hairline to obtain a two-dimensional hair transplant image.

[0174] In one embodiment, regarding determining the model point cloud of the area to be transplanted based on the first registration matrix, the existing 3D model, and the hair transplant facial point cloud, the model point cloud determination module 160 is further configured to: transform the area to be transplanted in the hair transplant facial point cloud according to the first registration matrix to obtain a transformed hair transplant area point cloud; and determine the model point cloud of the area to be transplanted based on the transformed hair transplant area point cloud and the existing 3D model. In one embodiment, regarding performing hair transplantation on the object to be transplanted based on the second registration matrix and the hair transplant planning path, the hair transplant module 180 is further configured to: transform the hair transplant planning path according to the second registration matrix to obtain a transformed hair transplant path; determine each hair transplant position and the order of each hair transplant position in the area to be transplanted based on the transformed hair transplant path; determine the rotation angle of multiple joints of the robotic arm for any hair transplant position; control the movement of the robotic arm according to the rotation angle of the multiple joints; and control the end effector of the robotic arm to perform hair transplantation at the hair transplant position.

[0175] In one embodiment, in determining the hair transplant planning path, the model point cloud determination module 160 is further configured to: divide the hair transplant area in the model point cloud of the area to be transplanted into at least one block; determine the hair transplant path composed of each hair transplant position in each block; and connect the hair transplant paths corresponding to each block in sequence according to the order of each block to obtain the hair transplant planning path.

[0176] In one embodiment, before performing hair transplantation on the subject based on the second registration matrix and the hair transplantation planning path, the hair transplantation device 100 further includes a hair extraction module: the hair extraction module is also used to acquire a real-time three-dimensional model of the subject to be extracted; identify hair follicle points in the stored image, and determine the area to be extracted in the stored image through density clustering; obtain the posterior head hair extraction point cloud through three-dimensional reconstruction based on the area to be extracted in the stored image and the stored image; register the model posterior head point cloud obtained by segmentation based on the stored three-dimensional model with the posterior head hair extraction point cloud to obtain a third registration matrix; based on the third registration matrix and the existing three-dimensional model, the hair transplantation device 100 further acquires a hair extraction point cloud; and based on the second registration matrix and the existing three-dimensional model, the hair transplantation device 100 further acquires a hair extraction point cloud; and based on the second registration matrix and the existing three-dimensional model, the hair transplantation device 100 acquires a real-time three-dimensional model of the subject to be extracted. The system stores a 3D model and a point cloud of the posterior head for hair extraction, and determines the model point cloud of the area to be extracted. The model point cloud of the area to be extracted is used to determine the hair extraction planning path. Based on the model point cloud of the area to be extracted, the system registers it with the real-time 3D model for hair extraction to obtain a fourth registration matrix. According to the fourth registration matrix and the hair extraction planning path, the system extracts hair from the object to be extracted and obtains the extracted hair follicles. Correspondingly, according to the second registration matrix and the hair transplant planning path, the system performs hair transplantation on the object to be transplanted. The hair transplant module 180 is also used to: perform hair transplantation on the object to be transplanted using the extracted hair follicles according to the second registration matrix and the hair transplant planning path.

[0177] In one embodiment, density clustering is used to determine the region to be extracted in the stored image. The extraction module is further configured to: cluster the hair follicle points in the stored image according to density to obtain multiple clustered regions; and determine the region to be extracted in the stored image based on the hair follicle point density of each clustered region.

[0178] In one embodiment, based on the hair follicle density of each cluster region, the region to be extracted in the stored image is determined. The extraction module is further configured to: take the region with the highest hair follicle density in each cluster region as the target extraction region; obtain the number of hair follicles in the target extraction region; if the number of hair follicles is less than a preset extraction number, re-cluster the remaining hair follicles in the stored image after removing the target extraction region to obtain an updated cluster region; redetermine the target extraction region based on the hair follicle density of the updated cluster region, and return to the step of obtaining the number of hair follicles in the target extraction region to continue execution until the number of hair follicles in the target extraction region is not less than the preset extraction number, and then take the target extraction region as the region to be extracted in the stored image.

[0179] In one embodiment, based on the retrieval region in the existing image and the existing image, a retrieval point cloud of the back of the head is obtained through 3D reconstruction. The retrieval module is further configured to: classify the existing image using a second classification model to obtain a target back of the head image; segment the target back of the head image using a segmentation model based on the retrieval region in the target back of the head image to obtain a 2D retrieval image; and obtain a retrieval point cloud of the back of the head through 3D reconstruction based on the 2D retrieval image and the target back of the head image.

[0180] In one embodiment, the hair transplant module 180 is further configured to: determine the desired hairstyle image; generate a three-dimensional hairstyle model using a deep learning model based on the two-dimensional hair transplant image, the two-dimensional hair extraction image, and the desired hairstyle image; and adjust the three-dimensional hairstyle model based on the model point cloud of the area to be extracted and the model point cloud of the area to be transplanted, to obtain the adjusted hairstyle model.

[0181] Each module in the aforementioned hair transplant device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0182] In one embodiment, a hair transplant system is provided, comprising a robotic arm and a computer device. The robotic arm includes multiple joints, and an end effector is mounted at the end of the robotic arm. The rotation angles of the multiple joints allow the robotic arm to perform multi-degree-of-freedom movements. The end effector is used to perform hair transplantation at the desired location. The computer device can be a terminal. The computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. The computer program is executed by the processor to implement a hair transplant method.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A hair transplant system, characterized in that, The system includes: A robotic arm, comprising multiple joints, with an end effector mounted at its end; the rotation angles of the multiple joints allow the robotic arm to perform multi-degree-of-freedom movements, and the end effector is used to perform hair transplantation at the desired location; A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Obtain the existing 3D model and the real-time 3D model of the object to be transplanted, wherein the existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted; Based on the existing image, the actual hairline is obtained through key point detection; based on the actual hairline, the existing hairline, and the existing image, a point cloud of the hair transplanted face is obtained through three-dimensional reconstruction. The model facial point cloud obtained by segmenting the existing 3D model is registered with the hair transplant facial point cloud to obtain a first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, the model point cloud of the area to be transplanted is determined; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path. The point cloud of the area to be transplanted is registered with the real-time 3D model to obtain a second registration matrix; hair transplantation is performed on the object to be transplanted according to the second registration matrix and the hair transplantation planning path. When the processor executes the computer program, it also performs the following steps: The existing image is classified using a first classification model to obtain the target facial image; Key point detection is performed on the target facial image to obtain the actual hairline.

2. The system according to claim 1, characterized in that, When the processor executes the computer program, it also performs the following steps: Based on the actual hairline and the existing hairline, a segmentation model is used to segment the target facial image to obtain a two-dimensional hair transplant image; Based on the two-dimensional hair transplant image and the target facial image, a point cloud of the transplanted face is obtained through three-dimensional reconstruction.

3. The system according to claim 2, characterized in that, When the processor executes the computer program, it also performs the following steps: Determine whether the actual hairline consists of only one curve; When the actual hairline includes at least two curves, the different curves in the at least two curves are connected end to end in sequence to obtain the connected hairline; based on the connected hairline and the existing hairline, the target facial image is segmented using a segmentation model to obtain a two-dimensional hair transplant image; When the actual hairline consists of only one curve, the target facial image is segmented using a segmentation model based on the actual hairline and the existing hairline to obtain a two-dimensional hair transplant image.

4. The system according to claim 1, characterized in that, When the processor executes the computer program, it also performs the following steps: Based on the first registration matrix, the area to be transplanted in the hair transplant facial point cloud is transformed to obtain the transformed hair transplant area point cloud. Based on the converted hair transplant area point cloud and the existing 3D model, the model point cloud of the area to be transplanted is determined.

5. The system according to claim 1, characterized in that, When the processor executes the computer program, it also performs the following steps: Based on the second registration matrix, the hair transplant planning path is transformed to obtain the transformed hair transplant path; Based on the converted hair transplant path, determine each hair transplant location and the order of each hair transplant location in the area to be transplanted; For any hair transplant site, the rotation angles of multiple joints of the robotic arm are determined, the movement of the robotic arm is controlled according to the rotation angles of the multiple joints, and the end effector of the robotic arm is controlled to perform hair transplantation at the hair transplant site.

6. The system according to claim 1, characterized in that, When the processor executes the computer program, it also performs the following steps: The hair transplant area in the model point cloud of the area to be transplanted is divided into at least one block; Determine the hair transplant path composed of each hair transplant location in each segment; Following the order of each section, the hair transplant paths corresponding to each section are connected sequentially to obtain the hair transplant planning path.

7. The system according to claim 1, characterized in that, When the processor executes the computer program, it also performs the following steps: Obtain the real-time 3D model of the object to be retrieved / sent; Hair follicle points in the stored image are identified, and the hair extraction area in the stored image is determined by density clustering. Based on the hair extraction area in the stored image and the stored image, a hair extraction point cloud of the back of the head is obtained by three-dimensional reconstruction. The point cloud of the rear part of the model head obtained by segmenting the existing 3D model is registered with the point cloud of the rear part of the head for take-off, to obtain a third registration matrix; based on the third registration matrix, the existing 3D model and the point cloud of the rear part of the head for take-off, the model point cloud of the region to be taken-off is determined; the model point cloud of the region to be taken-off is used to determine the take-off planning path. Based on the model point cloud of the region to be extracted and the real-time extraction 3D model, a fourth registration matrix is ​​obtained; according to the fourth registration matrix and the extraction planning path, the hair follicles to be extracted are extracted from the object to be extracted. Accordingly, the step of performing hair transplantation on the recipient based on the second registration matrix and the hair transplantation planning path includes: Based on the second registration matrix and the hair transplant planning path, the extracted hair follicles are used to perform hair transplantation on the recipient.

8. The system according to claim 7, characterized in that, When the processor executes the computer program, it also performs the following steps: The existing image is classified using a second classification model to obtain the image of the rear of the target head; Based on the retrieval region in the target head rear image, a segmentation model is used to segment the target head rear image to obtain a two-dimensional retrieval image; Based on the two-dimensional image of the target head and the image of the back of the head, a point cloud of the target head is obtained through three-dimensional reconstruction.

9. The system according to claim 8, characterized in that, When the processor executes the computer program, it also performs the following steps: Determine the desired hairstyle image; Based on the two-dimensional hair transplant image, the two-dimensional hair extraction image, and the desired hairstyle image, a three-dimensional hairstyle model is generated using a deep learning model. Based on the point cloud of the hair extraction area and the point cloud of the hair transplant area, the three-dimensional hairstyle model is adjusted to obtain the adjusted hairstyle model.

10. The system according to claim 1, characterized in that, When the processor executes the computer program, it also performs the following steps: A key point detection algorithm is used to detect key points in the target facial image, and the hairline key point in the target facial image is detected. The key points of the hairline are fitted into a curve to obtain the actual hairline.

11. A hair transplant device, characterized in that, The device is used in the hair transplant system according to any one of claims 1 to 10, the device comprising: The acquisition module is used to acquire the existing 3D model and the real-time 3D model of the object to be transplanted, wherein the existing 3D model is obtained by 3D reconstruction based on the existing image of the object to be transplanted. The facial point cloud acquisition module is used to obtain the actual hairline based on the existing image through key point detection; and to obtain the hair transplant facial point cloud through three-dimensional reconstruction based on the actual hairline, the existing hairline, and the existing image. The model point cloud determination module is used to register the model facial point cloud obtained by segmenting the existing 3D model with the hair transplant facial point cloud to obtain a first registration matrix; based on the first registration matrix, the existing 3D model and the hair transplant facial point cloud, determine the model point cloud of the area to be transplanted; the model point cloud of the area to be transplanted is used to determine the hair transplant planning path. The hair transplant module is used to register the model point cloud of the area to be transplanted with the real-time 3D model to obtain a second registration matrix; and to perform hair transplantation on the object to be transplanted according to the second registration matrix and the hair transplantation planning path. The facial point cloud acquisition module is further configured to classify the stored image using a first classification model to obtain a target facial image; and to perform key point detection on the target facial image to obtain the actual hairline.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor of the hair transplant system according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, The computer program is executed by the processor of the hair transplant system according to any one of claims 1 to 10.

Images