Hair extraction result prediction method and hair extraction robot control method

Abstract

This application relates to a method for predicting hair extraction results, a method for controlling a hair extraction robot, a method for processing hair extraction results, a method for acquiring model training data, a hair extraction system, a surgical system, a computer device, a storage medium, and a computer program product. The method includes: acquiring first hair information before hair extraction; obtaining planned motion data for a robotic arm and planned motion data for an end effector based on the first hair information; performing multimodal fusion of the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector to obtain first model input data; and inputting the first model input data into a pre-trained hair extraction result prediction model to obtain a predicted hair extraction result. This method uses overquantified data to obtain predicted hair extraction results, resulting in higher accuracy. Furthermore, the hair extraction results can be judged before each extraction to ensure the accuracy of subsequent hair extractions.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular to a method for predicting hair extraction results, a method for controlling a hair extraction robot, a method for processing hair extraction results, a method for acquiring model training data, a hair extraction system, a surgical system, a computer device, a storage medium, and a computer program product. Background Technology

[0002] Like other tissues in the human body, human hair can be transplanted. Hair transplantation involves using special instruments to completely remove the tissue surrounding the hair follicle, detach it from its original location on the scalp, and then transplant it to a location that needs hair and has been prepared to receive it.

[0003] In traditional techniques, hair needs to be extracted before hair transplantation. This extraction is usually done by machine, and the operator visually assesses the results.

[0004] However, when operators rely on visual judgment, the assessment process is time-consuming, and for inexperienced operators, the assessment results may be inaccurate. Summary of the Invention

[0005] Therefore, it is necessary to provide a hair extraction result prediction method, a hair extraction robot control method, a hair extraction result processing method, a model training data acquisition method, a hair extraction system, a surgical system, a computer device, a storage medium, and a computer program product that can improve the accuracy of the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for predicting hair extraction results, the method comprising:

[0007] Obtain initial hair information before hair extraction;

[0008] Based on the first hair information, the planned motion data of the robotic arm and the planned motion data of the end effector are obtained;

[0009] The first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are fused in a multimodal manner to obtain the first model input data.

[0010] The input data of the first model is input into the pre-trained hair extraction result prediction model to obtain the predicted hair extraction result.

[0011] In one embodiment, the step of multimodal fusion of the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector to obtain the first model input data includes:

[0012] The first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are preprocessed to obtain the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector with the same dimensions.

[0013] The first model input data is obtained by multimodal fusion of the first hair information of the same dimension, the planned motion data of the robotic arm, and the planned motion data of the end effector.

[0014] In one embodiment, obtaining the first hair information before hair extraction includes:

[0015] A first hair image of the target hair area is acquired using an image acquisition device, and first hair information is obtained by identifying the first hair image; or

[0016] The first hair information is obtained by locating the hair in the target hair area using a positioning instrument.

[0017] Secondly, this application also provides a control method for a hair extraction robot, the hair extraction robot control method comprising:

[0018] The planned motion data of the robotic arm and the planned motion data of the end effector of the hair extraction robot are corrected based on the predicted hair extraction results to obtain the target motion data of the robotic arm and the target motion data of the end effector; the predicted hair extraction results are obtained according to the hair extraction result prediction method described in any of the above embodiments.

[0019] The movement of the robotic arm is controlled according to the target motion data of the robotic arm, and the movement of the end effector is controlled according to the target motion data of the end effector.

[0020] In one embodiment, after controlling the movement of the robotic arm based on the target motion data of the robotic arm, and controlling the movement of the end effector based on the target motion data of the end effector, the method further includes:

[0021] During the movement of the robotic arm, the step of acquiring the first hair information before hair extraction continues until the end effector completes hair extraction.

[0022] Thirdly, this application also provides a method for processing hair extraction results, the method comprising:

[0023] The hair extraction robot is controlled to move according to the hair extraction robot control method in any of the above embodiments in order to extract hair.

[0024] Obtain the second hair information after hair extraction, and obtain the actual motion data of the robotic arm and the actual motion data of the end effector;

[0025] The second hair information, the first hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector are fused using multimodal methods to obtain the second model input data.

[0026] The second model input data is input into the pre-trained hair extraction result judgment model to obtain the hair extraction result.

[0027] In one embodiment, acquiring the second hair information after hair extraction, acquiring the actual motion data of the robotic arm and the actual motion data of the end effector, includes at least one of the following:

[0028] Acquire a second hair image after hair extraction, extract second hair information from the second hair image, and obtain the actual motion data of the robotic arm and the end effector through sensors; or

[0029] Multiple target hair images are acquired during the hair extraction process, including a second hair image after hair extraction. Second hair information is extracted from the second hair image. The actual motion data of the robotic arm and the actual motion data of the end effector are determined based on the multiple target hair images.

[0030] In one embodiment, the motion data of the end effector includes trajectory data and force data; the acquisition of the actual motion data of the robotic arm and the actual motion data of the end effector through sensors includes:

[0031] The actual motion data of the robotic arm and the trajectory data of the end effector are acquired through sensors;

[0032] The current corresponding to the end effector is obtained by the sensor, and the force data of the end effector is calculated based on the current.

[0033] In one embodiment, the motion data of the end effector includes trajectory data and force data; determining the motion data of the robotic arm and the actual motion data of the end effector based on multiple images of the target hair includes:

[0034] The actual motion data of the robotic arm and the trajectory data of the end effector are determined based on multiple images of the target hair.

[0035] The force data of the end effector is determined based on the deformation of the scalp in the multiple target hair images.

[0036] In one embodiment, after inputting the second model input data into a pre-trained hair extraction result judgment model to obtain the hair extraction result, the process includes:

[0037] The truncation rate is calculated based on the extraction results of each hair.

[0038] Fourthly, this application also provides a method for processing hair extraction results, the method comprising:

[0039] The hair extraction robot control method described in any of the above embodiments controls the movement of the hair extraction robot to extract hair.

[0040] Acquire the images of the hair to be processed collected during the hair extraction process;

[0041] The hair image to be processed is input into a pre-trained hair extraction result judgment model to obtain the hair extraction result.

[0042] Fifthly, this application also provides a method for acquiring model training data, which is used to train a hair extraction result prediction model or a hair extraction result judgment model. The method for acquiring model training data includes:

[0043] Acquire hair information, end effector motion data, robotic arm motion data, and hair extraction results collected during the hair retrieval process;

[0044] Model training data is obtained based on the hair information, the motion data of the end effector, the motion data of the robotic arm, and the hair extraction results.

[0045] In one embodiment, the acquisition of hair information, end effector motion data, robotic arm motion data, and hair extraction results collected during the hair retrieval process includes at least one of the following:

[0046] The hair information acquisition device model, robotic arm model, end effector model, and head model are imported into the initial virtual environment to obtain the target virtual environment. In this target virtual environment, hair is extracted from the head model using the hair information acquisition device model, robotic arm model, and end effector model. The motion data of the end effector model, the motion data of the robotic arm model, the hair information of the target hair, and the hair extraction result are recorded during the extraction process.

[0047] Acquire the actual hair information collected by the hair information acquisition device, the motion data of the end effector, and the motion data of the robotic arm during the actual operation; receive the actual hair extraction results based on the actual hair information.

[0048] In one embodiment, obtaining model training data based on the hair information, motion data of the end effector, motion data of the robotic arm, and hair extraction results includes at least one of the following:

[0049] The pose difference between the end effector model and the target hair is calculated based on the motion data of the end effector model and the hair information of the target hair; when the hair retrieval result is the target hair retrieval result, the statistical value of the pose difference corresponding to the target hair retrieval result is obtained as a threshold; the simulated hair extraction result corresponding to each target hair is calculated based on the pose difference of each target hair and the threshold; the simulated hair extraction result, the motion data of the end effector model, the motion data of the robotic arm model, and the hair information of the target hair are used as model training data; or

[0050] Model training data is obtained based on the motion data of the end effector, the motion data of the robotic arm, the actual hair information, and the actual hair extraction results.

[0051] Sixthly, this application also provides a hair extraction system, the hair extraction system comprising:

[0052] Hair information collection device, used to collect hair information from hair;

[0053] A robotic arm used to move an end effector to a target hair area;

[0054] An end effector, installed at the end of the robotic arm, is used to perform pick-up and release operations;

[0055] A controller is used to execute the steps of the method described in any of the above embodiments.

[0056] Seventhly, this application also provides a surgical system, including the hair extraction system described in any of the above embodiments.

[0057] Eighthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory to implement the steps of the method described in any of the above embodiments.

[0058] The aforementioned hair extraction result prediction method, hair extraction robot control method, hair extraction result processing method, model training data acquisition method, hair extraction system, surgical system, computer equipment, storage medium, and computer program product acquire first hair information before hair extraction. This allows for the acquisition of planned motion data for the robotic arm and the end effector based on the first hair information. Then, the hair extraction result is predicted through model evaluation. The predicted hair extraction result is obtained through quantified data, resulting in higher accuracy. Furthermore, the hair extraction result can be judged before each extraction to ensure the accuracy of subsequent hair extractions. Attached Figure Description

[0059] Figure 1 This is a system architecture diagram of a hair extraction system in one embodiment;

[0060] Figure 2 This is a system architecture diagram of a surgical system in one embodiment;

[0061] Figure 3 This is an enlarged view of the end effector of the robotic arm in one embodiment;

[0062] Figure 4 This is a flowchart illustrating a hair extraction result prediction method in one embodiment.

[0063] Figure 5 This is a schematic diagram of the steps for generating input data for the first model in one embodiment;

[0064] Figure 6 A schematic diagram of the steps for generating input data for the first model in another embodiment;

[0065] Figure 7 This is a flowchart illustrating the control method for a hair extraction robot in one embodiment;

[0066] Figure 8 This is a schematic diagram of the structure of a pre-trained hair extraction result judgment model in one embodiment;

[0067] Figure 9 This is a schematic diagram of the structure of a pre-trained hair extraction result judgment model in another embodiment;

[0068] Figure 10 This is a schematic diagram of the structure of a pre-trained hair extraction result judgment model in one embodiment;

[0069] Figure 11 This is a flowchart illustrating a hair extraction result processing method in one embodiment;

[0070] Figure 12This is a schematic diagram of the second hair information extraction step in one embodiment;

[0071] Figure 13 This is a schematic diagram of the actual motion data of the robotic arm in one embodiment;

[0072] Figure 14 This is a flowchart illustrating the hair extraction result processing method in another embodiment;

[0073] Figure 15 This is a flowchart illustrating a method for acquiring model training data in one embodiment;

[0074] Figure 16 This is a flowchart illustrating a method for acquiring model training data in another embodiment;

[0075] Figure 17 This is a flowchart illustrating the hair extraction result processing method in yet another embodiment;

[0076] Figure 18 This is a flowchart illustrating the hair extraction result processing method in another embodiment;

[0077] Figure 19 This is a schematic diagram of the model training steps in one embodiment;

[0078] Figure 20 This is a schematic diagram illustrating the optimization of the sending and receiving process by truncation probability in one embodiment;

[0079] Figure 21 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0080] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0081] Specifically, in combination Figure 1As shown, a hair extraction system is provided, comprising a hair information acquisition device 100, a robotic arm 200, an end effector 300, and a controller 400. The end effector 300 is mounted at the end of the robotic arm 200. The hair information acquisition device 100 is used to acquire hair information; the robotic arm 200 moves the end effector 300 to the target hair area; the end effector 300 performs the hair retrieval operation; and the controller 400 controls the robotic arm 200. Specifically, the hair information acquisition device 100 can be a camera, which captures images of the hair to obtain hair information, including hair position and orientation. The robotic arm 200 can move the end effector 300 to the hair position based on the hair position and orientation, allowing the end effector 300 to perform the corresponding hair retrieval operation via a motor, etc. Optionally, the end effector 300 can also calculate the force applied to it by reading the current reading of the motor on the end effector 300. The controller 400 primarily performs two functions: first, it acquires hair information collected by the hair information acquisition device 100 and controls the movement of the robotic arm 200 and the end effector 300 to perform hair retrieval operations based on this information; second, it calculates a predicted hair extraction result based on the first hair information, the planned motion data of the robotic arm 200, and the planned motion data of the end effector 300, and calculates the final hair extraction result based on the second hair information, the first hair information, the actual motion data of the robotic arm 200, and the actual motion data of the end effector 300.

[0082] Specifically, in combination Figure 2 As shown, a surgical system is provided, which is mainly suitable for using a robot to perform the retrieval and delivery operation. The surgical system includes... Figure 1 The hair extraction system shown. The patient's head is exposed 500 mm, combined with... Figure 3 The hair information acquisition device 100 and the end effector 300 are installed at the end of the robotic arm 200. In this way, the hair information acquisition device can collect hair information and send it to the controller 400. The controller 400 obtains the hair information and controls the robotic arm 200 to move based on the hair information, so that the end effector 300 moves to the corresponding hair area and controls the end effector 300 to perform hair retrieval operation.

[0083] In one embodiment, such as Figure 4 As shown, a method for predicting hair extraction results is provided, which is then applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0084] S402: Obtain the first hair information before hair extraction.

[0085] Specifically, the first hair information refers to the position and orientation information of the hair before extraction. Specifically, the first hair information includes root data and end data. The root data includes the position and orientation of the hair root, and the end data includes the position and orientation of the hair end. The hair information acquisition device can acquire a hair image and send it to the controller, which then identifies the hair image to determine the first hair information. In other embodiments, the hair information acquisition device can directly acquire the hair information and send it to the controller.

[0086] Specifically, obtaining the first hair information before hair extraction includes: when the hair information acquisition device is an image acquisition device, acquiring the first hair image of the target hair area through the image acquisition device, and identifying the first hair image to obtain the first hair information; or when the hair acquisition device is a positioning instrument, locating the hair in the target hair area through the positioning instrument to obtain the first hair information.

[0087] The process of identifying the first hair information in the first hair image can be as follows: first, identify the hair in the first hair image; then, identify the hair root and hair end; and then, based on the positioning function of the image acquisition device, such as the coordinate matching relationship between the camera and the real space, obtain the hair pose information, thus obtaining the hair root data and hair end data.

[0088] S404: Obtain the planned motion data of the robotic arm and the planned motion data of the end effector based on the first hair information.

[0089] The planned motion data for the robotic arm refers to the planned motion data for moving the end effector of the robotic arm to the hair area, including information such as the movement trajectory and speed of the robotic arm. It can be generated based on the current pose of the robotic arm and the first hair information. The specific planning method can be preset by the user and is not specifically limited here.

[0090] The planned motion data of the end effector is the planned motion data of the end effector after it moves to the hair area to perform the hair retrieval operation. It can include the insertion angle of the end effector needle, the insertion depth of the needle, and the force data of the end effector.

[0091] Specifically, the controller plans the motion data for moving the end effector of the robotic arm to the hair area based on the current position of the robotic arm, the current position of the end effector, and the first hair information, as well as the planned motion data for the end effector to perform hair retrieval operation after moving to the hair area.

[0092] S406: Multimodal fusion of the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector is performed to obtain the first model input data.

[0093] S408: Input the first model input data into the pre-trained hair extraction result prediction model to obtain the predicted hair extraction result.

[0094] Specifically, since the first hair information, the robotic arm's planned motion data, and the end effector's planned motion data are three different types of data, the first model input data can be obtained through multimodal parameters, thereby fusing real-time information from multiple modules. The multimodal fusion method can be a model-based fusion strategy or a model-independent fusion strategy.

[0095] In one optional embodiment, the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are multimodally fused to obtain the first model input data, including: preprocessing the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector to obtain the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector with the same dimension; and multimodally fusing the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector with the same dimension to obtain the first model input data.

[0096] Multimodal fusion can employ model-based fusion strategies, such as neural network-based fusion, where the neural network can be an LSTM model, a CNN convolutional neural network, or other types of neural networks. In other embodiments, multimodal fusion can also employ model-independent fusion strategies, such as pre-fusion and concatenating multimodal features in shallow layers of the model.

[0097] Combination Figure 5 and Figure 6 As shown, where Figure 5 In one embodiment, a model-based fusion strategy is used. The first hair information is a hair image. The hair image is input into a CNN (Convolutional Neural Network) to extract features of the first hair information. The planned motion data of the robotic arm and the planned motion input of the end effector are respectively input into a DNN (Deep Neural Network) to obtain corresponding features. Then, after processing through an embedding layer, the three types of data are concatenated to form a deep neural network, i.e., a pre-trained hair extraction result prediction model, to obtain the predicted hair extraction result. Figure 6 In another embodiment, a model-based fusion strategy is employed. Figure 6The first hair information is the hair image. First, the hair image is used to extract the first hair information to obtain the hair root data and hair tail data, that is, the position and posture of the hair root and hair tail. Then, the hair root data and hair tail data are input into LSTM (Long Short-Term Memory Artificial Neural Network) to extract the features of the first hair information. The planned motion data of the robotic arm and the planned motion input of the end effector are respectively input into the DNN deep neural network to obtain the corresponding features. Then, after processing through the embedding layer, the three types of data are connected as a deep neural network, that is, the pre-trained hair extraction result prediction model, to obtain the predicted hair extraction result.

[0098] One point to note is that the hair extraction result prediction model obtained through pre-training in the above embodiments can be a neural network or a linear regression model, or it can be replaced by other artificial intelligence methods, such as LSTM (Long Short-Term Memory), other neural networks such as CNN, or other neural networks such as Multilayer Perceptrons (MLPs), etc. There are no restrictions on this.

[0099] The predicted hair extraction result is calculated before extraction, based on the planned motion data of the robotic arm and the end effector. This predicted result can be either the hair truncation rate or the extraction error. The truncation rate refers to the probability of hair truncation after extraction, while the extraction error refers to the difference between the actual extraction result and the expected result. This method can be used to compare the hair extraction effects of different robots, helping doctors select the appropriate robot.

[0100] The above-mentioned hair extraction result prediction method first obtains the first hair information before hair extraction. This allows the robot arm's planned motion data and the end effector's planned motion data to be obtained based on the first hair information. Then, the hair extraction result is predicted through model evaluation. The predicted hair extraction result is obtained through quantified data, which has higher accuracy. In addition, the hair extraction result can be judged before each hair extraction to ensure the accuracy of subsequent hair extraction.

[0101] In one embodiment, such as Figure 7 As shown, a control method for a hair extraction robot is provided, which can be applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0102] S702: Correct the planned motion data of the robotic arm and the planned motion data of the end effector of the hair extraction robot according to the predicted hair extraction result, and obtain the target motion data of the robotic arm and the target motion data of the end effector; the predicted hair extraction result is obtained according to the hair extraction result prediction method in any of the above embodiments.

[0103] S704: Controls the movement of the robotic arm based on the target motion data of the robotic arm, and controls the movement of the end effector based on the target motion data of the end effector.

[0104] Specifically, the predicted hair extraction result is obtained based on the planned motion data of the robotic arm and the end effector. When the predicted hair extraction result indicates that hair extraction will lead to hair truncation or the extraction error exceeds a certain value, it is necessary to correct the planned motion data of the robotic arm and the end effector. For example, this involves correcting the robotic arm's trajectory and speed, as well as the end effector's needle insertion angle, insertion depth, and force data. For instance, if the hair is truncated on the skin, the robotic arm's trajectory needs to be corrected so that the end effector moves further towards the hair area. If the hair is truncated subcutaneously, the end effector's needle insertion depth, angle, and force data need to be corrected.

[0105] In one optional embodiment, after controlling the movement of the robotic arm based on the target motion data of the robotic arm and controlling the movement of the end effector based on the target motion data of the end effector, the method further includes: during the movement of the robotic arm, continuing to execute the step of obtaining the first hair information before hair extraction, until the end effector completes hair extraction.

[0106] Specifically, as the robotic arm carrying the end effector moves towards the hair area, the acquired first hair information becomes increasingly precise. This leads to more accurate planned motion data for both the robotic arm and the end effector, resulting in more accurate predicted hair extraction results from the model. Consequently, the target motion data for both the robotic arm and the end effector, adjusted based on these predicted results, becomes more precise. Therefore, throughout the robotic arm's movement, the controller can acquire the first hair information in real time, predict the predicted hair extraction results, and then adjust the target motion data for both the robotic arm and the end effector until the end effector completes hair extraction. This method can be applied to improve the accuracy of real-time hair retrieval by robots. For each hair retrieval, an effectiveness evaluation is generated. By incorporating this evaluation into the optimization objective function, the hair retrieval error is continuously reduced, improving surgical accuracy and lowering the truncation rate.

[0107] The controller controls the movement of the robotic arm and the end effector based on the corrected target motion data of the robotic arm and the end effector, thereby completing the pick-up and release.

[0108] In the above embodiments, before hair retrieval, a predicted hair extraction result is obtained based on the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector. Then, the planned motion data of the robotic arm and the planned motion data of the end effector are corrected based on the predicted hair extraction result to obtain the target motion data of the robotic arm and the target motion data of the end effector, thereby improving the hair retrieval accuracy.

[0109] Specifically, the hair extraction result processing method in this application has two implementation methods. One method is to acquire the actual motion data of the robotic arm and the end effector, and then obtain the actual hair extraction result based on the second hair information, the first hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector. The other method is to obtain the actual hair extraction result based on the hair images to be processed collected during the hair extraction process. When determining the second model input data of the hair extraction result judgment model, a model-based fusion strategy can be used to perform neural network-based fusion, using methods such as LSTM and convolutional layers. Alternatively, a model-independent fusion strategy can be used, such as pre-fusion and concatenating multimodal features in the shallow layers of the model. In addition, the hair extraction result judgment model can use a group of images or videos captured by a camera as input, using an artificial intelligence model in the field of image processing. The input of the hair extraction result judgment model can also be calculated from the group of images or videos, such as calculating the actual motion data of the robotic arm and the actual motion data of the end effector from the images.

[0110] Specifically, it can be combined with Figures 8 to 10 As shown, Figure 8 This is a schematic diagram of the structure of a pre-trained hair extraction result judgment model in one embodiment. Figure 9 This is a schematic diagram of the structure of a pre-trained hair extraction result judgment model in another embodiment. Figure 10 This is a schematic diagram of the structure of a pre-trained hair extraction result judgment model in another embodiment. Figure 8The first and second hair information are hair images. These images are input into a CNN (Convolutional Neural Network) to extract features from the first and second hair information. The actual motion data of the robotic arm and the actual motion input of the end effector are input into a DNN (Deep Neural Network) to obtain corresponding features. These features are then processed through an embedding layer, and the four types of data are concatenated to form a deep neural network, i.e., a pre-trained hair extraction result judgment model, to obtain the actual hair extraction result. Figure 9 The first and second hair information are hair images. First, the first and second hair information are extracted from the hair images to obtain hair root data and hair tail data, that is, the position and posture of the hair root and hair tail. Then, the hair root data and hair tail data are input into LSTM (Long Short-Term Memory Artificial Neural Network) to extract the features of the first and second hair information. The actual motion data of the robotic arm and the actual motion input of the end effector are respectively input into the DNN deep neural network to obtain the corresponding features. Then, after processing through the embedding layer, the four types of data are connected as a deep neural network, that is, the pre-trained hair extraction result judgment model, to obtain the actual hair extraction result.

[0111] One point to note is that the hair extraction result judgment model obtained through pre-training in the above embodiments can be a neural network or a linear regression model, or it can be replaced by other artificial intelligence methods, such as LSTM (Long Short-Term Memory), other neural networks such as CNN, or other neural networks such as Multilayer Perceptrons (MLPs), etc., without any limitation here.

[0112] Figure 10 The deep learning model used in this context needs to model sequences, which can be addressed using recurrent networks. Figure 10 The list includes Recurrent Convolutional Neural Networks (R-CNN), but practical applications are not limited to this. Figure 10 The actual hair extraction result is obtained based on the hair images to be processed collected during the hair extraction process.

[0113] In one embodiment, such as Figure 11 As shown, a method for processing hair extraction results is provided, which can be applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0114] S1102: Control the movement of the hair extraction robot according to the hair extraction robot control method to extract hair.

[0115] Specifically, in this embodiment, the movement of the robotic arm and end effector is controlled according to the hair extraction robot control method to extract hair, and the hair extraction result is obtained after hair extraction.

[0116] S1104: Obtain the second hair information after hair extraction, and obtain the actual motion data of the robotic arm and the actual motion data of the end effector.

[0117] The second hair information is obtained after hair extraction, and it may include data from empty spaces after hair extraction. Specifically, combined with... Figure 12 As shown, the controller acquires the hair image after hair extraction, then performs binarization on the hair image, and identifies gaps in the binarized hair image. The identification of gaps can be performed using image segmentation techniques, such as Hough transform, Sobel operator, Prewitt operator, wavelet multi-scale edge detection, and edge detection based on mathematical morphology. For example, it can identify circles and find the circle closest to the end effector to obtain gap data. In this way, the position information of the gap data contour is obtained, which can be 3D position information, including the two-dimensional region coordinates of the contour and the depth of the contour.

[0118] The actual motion data of the robotic arm and the end effector can be acquired through sensors or by identifying multiple target hair images collected during the hair extraction process. The actual motion data of the robotic arm can include its direction of movement and speed, etc. The actual motion data of the end effector can include the needle insertion angle, depth, and force, etc. Specifically, combined with... Figure 13 As shown, Figure 13 The actual motion data of the robotic arm is shown. The robotic arm needs to plan its trajectory from position 0 to the target position goal, which will generate n poses. Since the robotic arm has six joints, the pose of the robotic arm in each pose is represented by six parameters, which are the pose parameters of the joints of the robotic arm.

[0119] S1106: Multimodal fusion of the second hair information, the first hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector is performed to obtain the second model input data.

[0120] S1108: Input the second model input data into the pre-trained hair extraction result judgment model to obtain the hair extraction result.

[0121] Specifically, since the first hair information, the second hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector are four different types of data, the second model input data can be obtained through multimodal parameters, thereby fusing real-time information from multiple modules. The multimodal fusion method can be a model-based fusion strategy or a model-independent fusion strategy.

[0122] In one optional embodiment, the first hair information, the second hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector are multimodally fused to obtain the second model input data. This includes: preprocessing the first hair information, the second hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector to obtain the first hair information, the second hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector with the same dimensions; and multimodally fusing the first hair information, the second hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector with the same dimensions to obtain the second model input data.

[0123] In one embodiment, the second model input data is input into a pre-trained hair extraction result judgment model, and after obtaining the hair extraction result, the following steps are taken: calculating the truncation rate based on the extraction result of each hair.

[0124] Specifically, the truncation rate is equal to the number of hairs truncated / the total number of hairs extracted * 100%, where the number of hairs truncated can be determined based on the actual hair extraction results, and the total number of hairs extracted is the total number of hairs extracted, that is, the number of times the end effector extracts hairs.

[0125] In addition to the above embodiments, the hair extraction results can be judged to automatically calculate the hair truncation rate, eliminating the need for doctors to calculate it themselves and improving the level of automation.

[0126] In one embodiment, acquiring second hair information after hair extraction, acquiring actual motion data of the robotic arm and actual motion data of the end effector includes at least one of the following: acquiring a second hair image after hair extraction, extracting second hair information from the second hair image, and acquiring actual motion data of the robotic arm and actual motion data of the end effector through sensors; or acquiring multiple target hair images collected during the hair extraction process, the multiple target hair images including the second hair image after hair extraction, extracting second hair information from the second hair image, and determining the actual motion data of the robotic arm and the actual motion data of the end effector based on the multiple target hair images.

[0127] This embodiment mainly provides the methods for obtaining the actual motion data of the robotic arm and the actual motion data of the end effector. The first method is to obtain the actual motion data of the robotic arm and the end effector through sensors. The sensors can be motion sensors, which can collect the actual motion data of the robotic arm and the end effector. For example, the actual motion data of the robotic arm can be expressed by the actual motion data of the six joints, and the actual motion data of the end effector can be expressed by the actual motion data collected by the motion sensor installed at the end of the robotic arm.

[0128] The second method involves acquiring multiple target hair images during the hair extraction process. For example, visual odometry can be used to calculate the motion data of the robotic arm. The motion data of the end effector can be obtained from multiple target hair images. For instance, the position of the end effector can be identified in multiple target hair images, and the position of the end effector can be determined according to the time sequence of the multiple target hair images to obtain the motion trajectory of the end effector.

[0129] In one embodiment, the motion data of the end effector includes trajectory data and force data; the actual motion data of the robotic arm and the actual motion data of the end effector are obtained through sensors, including: obtaining the actual motion data of the robotic arm and the trajectory data of the end effector through sensors; obtaining the current corresponding to the end effector through sensors, and calculating the force data of the end effector based on the current.

[0130] In one embodiment, the motion data of the end effector includes trajectory data and force data; determining the actual motion data of the robotic arm and the actual motion data of the end effector based on multiple target hair images includes: determining the actual motion data of the robotic arm and the trajectory data of the end effector based on multiple target hair images; and determining the force data of the end effector based on the deformation of the scalp in the multiple target hair images.

[0131] The motion data of the end effector includes trajectory data and force data. The trajectory data of the actuator can be obtained through sensors or images of the target hair, while the force data can be determined based on the current of the end effector and / or the deformation of the scalp.

[0132] In one embodiment, such as Figure 14 As shown, a method for processing hair extraction results is provided, which can be applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0133] S1402: The hair extraction robot control method in any of the above embodiments controls the movement of the hair extraction robot to extract hair.

[0134] Specifically, in this embodiment, the movement of the robotic arm and end effector is controlled according to the hair extraction robot control method to extract hair, and the hair extraction result is obtained after hair extraction.

[0135] S1404: Acquire the image of the hair to be processed collected during the hair extraction process.

[0136] S1406: Input the hair image to be processed into the pre-trained hair extraction result judgment model to obtain the hair extraction result.

[0137] The hair images to be processed are those collected during the hair extraction process. In this embodiment, the input to the hair extraction result judgment model is the collected hair images to be processed. Multiple hair images to be processed are input into the pre-trained hair extraction result judgment model to obtain the hair extraction result.

[0138] It should be noted that the hair extraction result judgment model in this embodiment is similar to the hair extraction result judgment model mentioned above, except that the input is different. Other limitations can be found above.

[0139] In addition to the above embodiments, the hair extraction results can be judged to automatically calculate the hair truncation rate, eliminating the need for doctors to calculate it themselves and improving the level of automation.

[0140] Specifically, the model training data is used to train the hair extraction result prediction model or the hair extraction result judgment model. The acquisition of model data in this application can include two methods: one is to acquire it in a virtual environment, and the other is to acquire it in a real environment.

[0141] The acquisition of data in a virtual environment can involve building an initial virtual environment, simulating the surgical procedure, and importing models of a hair information acquisition device, a robotic arm, an end effector, and a head into this initial virtual environment. In this initial virtual environment, during each hair retrieval during the simulated surgery, two sets of data are obtained: the first set shows whether truncation occurred after hair retrieval, and the second set shows the truncation probability. By simulating multiple surgeries and recording the surgical data, a deep learning dataset is obtained.

[0142] In a real-world setting, data can be obtained manually by multiple doctors during an actual surgery. By scoring the probability of truncation during each retrieval and determining whether truncation occurred, two types of data are obtained: whether truncation occurred and the probability of truncation. The following sections will provide a detailed description of these two implementation methods.

[0143] In one embodiment, such as Figure 15 As shown, a method for obtaining model training data is provided, which can be applied to... Figure 1Taking the controller in the example, the following steps are included:

[0144] S1502: Import the hair information acquisition device model, robotic arm model, end effector model, and head model into the initial virtual environment to obtain the target virtual environment.

[0145] Specifically, the initial virtual environment can be a virtual surgical model built in ROS (Robot Operating System), which is the target virtual environment. This involves importing a 3D model of a real head and a 3D model of a robotic arm and building a dynamic model of the robotic arm, as well as importing a 3D model of the end effector and building its dynamic model. In addition, a virtual camera system needs to be built to complete the construction of the virtual surgical scene, thereby obtaining the target virtual environment.

[0146] S1504: In the target virtual environment, hair is extracted from the head model through the hair information acquisition device model, the robotic arm model, and the end effector model. The motion data of the end effector model, the motion data of the robotic arm model, the hair information of the target hair, and the hair extraction result of the target hair are recorded during the extraction process.

[0147] S1506: Model training data is obtained based on the motion data of the end effector model, the motion data of the robotic arm model, the hair information of the target hair, and the hair retrieval results of the target hair.

[0148] Specifically, a hair retrieval operation is performed in the target virtual environment, such as simulating 100 hair retrievals. The pose error between the end effector and the entire hair (including subcutaneous tissue) is calculated during hair retrieval, and whether truncation occurs during retrieval is recorded. If no truncation occurs, the data of no truncation, including hair information and the pose error between the end effector and the entire hair (including subcutaneous tissue), is stored for subsequent calculation of the truncation rate. If truncation occurs, the mathematical statistics of all pose errors at the time of calculation are calculated, for example, the average of all pose errors is used as a threshold. The pose error of each hair is then compared with the threshold. If the pose error of a hair is greater than or equal to the threshold, the hair is determined to be truncated, and it is standardized to a first range, such as 0-0.5, and multiplied by 100% to obtain the truncation probability. If the pose error of a hair is less than the threshold, the hair is determined to be untrunched, and it is standardized to a second range, such as 0.5-1, and multiplied by 100% to obtain the truncation probability. In other embodiments, the first and second ranges may take other values, which are not specifically limited here.

[0149] In the above embodiments, the hair retrieval result includes the truncation probability and the result of whether or not the hair was truncated. The truncation probability can be calculated by taking the pose error between the tip of the needle and the entire hair (including the subcutaneous tissue) at each retrieval, calculating the average of all pose errors when truncation occurs, and using this average as a threshold. Based on the threshold, all pose errors are divided into two groups (i.e., the group less than the threshold and the group greater than the threshold). The two groups are then standardized to a first range and a second range, respectively, and the result is multiplied by 100%. The result of whether or not the hair was truncated can be obtained by determining whether the retrieval needle in the virtual environment truncates the virtual hair. If the retrieval needle truncates the virtual hair in the virtual environment, the result is hair truncation; if the retrieval needle does not truncate the virtual hair in the virtual environment, the result is hair not truncated.

[0150] In one embodiment, model training data is obtained based on the motion data of the end effector model, the motion data of the robotic arm model, the hair information of the target hair, and the hair extraction result of the target hair. This includes: calculating the pose difference between the end effector model and the target hair based on the motion data of the end effector model and the hair information of the target hair; when the hair extraction result is the target hair extraction result, obtaining the statistical value of the pose difference corresponding to the target hair extraction result as a threshold; calculating the simulated hair extraction result corresponding to each target hair based on the pose difference of each target hair and the threshold; and using the simulated hair extraction result, the motion data of the end effector model, the motion data of the robotic arm model, and the hair information of the target hair as model training data.

[0151] Specifically, in combination Figure 16 As shown, the target hair extraction result is truncated. The controller calculates the pose difference between the end effector model and the target hair at each extraction and uses the statistical value of the pose difference corresponding to the target hair extraction result as a threshold. Thus, the simulated hair extraction result, i.e. the extraction probability, is calculated based on the relationship between the pose difference between the end effector model and the target hair and the threshold at each extraction.

[0152] In one embodiment, such as Figure 17 As shown, a method for obtaining model training data is provided, which can be applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0153] S1702: Acquire the actual hair information collected by the hair information acquisition device, the motion data of the end effector, and the motion data of the robotic arm during actual operation.

[0154] Specifically, in this embodiment, the data is manually obtained by multiple doctors during the actual surgical procedure. By scoring the probability of truncation during each retrieval and determining whether truncation occurred during each retrieval, two types of data are obtained: whether truncation occurred and the probability of truncation.

[0155] S1704: Receive the actual hair extraction results based on the actual hair information.

[0156] S1706: Model training data is obtained based on the motion data of the end effector, the motion data of the robotic arm, the actual hair information, and the actual hair extraction results.

[0157] Specifically, real-time hair retrieval results can include two types of data: whether truncation occurred and the probability of truncation.

[0158] The cutoff rate is obtained through a doctor's scoring system, ranging from 0 to 100, with 0 being the best (no cutoff) and 100 being the worst (complete cutoff). The determination of whether cutoff has occurred can be made visually or through a trigonoscope. After hair extraction, the pores are rinsed, and then carefully observed. If using a trigonoscope, it can be placed over the pore to obtain a clear, magnified image, which is then visually assessed.

[0159] Specifically combined Figure 18 As shown, in a real-world environment, the robot's camera captures images, while a doctor observes the hair retrieval images from other perspectives. The doctor scores the observed hair retrieval images; for example, the score for truncated hair falls in the third range, and the score for untruncated hair falls in the fourth range. Optionally, a score greater than 50 for truncated hair and less than 50 for untruncated hair is used to obtain the score. Dividing the score by 100 and multiplying by 100% yields the truncation probability. In other embodiments, other scores can be used, simply distinguishing between truncated and untruncated hair. For example, a score of 1 for truncated hair and 0 for untruncated hair. If the doctor's score is greater than or equal to 50, the truncated hair retrieval and its score are obtained; if it is less than 50, the untruncated hair retrieval and its score are obtained. The controller then divides the number of truncated hair retrievals by the total number of hair retrievals and multiplies by 100% to obtain the truncation rate.

[0160] Among them, the combination Figure 21 The diagram illustrates the training process of a hair extraction result prediction model or a hair extraction result judgment model. In this embodiment, the model training data is obtained, and the hair extraction result prediction model is trained using motion data from the end effector, motion data from the robotic arm, actual hair information before extraction, and truncation probability. The hair extraction result judgment model is trained using motion data from the end effector, motion data from the robotic arm, actual hair information before extraction, actual hair information after extraction, and whether truncation (and / or truncation probability).

[0161] In order to enable those skilled in the art to fully understand this application, in conjunction with Figure 20As shown, this application provides a process for optimizing the hair retrieval process through truncation probability. Specifically, the hair information acquisition device obtains first hair information, and the controller calculates the planned motion data of the robotic arm and the planned motion data of the end effector based on the first hair information. This allows the truncation probability to be predicted using a trained hair extraction result prediction model. Optionally, the hair information acquisition device can acquire the first hair information in real time and update the planned motion data of the robotic arm and the end effector by minimizing the truncation probability and using the real-time acquired first hair information to obtain the target motion data of the robotic arm and the end effector. The robotic arm then moves to the hair retrieval position based on its target motion data, thus obtaining the actual motion data of the robotic arm. The end effector performs the hair retrieval operation based on its target motion data, thus obtaining the actual motion data of the end effector. The hair information acquisition device then acquires second hair information after retrieval. The first hair information, the actual motion data of the robotic arm, the actual motion data of the end effector, and the second hair information are then input into the hair extraction result judgment model to determine whether truncation occurred during the retrieval. Finally, the controller calculates the truncation probability based on the total number of truncated hairs and the total number of hairs extracted.

[0162] In the above embodiments, before hair extraction, the first hair information is obtained. This allows the robotic arm's planned motion data and the end effector's planned motion data to be obtained based on the first hair information. Then, the hair extraction result is predicted through model evaluation. The predicted hair extraction result is obtained through quantified data, which is more accurate. Furthermore, the hair extraction result can be judged before each extraction to ensure the accuracy of subsequent hair extraction.

[0163] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0164] Based on the same inventive concept, this application also provides a hair extraction result prediction device, a hair extraction robot control device, a hair extraction result processing device, and a model training data acquisition device for implementing the hair extraction result prediction method, hair extraction robot control method, hair extraction result processing method, and model training data acquisition method described above. The solution provided by this device is similar to the solution described in the above methods. Therefore, the specific limitations of one or more hair extraction result prediction devices, hair extraction robot control devices, hair extraction result processing devices, and model training data acquisition method devices provided in this application can be found in the limitations of the hair extraction result prediction method, hair extraction robot control method, hair extraction result processing method, and model training data acquisition method described above, and will not be repeated here.

[0165] It should be noted that each module in the hair extraction result prediction device, hair extraction robot control device, hair extraction result processing device, and model training data acquisition method device of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0166] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 21 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a hair extraction result prediction method, a hair extraction robot control method, a hair extraction result processing method, and a model training data acquisition method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0167] Those skilled in the art will understand that Figure 21The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0168] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0170] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0171] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A hair extraction system, characterized in that, The hair extraction system includes: Hair information collection device, used to collect hair information from hair; A robotic arm used to move an end effector to a target hair area; An end effector, installed at the end of the robotic arm, is used to perform pick-up and release operations; The controller is used to perform the following steps: Based on the predicted hair extraction results, the planned motion data of the robotic arm and the planned motion data of the end effector of the hair extraction robot are corrected to obtain the target motion data of the robotic arm and the target motion data of the end effector. The predicted hair extraction results are the results of hair extraction based on the planned motion data of the robotic arm and the planned motion data of the end effector before hair extraction, including: obtaining first hair information before hair extraction; the first hair information is used to obtain the planned motion data of the robotic arm and the planned motion data of the end effector. The movement of the robotic arm is controlled according to the target motion data of the robotic arm, and the movement of the end effector is controlled according to the target motion data of the end effector; the second hair information after hair extraction is obtained, and the actual motion data of the robotic arm and the actual motion data of the end effector are obtained; The second hair information, the first hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector are fused using multimodal methods to obtain the second model input data. The second model input data is input into the pre-trained hair extraction result judgment model to obtain the hair extraction result.

2. The system according to claim 1, wherein: After controlling the movement of the robotic arm based on the target motion data of the robotic arm, and controlling the movement of the end effector based on the target motion data of the end effector, the controller is further configured to: During the movement of the robotic arm, the step of acquiring the first hair information before hair extraction continues until the end effector completes hair extraction.

3. The system according to claim 1, wherein: The prediction method for predicting hair extraction results also includes: The first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are fused in a multimodal manner to obtain the first model input data. The input data of the first model is input into the pre-trained hair extraction result prediction model to obtain the predicted hair extraction result.

4. The system according to claim 3, characterized in that, The step of fusing the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector into multimodal data to obtain the first model input data includes: The first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are preprocessed to obtain the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector with the same dimensions. The first model input data is obtained by multimodal fusion of the first hair information of the same dimension, the planned motion data of the robotic arm, and the planned motion data of the end effector.

5. The system according to claim 3, characterized in that, The acquisition of the first hair information before hair extraction includes: A first hair image of the target hair area is acquired using an image acquisition device, and first hair information is obtained by identifying the first hair image; or The first hair information is obtained by locating the hair in the target hair area using a positioning instrument.

6. The system according to claim 1, characterized in that, The process of acquiring the second hair information after hair extraction, acquiring the actual motion data of the robotic arm and the actual motion data of the end effector includes at least one of the following: The second hair image after hair extraction is obtained, the second hair information is extracted from the second hair image, and the actual motion data of the robotic arm and the actual motion data of the end effector are obtained through the sensor. or Multiple target hair images are acquired during the hair extraction process, including a second hair image after hair extraction. Second hair information is extracted from the second hair image. The actual motion data of the robotic arm and the actual motion data of the end effector are determined based on the multiple target hair images.

7. The system according to claim 6, characterized in that, The actual motion data of the end effector includes trajectory data and force data; the actual motion data of the robotic arm and the actual motion data of the end effector acquired through sensors include: The actual motion data of the robotic arm and the trajectory data of the end effector are acquired through sensors; The current corresponding to the end effector is obtained by the sensor, and the force data of the end effector is calculated based on the current.

8. The system according to claim 6, characterized in that, The actual motion data of the end effector includes trajectory data and force data; The step of determining the actual motion data of the robotic arm and the actual motion data of the end effector based on multiple images of the target hair includes: The actual motion data of the robotic arm and the trajectory data of the end effector are determined based on multiple images of the target hair. The force data of the end effector is determined based on the deformation of the scalp in the multiple target hair images.

9. The system according to claim 1, characterized in that, After inputting the second model input data into the pre-trained hair extraction result judgment model to obtain the hair extraction result, the controller is further configured to: The truncation rate is calculated based on the extraction results of each hair.

10. The system according to claim 1, characterized in that, The controller is also used for: Acquire the images of the hair to be processed collected during the hair extraction process; The hair image to be processed is input into a pre-trained hair extraction result judgment model to obtain the hair extraction result.

11. The system according to claim 1, characterized in that, The controller is also used for: Obtain model training data, which is used to train a hair extraction result prediction model or a hair extraction result judgment model, wherein the hair extraction result prediction model is used to predict hair extraction results or the hair extraction result judgment model is used to process hair extraction results.

12. The system according to claim 11, characterized in that, The acquisition of model training data includes: Acquire hair information, end effector motion data, robotic arm motion data, and hair extraction results collected during the hair retrieval process; Model training data is obtained based on the hair information, the motion data of the end effector, the motion data of the robotic arm, and the hair extraction results.

13. The system according to claim 12, characterized in that, The hair information collected during the hair extraction process, the motion data of the end effector, the motion data of the robotic arm, and the hair extraction results include at least one of the following: The hair information acquisition device model, robotic arm model, end effector model, and head model are imported into the initial virtual environment to obtain the target virtual environment. In this target virtual environment, hair is retrieved from the head model using the hair information acquisition device model, robotic arm model, and end effector model. The motion data of the end effector model, the motion data of the robotic arm model, the hair information of the target hair, and the hair retrieval result are recorded during the retrieval process. Acquire the actual hair information collected by the hair information acquisition device, the motion data of the end effector, and the motion data of the robotic arm during the actual operation; receive the actual hair extraction results based on the actual hair information.

14. The system according to claim 13, characterized in that, The model training data obtained based on the hair information, the motion data of the end effector, the motion data of the robotic arm, and the hair extraction results includes at least one of the following: The pose difference between the end effector model and the target hair is calculated based on the motion data of the end effector model and the hair information of the target hair. When the hair retrieval result is the target hair retrieval result, the statistical value of the pose difference corresponding to the target hair retrieval result is obtained as a threshold. The simulated hair extraction result corresponding to each target hair is calculated based on the pose difference of each target hair and the threshold. The simulated hair extraction result, the motion data of the end effector model, the motion data of the robotic arm model, and the hair information of the target hair are used as model training data. or Model training data is obtained based on the motion data of the end effector, the motion data of the robotic arm, the actual hair information, and the actual hair extraction results.

15. A surgical system, characterized in that, Includes the hair extraction system of claim 14.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program stored in the memory, it performs the following steps: Based on the predicted hair extraction results, the planned motion data of the robotic arm and the planned motion data of the end effector of the hair extraction robot are corrected to obtain the target motion data of the robotic arm and the target motion data of the end effector. The predicted hair extraction results are the results of hair extraction based on the planned motion data of the robotic arm and the planned motion data of the end effector before hair extraction, including: obtaining first hair information before hair extraction; the first hair information is used to obtain the planned motion data of the robotic arm and the planned motion data of the end effector. The movement of the robotic arm is controlled according to the target motion data of the robotic arm, and the movement of the end effector is controlled according to the target motion data of the end effector; the second hair information after hair extraction is obtained, and the actual motion data of the robotic arm and the actual motion data of the end effector are obtained; The second hair information, the first hair information, the actual motion data of the robotic arm, and the actual motion data of the end effector are fused using multimodal methods to obtain the second model input data. The second model input data is input into the pre-trained hair extraction result judgment model to obtain the hair extraction result.

17. The computer device according to claim 16, characterized in that, After the processor executes the computer program stored in the memory to control the movement of the robotic arm based on the target motion data of the robotic arm, and to control the movement of the end effector based on the target motion data of the end effector, the process further includes: During the movement of the robotic arm, the step of acquiring the first hair information before hair extraction continues until the end effector completes hair extraction.

18. The computer device according to claim 16, characterized in that, The prediction method for predicting hair extraction results when the processor executes the computer program stored in the memory also includes: The first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are fused in a multimodal manner to obtain the first model input data. The input data of the first model is input into the pre-trained hair extraction result prediction model to obtain the predicted hair extraction result.

19. The computer device according to claim 18, characterized in that, When the processor executes the computer program stored in the memory, the process of performing multimodal fusion of the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector to obtain the first model input data includes: The first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector are preprocessed to obtain the first hair information, the planned motion data of the robotic arm, and the planned motion data of the end effector with the same dimensions. The first model input data is obtained by multimodal fusion of the first hair information of the same dimension, the planned motion data of the robotic arm, and the planned motion data of the end effector.

20. The computer device according to claim 18, characterized in that, The acquisition of first hair information prior to hair extraction, implemented by the processor when executing the computer program stored in the memory, includes: A first hair image of the target hair area is acquired using an image acquisition device, and first hair information is obtained by identifying the first hair image; or The first hair information is obtained by locating the hair in the target hair area using a positioning instrument.

21. The computer device according to claim 16, characterized in that, When the processor executes the computer program stored in the memory, the acquisition of the second hair information after hair extraction, the acquisition of the actual motion data of the robotic arm, and the acquisition of the actual motion data of the end effector include at least one of the following: The second hair image after hair extraction is obtained, the second hair information is extracted from the second hair image, and the actual motion data of the robotic arm and the actual motion data of the end effector are obtained through the sensor. or Multiple target hair images are acquired during the hair extraction process, including a second hair image after hair extraction. Second hair information is extracted from the second hair image. The actual motion data of the robotic arm and the actual motion data of the end effector are determined based on the multiple target hair images.

22. The computer device according to claim 21, characterized in that, The actual motion data of the end effector implemented by the processor when executing the computer program stored in the memory includes trajectory data and force data; The actual motion data of the robotic arm and the actual motion data of the end effector acquired through sensors include: The actual motion data of the robotic arm and the trajectory data of the end effector are acquired through sensors; The current corresponding to the end effector is obtained by the sensor, and the force data of the end effector is calculated based on the current.

23. The computer device according to claim 22, characterized in that, The actual motion data of the end effector involved when the processor executes the computer program stored in the memory includes trajectory data and force data; the determination of the actual motion data of the robotic arm and the actual motion data of the end effector based on multiple target hair images implemented when the processor executes the computer program stored in the memory includes: The actual motion data of the robotic arm and the trajectory data of the end effector are determined based on multiple images of the target hair. The force data of the end effector is determined based on the deformation of the scalp in the multiple target hair images.

24. The computer device according to claim 16, characterized in that, After the processor executes the computer program stored in the memory to input the second model input data into the pre-trained hair extraction result judgment model to obtain the hair extraction result, the process further includes: The truncation rate is calculated based on the extraction results of each hair.

25. The computer device according to claim 16, characterized in that, When the processor executes the computer program stored in the memory, it also performs the following steps: Acquire the images of the hair to be processed collected during the hair extraction process; The hair image to be processed is input into a pre-trained hair extraction result judgment model to obtain the hair extraction result.

26. The computer device according to claim 16, characterized in that, When the processor executes the computer program stored in the memory, it also performs the following steps: Obtain model training data, which is used to train a hair extraction result prediction model or a hair extraction result judgment model, wherein the hair extraction result prediction model is used to predict hair extraction results or the hair extraction result judgment model is used to process hair extraction results.

27. The computer device according to claim 26, characterized in that, The acquisition of model training data implemented by the processor when executing the computer program stored in the memory includes: Acquire hair information, end effector motion data, robotic arm motion data, and hair extraction results collected during the hair retrieval process; Model training data is obtained based on the hair information, the motion data of the end effector, the motion data of the robotic arm, and the hair extraction results.

28. The computer device according to claim 27, characterized in that, When the processor executes the computer program stored in the memory, the acquisition of hair information, end effector motion data, robotic arm motion data, and hair extraction results collected during the hair retrieval process includes at least one of the following: The hair information acquisition device model, robotic arm model, end effector model, and head model are imported into the initial virtual environment to obtain the target virtual environment. In this target virtual environment, hair is retrieved from the head model using the hair information acquisition device model, robotic arm model, and end effector model. The motion data of the end effector model, the motion data of the robotic arm model, the hair information of the target hair, and the hair retrieval result are recorded during the retrieval process. Acquire the actual hair information collected by the hair information acquisition device, the motion data of the end effector, and the motion data of the robotic arm during the actual operation; receive the actual hair extraction results based on the actual hair information.

29. The computer device according to claim 28, characterized in that, The model training data obtained based on the hair information, the motion data of the end effector, the motion data of the robotic arm, and the hair extraction results includes at least one of the following: The pose difference between the end effector model and the target hair is calculated based on the motion data of the end effector model and the hair information of the target hair. When the hair retrieval result is the target hair retrieval result, the statistical value of the pose difference corresponding to the target hair retrieval result is obtained as a threshold. The simulated hair extraction result corresponding to each target hair is calculated based on the pose difference of each target hair and the threshold. The simulated hair extraction result, the motion data of the end effector model, the motion data of the robotic arm model, and the hair information of the target hair are used as model training data. or Model training data is obtained based on the motion data of the end effector, the motion data of the robotic arm, the actual hair information, and the actual hair extraction results.

Images