Robotic positioning method and system

Abstract

The application relates to a robot positioning method and system. The method comprises the following steps: a server acquires a target part image of a user collected by an image collection device and a medical image collected by a medical device, determines a registration relationship between the target part image and the medical image according to the target part image, the medical image and a preset registration method, and determines positioning information of a robot based on the registration relationship and a preset mapping relationship, wherein the preset mapping relationship comprises a mapping relationship between a position of the robot and a position of the image collection device. In the scheme, additional markers do not need to be pasted or set on the user, thereby avoiding additional harm to the user, a mapping relationship among the robot, the image collection device and the medical device is constructed through the preset registration method, positioning operation of the robot is realized by referring to the medical image, and preoperative planning of the robot is more accurate.

Description

[0001] This invention patent application is a divisional application of Chinese invention patent application filed on May 10, 2021, with application number 2021105057326 and title "Robot Positioning Method, Apparatus, System and Computer Equipment". Technical Field

[0002] This application relates to the field of robotics technology, and in particular to a robot positioning method and system. Background Technology

[0003] Current neurosurgical techniques can be broadly categorized into two types: marker-based and markerless. Marker-based techniques primarily involve implanting markers in the patient's skull or attaching markers to the head. The patient wears these markers during preoperative imaging scans, and the markers' corresponding positional information is determined in both image space and physical space. Based on the transformation between image space and physical space, the robot's preoperative planning is completed.

[0004] However, these markers often cause additional harm to patients, and once the markers are displaced relative to the patient's head in the preoperative images, it will cause data deviation in the robot's preoperative planning, resulting in inaccurate robot positioning. Summary of the Invention

[0005] Therefore, it is necessary to provide a robot positioning method and system to address the aforementioned technical problems.

[0006] Firstly, a robot localization method is provided, the method comprising:

[0007] Acquire images of the user's target body part captured by image acquisition devices and medical images captured by medical devices;

[0008] Based on the target area image, medical image, and preset registration method, determine the registration relationship between the target area image and the medical image;

[0009] Based on the registration relationship and the preset mapping relationship, the robot's positioning information is determined; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0010] In one embodiment, determining the registration relationship between the target area image and the medical image based on the target area image, the medical image, and a preset registration method includes:

[0011] Determine a predetermined number of reference points from medical images and obtain the location information of these reference points;

[0012] Based on the location information of each reference point, the target point corresponding to each reference point is determined from the target area image;

[0013] Based on the correspondence between the target point and the reference point, the registration relationship between the target area image and the medical image is determined.

[0014] In one embodiment, determining the target point corresponding to each reference point from the target region image based on the position information of each reference point includes:

[0015] A reference point group is determined based on the reference points; the reference point group includes at least four reference points.

[0016] Based on the position information of each reference point, determine the vector information of the reference point group; the vector information includes the distance and direction between each pair of reference points.

[0017] Based on the vector information of the reference point group, the target point group corresponding to the reference point group is determined from the target part image;

[0018] The target point corresponding to each reference point is determined based on the target point group.

[0019] In one embodiment, determining the target point group corresponding to the reference point group from the target region image based on the vector information of the reference point group includes:

[0020] Obtain the first distance between any two reference points in a reference point group in a medical image;

[0021] Obtain the second distance between each pair of candidate points in multiple candidate point groups in the target region image;

[0022] Calculate the first deviation between the first distance and the second distance;

[0023] The candidate point group with the smallest first deviation is determined as the target point group.

[0024] In one embodiment, after determining the registration relationship between the target area image and the medical image based on the correspondence between the target point and the reference point, the method further includes:

[0025] Obtain the positional information of each candidate point in the target region image;

[0026] Based on the location information of each candidate point, the location information of each reference point, and the preset ICP algorithm, the target point corresponding to each reference point is determined;

[0027] Based on the correspondence between the target point and the reference point, adjust the registration relationship between the target area image and the medical image.

[0028] In one embodiment, the above-mentioned acquisition of images of the user's target area via an image acquisition device includes:

[0029] Two-dimensional and depth images of the user are acquired using image acquisition equipment;

[0030] The target region is determined based on the two-dimensional image and the depth image.

[0031] In one embodiment, determining the target region image based on the two-dimensional image and the depth image includes:

[0032] Candidate feature points in a two-dimensional image are obtained based on a preset feature point extraction algorithm.

[0033] Based on the mapping relationship between the two-dimensional image and the depth image, the target feature points corresponding to the candidate feature points in the depth image are determined;

[0034] An image of the target region is generated based on the three-dimensional information of the target feature points.

[0035] Secondly, a robot positioning device is provided, the device comprising:

[0036] The acquisition module is used to acquire images of the user's target body part captured by the image acquisition device and medical images captured by the medical device;

[0037] The registration module is used to determine the registration relationship between the target area image and the medical image based on the target area image, the medical image, and the preset registration method.

[0038] The positioning module is used to determine the robot's positioning information based on the registration relationship and the preset mapping relationship; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0039] Thirdly, a robot positioning system is provided, which includes: a server, an image acquisition device, a medical device, and a target robot;

[0040] A server is used to execute the robot localization method provided in the first aspect above;

[0041] Medical equipment used to collect medical images from users;

[0042] An image acquisition device is used to acquire images of the user's target body parts; and the location of the image acquisition device is related to the location of the target robot.

[0043] Fourthly, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the robot localization method described in any of the first aspects above.

[0044] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the robot localization method described in any of the first aspects above.

[0045] The aforementioned robot positioning method and system involve a server acquiring images of the user's target area from an image acquisition device and medical images from a medical device. Based on the target area image, the medical image, and a preset registration method, the server determines the registration relationship between the two. Based on this registration relationship and a preset mapping relationship, the robot's positioning information is determined. The preset mapping relationship includes the mapping between the robot's position and the image acquisition device's position. This solution eliminates the need for additional markers or settings on the user, avoiding any potential harm. The preset registration method establishes a mapping relationship between the robot, the image acquisition device, and the medical device, enabling robot positioning by referencing medical images and resulting in more accurate preoperative planning for the robot. Attached Figure Description

[0046] Figure 1 This is a diagram illustrating the application environment of a robot localization method in one embodiment;

[0047] Figure 2 This is a flowchart illustrating a robot localization method in one embodiment;

[0048] Figure 3 This is a flowchart illustrating a robot localization method in one embodiment;

[0049] Figure 4 This is a flowchart illustrating a robot localization method in one embodiment;

[0050] Figure 5 This is a flowchart illustrating a robot localization method in one embodiment;

[0051] Figure 6 This is a schematic diagram of the registration algorithm in a robot localization method in one embodiment;

[0052] Figure 7 This is a schematic diagram of the registration algorithm in a robot localization method in one embodiment;

[0053] Figure 8 This is a flowchart illustrating a robot localization method in one embodiment;

[0054] Figure 9 This is a flowchart illustrating a robot localization method in one embodiment;

[0055] Figure 10 This is a flowchart illustrating a robot localization method in one embodiment;

[0056] Figure 11 This is a schematic diagram of feature extraction in a robot localization method in one embodiment;

[0057] Figure 12 This is a schematic diagram of feature extraction in a robot localization method in one embodiment;

[0058] Figure 13 This is a flowchart illustrating a robot localization method in one embodiment;

[0059] Figure 14 This is a schematic diagram of the robot positioning system in one embodiment;

[0060] Figure 15 This is a structural block diagram of a robot positioning device in one embodiment;

[0061] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The robot localization method provided in this application can be applied to, for example... Figure 1 In the application environment shown, server 101 communicates with medical device 102 and image acquisition device 103 via a network. Server 101 can be a standalone server or a server cluster consisting of multiple servers; medical device 102 can be a CT scanner or other medical equipment; image acquisition device 103 can be a depth camera, a phase laser acquisition device, or a point laser acquisition device; the image acquisition device can be located at the end of the surgical robot's operating arm or at a fixed location in the operating room. For example, in a neurosurgical operating room scenario, image acquisition device 103 can be a depth camera fixed to the end of the surgical robot's operating arm for acquiring images of the user's face; the medical device can be a CT scanner for acquiring CT images of the user's brain, but this embodiment does not limit this.

[0064] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. It should be noted that this application... Figures 2-13The robot localization method provided in this embodiment can be executed by a server, or it can be a robot localization device. The robot localization device can be part or all of the server through software, hardware, or a combination of software and hardware. In the following method embodiments, the execution subject is always described using a server as an example.

[0065] In one embodiment, such as Figure 2 As shown, a robot localization method is provided, which involves a server acquiring images of a user's target body part captured by an image acquisition device and medical images captured by a medical device. Based on the target body part image, the medical images, and a preset registration method, the registration relationship between the target body part image and the medical images is determined. Based on the registration relationship and a preset mapping relationship, the robot's localization information is determined. The method includes the following steps:

[0066] S201. Acquire images of the user's target body part captured by the image acquisition device and medical images captured by the medical device.

[0067] For example, when the image acquisition device is a depth camera, the acquired image of the user's target area can be a depth map of the target area. The depth camera can also acquire two-dimensional RGB images, so the target area image can also be a two-dimensional RGB image of the target area. Alternatively, the target area image can be an image determined based on the depth map and the two-dimensional RGB image of the target area. For example, in a neurosurgical scenario, the target area can be the user's face.

[0068] The medical device can be a computed tomography (CT) scanner. Optionally, during the process of acquiring medical images using the medical device, CT image data of the user's scanned area can be acquired. Based on the CT image data, CT image data of the target area can be determined. Then, three-dimensional reconstruction can be performed based on the CT image data of the target area to obtain the medical image. For example, based on the user's CT image data, the CT image data corresponding to the user's face can be segmented and determined. Then, three-dimensional reconstruction can be performed based on the segmented CT image data corresponding to the face to obtain the medical image of the target area.

[0069] In this embodiment, the server communicates with the image acquisition device to obtain image data acquired by the image acquisition device. For example, the server can acquire the depth map and two-dimensional RGB image acquired by the depth camera, and determine the target area image based on the depth map and two-dimensional RGB image. The server also communicates with the medical device to obtain medical image data acquired by the medical device. For example, the server can perform image segmentation and three-dimensional reconstruction based on the medical image data to determine the medical image corresponding to the medical device. This embodiment does not limit this.

[0070] S202. Determine the registration relationship between the target area image and the medical image based on the target area image, the medical image, and the preset registration method.

[0071] The preset registration method can include a variety of registration methods, including global registration and local registration. Global registration can be based on the correspondence between the target area image and the plane in the medical image, while local registration can be based on the correspondence between the target area image and each point in the medical image.

[0072] In this embodiment, after receiving the target area image and the medical image, the server can perform registration using various preset registration methods. For example, global registration can be used. Global registration determines the planar correspondence between the target area image and the medical image. It should be noted that three points are sufficient for coplanarity; however, to more accurately determine the positional relationship, distance, and direction of points in the reference plane of the medical image, four points can be identified in the medical image to form a reference plane. Based on this reference plane, the target plane formed by the corresponding four points in the target area image is then determined. Optionally, the registration relationship between the target area image and the medical image is determined based on the correspondence between the target plane in the target area image and the reference plane in the medical image. Optionally, local registration can be used, i.e., multiple reference points are determined from the medical image, and target points corresponding to these reference points are determined from the target area image based on these reference points. The registration relationship between the target area image and the medical image is then adjusted based on the correspondence between the reference points and the target points. This embodiment does not limit this approach.

[0073] S203. Based on the registration relationship and the preset mapping relationship, determine the robot's positioning information; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0074] The mapping relationship between the robot's position and the image acquisition device's position is determined through a preset calibration method. Optionally, the server can construct an intermediate reference object and determine the positional correspondence between the image acquisition device and the robot based on the positional correspondence between the image acquisition device and the intermediate reference object, as well as the positional correspondence between the robot and the intermediate reference object. The image acquisition device can be mounted on the robot's main body, for example, at the end of the robot's manipulator arm. Alternatively, the image acquisition device can be mounted at any fixed location in the operating room, and the server determines the mapping relationship between the robot's position and the image acquisition device's position based on the specific location of the image acquisition device, the robot's position, and the position of the intermediate reference object.

[0075] In this embodiment, the server can determine the mapping relationship between the position of the medical device and the position of the image acquisition device based on the registration relationship between the target area image and the medical image. Thus, based on the mapping relationship between the position of the robot and the position of the image acquisition device, the server can determine the mapping relationship between the position of the medical device and the position of the robot. Based on the mapping relationship between the position of the medical device and the position of the robot, the preoperative planning and positioning of the robot can be realized based on the medical image. This embodiment does not limit this.

[0076] In the aforementioned robot localization method, the server acquires images of the user's target area from an image acquisition device and medical images from a medical device. Based on the target area image, the medical image, and a preset registration method, the server determines the registration relationship between the target area image and the medical image. Based on this registration relationship and a preset mapping relationship, the robot's localization information is determined. The preset mapping relationship includes the mapping relationship between the robot's position and the image acquisition device's position. This solution eliminates the need for additional markers or settings on the user, avoiding any additional harm. The preset registration method establishes a mapping relationship between the robot, the image acquisition device, and the medical device, enabling robot localization by referencing medical images and making preoperative planning for the robot more accurate.

[0077] The server can employ various registration methods to register images of target areas with medical images. In one embodiment, such as... Figure 3 As shown, the above determination of the registration relationship between the target area image and the medical image based on the target area image, the medical image, and the preset registration method includes:

[0078] S301. Determine a preset number of reference points from medical images and obtain the location information of the reference points.

[0079] In this embodiment, the server determines a preset number of reference points from the medical image. It should be noted that during the registration process, registration can be performed based on the correspondence between the target plane in the target area image and the reference plane in the medical image. While three points are sufficient for coplanarity, to more accurately determine the positional relationships, distances, and directions between points on the reference plane in the medical image, the preset number can be four. The server acquires four reference points from the medical image and determines the corresponding positional information of the reference points based on the coordinate system of the medical image. This embodiment does not impose limitations on this.

[0080] S302. Based on the position information of each reference point, determine the target point corresponding to each reference point from the target part image.

[0081] In this embodiment, after determining the location information of the reference point, the server searches for the point in the target area image that is closest to the reference point as the target point. Optionally, the server can obtain the location information of each point in the target area image, and determine the point that is closest to the reference point as the target point based on the location information of each point and the location information of the reference point. The server can determine the corresponding reference point and target point on similar planes based on the principle that four points are coplanar; the server can also determine the target point corresponding to the reference point based on a preset nearest point search method (Iterative Closest Point, ICP algorithm), but this embodiment does not limit this method.

[0082] S303. Based on the correspondence between the target point and the reference point, determine the registration relationship between the target area image and the medical image.

[0083] In this process, after determining the target points in the target area image that correspond to the reference points, the server acquires the position information of each target point. Based on the position information of the target points and the reference points, it determines the transformation relationship between their position information, that is, it determines the correspondence between the target points and the reference points. Optionally, the server determines a transformation matrix between the reference points and the target points based on the coordinates of the reference points and the target points. This transformation matrix represents the transformation relationship between the coordinate system of the reference points and the coordinate system of the target points. Further, this transformation relationship can also represent the registration relationship between the target area image and the medical image; this embodiment does not limit this aspect.

[0084] In this embodiment, the server determines the target point from the target area image based on the reference point in the medical image, and then determines the corresponding registration relationship based on the position information of the reference point and the target point. This method involves a small amount of data, a simple processing procedure, and a short processing time, and can effectively obtain the registration relationship between the target area image and the medical image.

[0085] In one scenario, the server can form a reference plane based on reference points in medical images to determine the target point corresponding to the target plane in the target area image. In one embodiment, such as... Figure 4 As shown, the above-mentioned method of determining the target point corresponding to each reference point from the target area image based on the position information of each reference point includes:

[0086] S401. Determine a reference point group based on the reference points; the reference point group includes at least four reference points.

[0087] In this embodiment, the server determines at least four reference points from the medical image to form a reference point group. Optionally, the server uses the principle of four points being coplanar to determine the platform on which the at least four reference points are located, thereby defining the at least four reference points as a reference point group.

[0088] S402. Determine the vector information of the reference point group based on the position information of each reference point; the vector information includes the distance and direction between each pair of reference points.

[0089] In this embodiment, after determining the reference point group, the server can obtain the position information of all reference points in the reference point group, and then determine the vector information of the reference point group based on the position information of each reference point. Optionally, the server can calculate the distance between any two reference points based on their coordinates. These two reference points can be adjacent or non-adjacent reference points; this embodiment does not limit this.

[0090] S403. Based on the vector information of the reference point group, determine the target point group corresponding to the reference point group from the target part image.

[0091] In this embodiment, the server determines the target point group corresponding to the reference point group from the target part image based on the vector information of the calculated reference point group. Optionally, the server can determine multiple candidate point groups from the target part image, and the number of candidate points in each candidate point group is the same as the number of reference points in the reference point group. The candidate point group with the closest relationship to the reference point group is determined as the target point group.

[0092] S404. Determine the target point corresponding to each reference point based on the target point group.

[0093] In this embodiment, after determining the target point group, the server determines the target point corresponding to each reference point through the positional correspondence between the points.

[0094] In this embodiment, the server uses the principle of four points being coplanar to determine the target point group corresponding to the reference point group in the medical image from the target area image, and then determines the target point of the target point group. The four-point coplanar method is simple to implement and improves the efficiency of determining the target point. At the same time, based on the principle of four points being coplanar, it also ensures the accuracy of target point selection.

[0095] Optionally, in determining the target point group corresponding to the reference point group, the server can determine the target point group by the distance between two points. In one embodiment, such as... Figure 5 As shown, the above-mentioned method of determining the target point group corresponding to the reference point group from the target region image based on the vector information of the reference point group includes:

[0096] S501. Obtain the first distance between any two reference points in the reference point group in the medical image.

[0097] In this embodiment, as Figure 6As shown, the reference point group in the medical image includes four points a, b, c, and d, which form the S1 plane. The server can calculate the distances between ab, ac, ad, bc, bd, and cd based on the position information of the four points a, b, c, and d, and use this distance as the first distance.

[0098] S502. Obtain the second distance between each pair of candidate points in multiple candidate point groups in the target area image.

[0099] In this embodiment, as Figure 7 As shown, the candidate point group in the target area image includes four points a', b', c', and d', which form the S2 plane. The server can calculate the distances between a'-b', a'-c', a'-d', b'-c', b'-d', and c'-d' based on the position information of the four points a', b', c', and d', and use these distances as the second distance.

[0100] S503. Calculate the first deviation between the first distance and the second distance.

[0101] In this embodiment, the server calculates the corresponding first deviation based on the obtained first distance and second distance. For example, the server calculates the first deviation between ab and a'-b', and the first deviation between ac and a'-c'. This embodiment does not limit this.

[0102] S504. The candidate point group with the smallest first deviation is determined as the target point group.

[0103] In this embodiment, there may be multiple candidate point groups in the target area image. The first deviation corresponding to each candidate point in each candidate point group is calculated. Optionally, the server can also use the average value of the first deviation of each candidate point as the deviation of each candidate point group, and determine the candidate point group with the smallest deviation as the target point group. This embodiment does not limit this.

[0104] In this embodiment, the target point group is determined based on the distance between each pair of reference points in the reference point group and the distance between each pair of candidate points in the candidate point group. Since the position information of each reference point and each candidate point is known, the distance calculation process is simple and can effectively determine the target point group corresponding to the reference group.

[0105] In another scenario, the server can directly map the reference point to the target point; the method provided in this embodiment can be executed as described above. Figure 4 Following the steps of the provided embodiments, it can also be used as a reference to... Figure 4 The provided embodiments present parallel implementation schemes, in one of the embodiments, such as Figure 8 As shown, after determining the registration relationship between the target area image and the medical image based on the correspondence between the target point and the reference point, the method further includes:

[0106] S601. Obtain the position information of each candidate point in the target area image.

[0107] In this embodiment, the server can obtain the position information of each candidate point in the target area image. The target area is a depth map, and the server can also obtain the depth value information of each candidate point. This embodiment does not limit this.

[0108] S602. Based on the location information of each candidate point, the location information of each reference point, and the preset ICP algorithm, determine the target point corresponding to each reference point.

[0109] The ICP algorithm refers to the Iterative Closest Point method. In this embodiment, the server, based on the ICP algorithm, determines the target point closest to the reference point according to the position information of each reference point in the medical image and the position information of each candidate point in the target area image. Optionally, the server determines the reference point from the medical image, searches for candidate points in the target area image as the corresponding closest points, calculates the rotation and translation matrices based on the reference point and candidate points, determines the new target point based on the reference point, rotation and translation matrices, calculates the distance between the new target point and the candidate points, and stops iterating when the distance is less than a preset threshold or the number of iterations is equal to the preset threshold, thus obtaining the target point corresponding to the reference point. This embodiment does not limit this step.

[0110] S603. Adjust the registration relationship between the target area image and the medical image based on the correspondence between the target point and the reference point.

[0111] In this embodiment, the server can obtain the correspondence between the target point and the reference point based on the determined target point. According to the registration relationship between the target part image and the medical image obtained in step 303 above, the server can further adjust the registration relationship between the target part image and the medical image by combining the correspondence between the target point and the reference point, so as to make the registration relationship more accurate.

[0112] In this embodiment, the server determines the target point corresponding to the reference point from the target area image based on the location information of the reference point in the medical image. The determination process is simple, and the target point determined by this method is relatively accurate.

[0113] The server acquires images of the target area through an image acquisition device. To further improve the accuracy of the data acquired by the image acquisition device, in one embodiment, such as... Figure 9 As shown, the above-mentioned acquisition of images of the user's target body parts via image acquisition device includes:

[0114] S701: Acquire two-dimensional and depth images of the user through an image acquisition device.

[0115] In this embodiment, before acquiring the user's two-dimensional and depth images through the image acquisition device, optionally, the user's surgical position can be determined in advance according to the preoperative plan, keeping the user's head fixed, and adjusting the position of the image acquisition device so that the user's target part is completely within the acquisition field of view of the image acquisition device. For example, the position of the image acquisition device can be adjusted so that the image acquisition device is perpendicular to the user's face. This embodiment does not limit this.

[0116] Optionally, if the image acquisition device is a depth camera, it can simultaneously acquire the user's 2D image and depth image; if the image acquisition device is another laser acquisition device, it needs to be modified so that the server can acquire the user's 2D image and depth image through the laser acquisition device.

[0117] S702. Determine the target area image based on the two-dimensional image and the depth image.

[0118] In this embodiment, the server can determine the target part based on the two-dimensional image, and determine the region corresponding to the target part in the depth image based on the correspondence between the two-dimensional image and the depth image in the image acquisition device, thereby forming an image of the target part based on the three-dimensional information of the acquired region.

[0119] In this embodiment, the server acquires the user's two-dimensional image and depth image through an image acquisition device. The two-dimensional image is a single image, and the depth image is also a single image. The processing data volume is small, saving data processing time and resources. Furthermore, the target part image of the user is determined based on the two different dimensions of the image, and the target part image is relatively accurate.

[0120] Optionally, after receiving images of different dimensions acquired by the image acquisition device, in one embodiment, such as Figure 10 As shown, the above method of determining the target region image based on the two-dimensional image and the depth image includes:

[0121] S801. Obtain candidate feature points in the two-dimensional image according to the preset feature point extraction algorithm.

[0122] The preset feature point extraction algorithm can be any image feature extraction model, such as extracting features from a two-dimensional image using a preset neural network model.

[0123] In this embodiment, the server inputs a two-dimensional image into a feature extraction model and outputs multiple candidate feature points in the two-dimensional image. For example, the two-dimensional image can be a facial image. Inputting the two-dimensional image into the feature extraction model can obtain candidate feature points for target areas in the two-dimensional image, such as candidate feature points for the eyes, mouth, eyebrows, and nose. Figure 11 As shown.

[0124] S802. Based on the mapping relationship between the two-dimensional image and the depth image, determine the target feature points in the depth image that correspond to the candidate feature points.

[0125] The mapping relationship between the two-dimensional image and the depth image is determined based on the parameters of the image acquisition device.

[0126] In this embodiment, the purpose of the server extracting target feature points is to determine the static regions of the user's face. It should be noted that the user's face is a region with complex facial expressions. To minimize variations in the resulting image of the target region and improve its accuracy, this embodiment determines the static regions of the face in a two-dimensional image and obtains candidate feature points within those regions. Generally, static facial regions refer to areas less affected by facial expressions or areas close to bony structures, such as the forehead and bridge of the nose.

[0127] Optionally, facial expression shape data from different individuals can be collected, and the areas where facial expression changes are less can be statistically analyzed and calculated as static facial regions. Alternatively, static facial regions can be determined using physiological structural information, for example, defining the areas of the face closest to bone structures as static facial regions. Figure 12 As shown, the shaded area can be understood as the static facial region. After determining the static facial region as the target region, the server obtains the candidate feature points included in the target region. Based on the mapping relationship between the two-dimensional image and the depth image, the server determines the target feature points corresponding to the candidate feature points from the depth image. This implementation does not impose any limitations on this.

[0128] S803. Generate an image of the target region based on the three-dimensional information of the target feature points.

[0129] In this embodiment, the server obtains the three-dimensional information of each target feature point based on the target feature points in the determined depth map, and generates an image of the target area based on the three-dimensional information of all target feature points.

[0130] In this embodiment, the server determines the static facial region that is less affected by facial expression changes as the target region based on the two-dimensional image. Then, based on the candidate feature points in the target region in the two-dimensional image and the mapping relationship between the two-dimensional image and the depth image, the server determines the target feature points in the depth image, thereby generating the target area image. Since the target region is a static facial region, the impact of facial expression changes on the target area image is reduced. Based on the target area image and the medical image, the registration accuracy and precision of the registration result are improved.

[0131] To better illustrate the above methods, such as Figure 13 As shown, this embodiment provides a robot localization method, specifically including:

[0132] S101. Acquire the user's two-dimensional image and depth image using an image acquisition device;

[0133] S102. Obtain candidate feature points in the two-dimensional image according to the preset feature point extraction algorithm;

[0134] S103. Based on the mapping relationship between the two-dimensional image and the depth image, determine the target feature points in the depth image that correspond to the candidate feature points;

[0135] S104. Generate an image of the target region based on the three-dimensional information of the target feature points;

[0136] S105. Determine a preset number of reference points from medical images and obtain the location information of the reference points;

[0137] S106. Based on the position information of each reference point, determine the target point corresponding to each reference point from the target part image;

[0138] S107. Determine the registration relationship between the target area image and the medical image based on the correspondence between the target point and the reference point;

[0139] S108. Based on the registration relationship and the preset mapping relationship, determine the robot's positioning information; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0140] In this embodiment, no additional markers need to be pasted or set for the user, thus avoiding any additional harm to the user. By using a preset registration method, a mapping relationship is established between the robot, the image acquisition device, and the medical device, enabling the robot to perform positioning operations by referring to medical images, making the robot's preoperative planning more accurate.

[0141] The robot localization method provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0142] It should be understood that, although Figure 2-13 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-13 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0143] In one embodiment, a robot positioning system is provided, such as Figure 14 As shown, the system includes: server 101, image acquisition device 103, medical device 102, and target robot 104.

[0144] Among them, medical equipment 102 collects the user's medical images, and image acquisition device 103 collects images of the user's target area.

[0145] In this embodiment, the correspondence between the positions of the image acquisition device 103 and the target robot 104 can be determined based on the position of the image acquisition device 103, the position of the target robot 104, and the calibration parameters of the image acquisition device 103. Therefore, the server can achieve the above-mentioned functionality based on the medical images of the user acquired by the medical device 102, the images of the user's target body parts acquired by the image acquisition device 103, and the correspondence between the positions of the image acquisition device 103 and the target robot 104. Figures 2 to 13 The robot positioning method provided in this embodiment enables the positioning operation of the target robot 104.

[0146] In this embodiment, the server can be set at any location on-site or in any location in the field, as long as it can communicate normally with the medical device 102 and the image acquisition device 103. The image acquisition device 103 can be set on the target robot 104 itself, for example, at the end of the operating arm of the target robot 104, or it can be set independently of the target robot 104, for example, at a fixed location in the operating room. Based on the same reference coordinates, the mapping relationship between the reference system where the image acquisition device 103 is located and the reference system where the target robot 104 is located is determined, that is, the association relationship between the position of the image acquisition device 103 and the position of the target robot 104 is obtained. This embodiment does not limit this.

[0147] In this embodiment, based on the mutual communication between the devices in the robot positioning system, there is no need to attach or set additional markers for the user, thus avoiding additional harm to the user. Through a preset registration method, a mapping relationship is constructed between the robot, the image acquisition device, and the medical device, enabling the robot to perform positioning operations by referring to medical images, making the robot's preoperative planning more accurate.

[0148] The robot positioning system provided in this embodiment is similar in principle and technical effect to the method embodiment described above, and will not be repeated here.

[0149] In one embodiment, such as Figure 15 As shown, a robot positioning device is provided, including: an acquisition module 01, a registration module 02, and a positioning module 03, wherein:

[0150] The acquisition module 01 is used to acquire images of the user's target body part captured by the image acquisition device and medical images captured by the medical device;

[0151] The registration module 02 is used to determine the registration relationship between the target area image and the medical image based on the target area image, the medical image, and the preset registration method.

[0152] The positioning module 03 is used to determine the robot's positioning information based on the registration relationship and the preset mapping relationship; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0153] In one embodiment, the registration module is used to determine a preset number of reference points from the medical image and obtain the location information of the reference points; determine the target point corresponding to each reference point from the target area image based on the location information of each reference point; and determine the registration relationship between the target area image and the medical image based on the correspondence between the target point and the reference point.

[0154] In one embodiment, the registration module is used to determine a reference point group based on reference points; the reference point group includes at least four reference points; determine vector information of the reference point group based on the position information of each reference point; the vector information includes the distance and direction between each pair of reference points; determine a target point group corresponding to the reference point group from the target part image based on the vector information of the reference point group; and determine the target point corresponding to each reference point based on the target point group.

[0155] In one embodiment, the registration module is used to obtain a first distance between pairs of reference points in a reference point group in a medical image; obtain a second distance between pairs of candidate points in multiple candidate point groups in a target area image; calculate a first deviation between the first distance and the second distance; and determine the candidate point group with the smallest first deviation as the target point group.

[0156] In one embodiment, the registration module is further configured to acquire the position information of each candidate point in the target area image; determine the target point corresponding to each reference point based on the position information of each candidate point, the position information of each reference point and the preset ICP algorithm; and adjust the registration relationship between the target area image and the medical image based on the correspondence between the target point and the reference point.

[0157] In one embodiment, the acquisition module is used to acquire a two-dimensional image and a depth image of the user through an image acquisition device; and to determine the target area image based on the two-dimensional image and the depth image.

[0158] In one embodiment, the acquisition module is used to acquire candidate feature points in a two-dimensional image according to a preset feature point extraction algorithm; determine target feature points in the depth image corresponding to the candidate feature points according to the mapping relationship between the two-dimensional image and the depth image; and generate a target part image according to the three-dimensional information of the target feature points.

[0159] For specific limitations regarding the robot localization device, please refer to the limitations on the robot localization method above, which will not be repeated here. Each module in the aforementioned robot localization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0160] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a robot localization method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0161] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0163] Acquire images of the user's target body part captured by image acquisition devices and medical images captured by medical devices;

[0164] Based on the target area image, medical image, and preset registration method, determine the registration relationship between the target area image and the medical image;

[0165] Based on the registration relationship and the preset mapping relationship, the robot's positioning information is determined; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0166] The computer device provided in the above embodiments has similar implementation principles and technical effects to the above method embodiments, and will not be described again here.

[0167] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0168] Acquire images of the user's target body part captured by image acquisition devices and medical images captured by medical devices;

[0169] Based on the target area image, medical image, and preset registration method, determine the registration relationship between the target area image and the medical image;

[0170] Based on the registration relationship and the preset mapping relationship, the robot's positioning information is determined; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

[0171] The computer-readable storage medium provided in the above embodiments has similar implementation principles and technical effects to the above method embodiments, and will not be described again here.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A robot positioning method, characterized by, The method includes: Two-dimensional and depth images of the user are acquired using image acquisition equipment; Candidate feature points in the two-dimensional image are obtained according to a preset feature point extraction algorithm; Based on the mapping relationship between the two-dimensional image and the depth image, the target feature point in the depth image corresponding to the candidate feature point is determined; Based on the three-dimensional information of the target feature points, an image of the target region is generated; Acquire medical images captured by medical devices; A predetermined number of reference points are determined from the medical image, and the position information of the reference points is obtained; based on the position information of each reference point, a target point corresponding to each reference point is determined from the target area image; based on the correspondence between the target point and the reference point, the registration relationship between the target area image and the medical image is determined. The location information of each candidate point in the target area image is obtained; the target point corresponding to each reference point is determined according to the location information of each candidate point, the location information of each reference point, and a preset ICP algorithm; the registration relationship between the target area image and the medical image is adjusted according to the correspondence between the target point and the reference point. Based on the registration relationship and the preset mapping relationship, the robot's positioning information is determined; the preset mapping relationship includes the mapping relationship between the robot's position and the position of the image acquisition device.

2. The method of claim 1, wherein, The step of determining the target feature point in the depth image corresponding to the candidate feature point based on the mapping relationship between the two-dimensional image and the depth image includes: In the two-dimensional image, the static facial region of the user is determined as the target region; the static facial region is a region that is not easily affected by changes in facial expressions, or a region of the face that is close to the bone structure. Obtain candidate feature points in the target region; Based on the mapping relationship, target feature points corresponding to the candidate feature points are determined from the depth image.

3. The method of claim 2, wherein, By collecting shape data of different facial expressions, the areas with smaller changes in facial expression are calculated based on the statistical analysis of the shape data as the static facial areas. Alternatively, static facial regions can be determined using physiological structural information.

4. The method of claim 2, wherein, The static facial areas of the user are the forehead and bridge of the nose.

5. The method of claim 1, wherein, The step of determining the target point corresponding to each of the reference points from the target region image based on the position information of each of the reference points includes: A set of reference points is determined based on the reference points; the set of reference points includes at least four reference points. Based on the position information of each reference point, the vector information of the reference point group is determined; the vector information includes the distance and direction between each pair of reference points. Based on the vector information of the reference point group, determine the target point group corresponding to the reference point group from the target part image; The target point corresponding to each reference point is determined based on the target point group.

6. The method of claim 5, wherein, The step of determining the target point group corresponding to the reference point group from the target region image based on the vector information of the reference point group includes: Obtain the first distance between any two reference points in the reference point group in the medical image; Obtain the second distance between each pair of candidate points in a group of multiple candidate points in the target region image; Calculate the first deviation between the first distance and the second distance; The candidate point group with the smallest first deviation is determined as the target point group.

7. A robot positioning system characterized in that, The system includes: a server, image acquisition equipment, medical equipment, and a target robot; The server is configured to execute the robot localization method according to any one of claims 1-6; The medical device is used to collect the user's medical images; The image acquisition device is used to acquire images of the user's target body parts; and the position of the image acquisition device is related to the position of the target robot.

Images