Navigation image registration method, apparatus, device, and storage medium

Abstract

The application discloses a navigation image registration method, device and equipment and a storage medium. The method comprises the following steps: performing image reconstruction on medical image point cloud data to obtain a first image, and performing image reconstruction on navigation point cloud data to obtain a second image; determining a first reference coordinate system corresponding to the first image based on a first nose tip position and a first eye position in the first image, obtaining a first conversion matrix and a first intermediate image; determining a second reference coordinate system corresponding to the second image based on a second nose tip position and a second eye position in the second image, obtaining a second conversion matrix and a second intermediate image; determining a registration matrix of the first intermediate image and the second intermediate image; and completing registration of the first image and the second image according to the first conversion matrix, the second conversion matrix and the registration matrix. The method solves the problem that the existing navigation image registration method cannot balance accuracy and speed, and improves the speed and accuracy of navigation image registration.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing, and more particularly to navigation image registration methods, apparatus, devices, and storage media. Background Technology

[0002] In surgical navigation, lesion location can be quickly determined through navigation image registration. Existing navigation image registration methods manually or automatically determine facial features in face images and perform registration based on these features to achieve the corresponding location of the target area. However, due to individual differences in facial features, and the possibility of using a ventilator during surgical navigation, features below the tip of the nose may be difficult to identify. Therefore, the speed and accuracy of navigation image registration need to be improved. In summary, existing navigation image registration methods suffer from the problem of not being able to simultaneously achieve both accuracy and speed. Summary of the Invention

[0003] This invention provides a navigation image registration method, apparatus, device, and storage medium to solve the problem that existing navigation image registration methods cannot simultaneously achieve both accuracy and speed.

[0004] According to one aspect of the present invention, a navigation image registration method is provided, the method comprising:

[0005] A first image is obtained by reconstructing the medical image point cloud data, and a second image is obtained by reconstructing the navigation point cloud data. The first image includes the first facial information of the target object, and the second image includes the second facial information of the target object. The first facial information includes the first nose tip position and the first eye position, and the second facial information includes the second nose tip position and the second eye position.

[0006] The first reference coordinate system corresponding to the first image is determined based on the first nose tip position and the first eye position, and the first image is mapped to the first reference coordinate system to obtain the first transformation matrix and the first intermediate image;

[0007] The second reference coordinate system corresponding to the second image is determined based on the second nose tip position and the second eye position, and the second image is mapped to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image;

[0008] Determine the registration matrix between the first intermediate image and the second intermediate image;

[0009] The registration of the first image and the second image is completed based on the first transformation matrix, the second transformation matrix, and the registration matrix.

[0010] According to another aspect of the present invention, a navigation image registration apparatus is provided, the apparatus comprising:

[0011] The image reconstruction module is used to reconstruct the medical image point cloud data to obtain a first image and to reconstruct the navigation point cloud data to obtain a second image. The first image includes the first facial information of the target object and the second image includes the second facial information of the target object. The first facial information includes the first nose tip position and the first eye position, and the second facial information includes the second nose tip position and the second eye position.

[0012] The first coordinate transformation module is used to determine the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and to map the first image to the first reference coordinate system to obtain the first transformation matrix and the first intermediate image;

[0013] The second coordinate transformation module is used to determine the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, and to map the second image to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image;

[0014] The registration matrix determination module is used to determine the registration matrix between the first intermediate image and the second intermediate image;

[0015] The image registration module is used to register the first image and the second image based on the first transformation matrix, the second transformation matrix, and the registration matrix.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory that is communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the navigation image registration method of any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the navigation image registration method of any embodiment of the present invention.

[0021] The technical solution of the navigation image registration method provided in this invention establishes a reference coordinate system and registers intermediate images to achieve navigation image registration, thereby improving the speed and accuracy of navigation image registration.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a navigation image registration method provided in an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of another navigation image registration method provided in an embodiment of the present invention;

[0026] Figure 3A This is a structural block diagram of a navigation image registration device provided in an embodiment of the present invention;

[0027] Figure 3B This is a structural block diagram of another navigation image registration device provided in an embodiment of the present invention;

[0028] Figure 3C This is a structural block diagram of another navigation image registration device provided in an embodiment of the present invention;

[0029] Figure 3D This is a structural block diagram of another navigation image registration device provided in an embodiment of the present invention;

[0030] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Figure 1 This is a flowchart of a navigation image registration method provided by an embodiment of the present invention. This embodiment is applicable to scenarios of navigation image registration based on facial features and can be executed by a navigation image registration device. The navigation image registration device can be implemented in hardware and / or software and configured in the processor of an electronic device.

[0034] like Figure 1 As shown, the navigation image registration method includes the following steps:

[0035] S110. Perform image reconstruction on medical image point cloud data to obtain a first image, and perform image reconstruction on navigation point cloud data to obtain a second image. The first image includes first facial information of the target object, and the second image includes second facial information of the target object. The first facial information includes the position of the first nose tip and the position of the first eye, and the second facial information includes the position of the second nose tip and the position of the second eye.

[0036] Existing tools, models, or programs can be used to convert medical images including first facial information into corresponding point cloud data, which serves as medical image point cloud data. In one embodiment, computed tomography (CT) image data including the head of the target object is acquired and converted into point cloud format to obtain medical image point cloud data; alternatively, data in Digital Imaging and Communications in Medicine (DICOM) format including the head of the target object can be read, converted into medical images of the corresponding format as needed, and then existing tools, models, or programs can be used to convert the medical images of the corresponding format into corresponding point cloud data.

[0037] Navigation point cloud data is obtained by acquiring navigation image data of the target object and converting the navigation image data into point cloud format. In one embodiment, an image including the face of the target object is acquired by a depth camera mounted on a robotic arm as a navigation image, and the two-dimensional coordinates of the navigation image in the pixel coordinate system are determined; combined with the parameter settings of the depth camera, the three-dimensional coordinates of the navigation image in the camera coordinate system are determined; and the navigation image is converted into point cloud format using existing tools, models, or programs as navigation point cloud data.

[0038] The first image includes first facial information, which may be the coordinates of the complete face of the target object in the first image in the first image coordinate system; the second image includes second facial information, which may be the coordinates of the complete face of the target object in the second image in the second image coordinate system.

[0039] In one embodiment, the first facial information is the coordinates of the complete face in the CT image of the target object's head in the CT image coordinate system; the second facial information is the coordinates of the complete face in the navigation image of the target object's head in the navigation image coordinate system.

[0040] The first facial information includes the position of the first nose tip and the position of the first eye, and the second facial information includes the position of the second nose tip and the position of the second eye. The first nose tip position and the first eye position of the target object are the coordinates of the nose tip and the two eyes of the target object in the first image coordinate system; the second nose tip position and the second eye position of the target object are the coordinates of the nose tip and the two eyes of the target object in the second image coordinate system.

[0041] In one embodiment, the first nose tip position is the coordinate of the nose tip of the target object in the CT image coordinate system in the head CT image of the target object, and the first eye position is the coordinate of the two eyes of the target object in the CT image coordinate system in the head CT image of the target object; the second nose tip position is the coordinate of the nose tip of the target object in the navigation image coordinate system in the head navigation image of the target object, and the second eye position is the coordinate of the two eyes of the target object in the navigation image coordinate system in the head navigation image of the target object.

[0042] Specifically, medical image point cloud data including the head of the target object is acquired, and existing image reconstruction methods for medical image point cloud data are used to reconstruct the image to obtain the corresponding medical image, which serves as the first image. A depth image including the head of the target object is acquired using a depth camera mounted on a robotic arm. The distance from each point in the depth image to the camera can be determined based on the depth image. Combining the two-dimensional coordinates of the target object's head in the pixel coordinate system and the parameter settings of the depth camera, the three-dimensional coordinates of the target object's head in the camera coordinate system are determined, which serve as navigation point cloud data. Image reconstruction is performed on the navigation point cloud data to obtain a second image including the second facial information of the target object.

[0043] S120. Determine the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and map the first image to the first reference coordinate system to obtain the first transformation matrix and the first intermediate image.

[0044] The first eye position and the first nose tip position of the target object in the first image can be extracted using existing image processing algorithms or by training the corresponding model. This solution does not impose any restrictions on this.

[0045] For example, based on the coordinates of the target object's two eyes and nose tip in the CT coordinate system in the CT image, a first reference coordinate system is established in the CT image. The first image in the CT coordinate system is mapped to the first reference coordinate system to obtain the transformation matrix between the CT coordinate system and the first reference coordinate system corresponding to the mapping. This transformation matrix is ​​used as the first transformation matrix, and the image mapped to the first reference coordinate system is the first intermediate image.

[0046] Furthermore, the first eye position includes the inner canthus positions of the first two eyes of the target object. Correspondingly, determining the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position in the first image includes the following steps:

[0047] Step a1: Determine the first line connecting the inner canthi of the two eyes and the first perpendicular line passing through the first line connecting the tip of the nose.

[0048] First, facial features of the target object's first nose tip position and first inner canthus position in the first image are extracted using existing image processing algorithms or trained corresponding models. The coordinates of the first nose tip and the first inner canthus position in the first image are determined as the first nose tip position and the first inner canthus position. Then, based on the first inner canthus position in the first image, the line connecting the two inner canthi is determined as the first connecting line. Finally, a perpendicular line is drawn from the first nose tip position to the first connecting line as the first perpendicular line.

[0049] For example, firstly, the first image is input into a trained target detection model, which outputs the position coordinates of the first nose tip and the first inner canthus of the eyes in the first image and the corresponding facial feature categories; then, a first line connecting the positions of the first inner canthus of the eyes is determined based on the position coordinates of the first inner canthus of the eyes in the first image and the corresponding facial feature categories; finally, a first perpendicular line passing through the first nose tip position and perpendicular to the first line connecting the positions of the first inner canthus of the eyes is determined based on the position coordinates of the first nose tip in the first image and the corresponding facial feature categories.

[0050] Step a2: Determine the first normal vector that passes through the first tip of the nose, is perpendicular to the plane containing the first vertical line and the first connecting line, and points to the outer side of the convex surface of the face.

[0051] Specifically, determine the plane containing the first perpendicular line and the first connecting line; then, determine the vector that passes through the first nose tip position, points to the outer side of the convex surface of the target object's face, and is perpendicular to the plane, as the first normal vector.

[0052] Step a3: Construct a first reference coordinate system with the first nose tip position as the origin and the first perpendicular line, the first target line and the first normal vector as the coordinate axes, wherein the first target line passes through the first nose tip position and is parallel to the first line.

[0053] In the first image, a first target line is determined that passes through the first nose tip position and is parallel to the first connecting line; a first reference coordinate system is constructed with the nose tip of the target object as the origin, the first connecting line and the nose tip forming a first plane, and the corresponding first normal vector pointing to the outer side of the convex surface of the face.

[0054] For example, the first nose tip position is taken as the origin, the first target line passing through the first nose tip position and parallel to the first connecting line is taken as the X-axis, the first perpendicular line is taken as the Y-axis, and the first normal vector is taken as the Z-axis. The directions of the X-axis, Y-axis and Z-axis are not specifically limited, as long as the directions of each axis in the first reference coordinate system and the second reference coordinate system are consistent.

[0055] S130. Determine the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, and map the second image to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image.

[0056] The second image corresponds to the camera coordinate system. A second reference coordinate system is established on the second image, which includes the second facial information of the target object. The second image in the camera coordinate system is mapped to the second reference coordinate system to obtain the transformation matrix between the camera coordinate system and the second reference coordinate system corresponding to the mapping. This matrix is ​​used as the second transformation matrix. The image mapped to the second reference coordinate system is the second intermediate image.

[0057] The second eye position includes the second inner canthus position of the target object's two eyes. Based on the second nose tip position and the second eye position in the second image, a second reference coordinate system corresponding to the second image is determined, including:

[0058] Step b1: Determine the second line connecting the inner canthi of the two eyes and the second perpendicular line passing through the second tip of the nose.

[0059] First, existing image processing algorithms or corresponding models are used to extract facial features of the target object's second nose tip and second inner canthus positions in the second image, and the coordinates of the second nose tip and second inner canthus positions in the first image are determined as the second nose tip position and the second inner canthus position. Then, based on the second inner canthus position in the second image, the line connecting the two inner canthi is determined as the second connecting line. Finally, a perpendicular line is drawn from the second nose tip position to the second connecting line as the second perpendicular line.

[0060] For example, firstly, the second image is input into a trained object detection model, which outputs the position coordinates of the second tip of the nose and the second inner canthus of the eyes in the second image and the corresponding facial feature categories; then, a second line connecting the positions of the second inner canthus of the eyes is determined based on the position coordinates of the second inner canthus of the eyes and the corresponding facial feature categories; finally, a second perpendicular line passing through the position of the second tip of the nose and perpendicular to the second line connecting the positions of the second inner canthus of the eyes is determined based on the position coordinates of the second tip of the nose and the corresponding facial feature categories.

[0061] Step b2: Determine the second normal vector that passes through the second tip of the nose, is perpendicular to the plane containing the second vertical line and the second line, and points to the outer side of the convex surface of the face.

[0062] Specifically, determine the plane containing the second perpendicular line and the second connecting line; then, determine the vector that passes through the second nose tip position and points to the outer side of the convex surface of the target object's face, perpendicular to the plane, as the second normal vector.

[0063] Step b3: Construct a second reference coordinate system with the second nose tip position as the origin and the second perpendicular line, the second target line, and the second normal vector as the coordinate axes. The second target line passes through the second nose tip position and is parallel to the second line.

[0064] In the second image, a second target line is determined that passes through the tip of the nose and is parallel to the second connecting line; a second reference coordinate system is constructed with the tip of the nose of the target object as the origin, the second connecting line and the tip of the nose forming a second plane, and the corresponding second normal vector pointing to the outer side of the convex surface of the face.

[0065] For example, the origin is the second nose tip position, the X-axis is the second target line passing through the first nose tip position and parallel to the first connecting line, the Y-axis is the second perpendicular line, and the Z-axis is the second normal vector. The directions of these axes should be consistent with those in the first and second reference coordinate systems. The advantage of this approach is that by performing preliminary registration of the first and second reference coordinate systems, the amount of data required for registering the first and second intermediate images is reduced, thus accelerating the navigation image registration process and minimizing the impact of individual differences in facial features on navigation image registration.

[0066] S140. Determine the registration matrix between the first intermediate image and the second intermediate image.

[0067] The registration matrix includes the rotation matrix and the translation matrix.

[0068] Specifically, existing image registration algorithms or models are used to register the first intermediate image and the second intermediate image, resulting in a registration matrix between the first and second intermediate images. It is understood that the rotation and / or translation matrices in the registration matrix may be zero.

[0069] S150. Complete the registration of the first image and the second image based on the first transformation matrix, the second transformation matrix and the registration matrix.

[0070] First, multiply the first transformation matrix and the second transformation matrix to obtain the first registration matrix; then, based on the translation matrix and rotation matrix of the first intermediate image and the second intermediate image and the first registration matrix, determine the translation matrix and rotation matrix for registering the first image and the second image, and complete the registration of the first image and the second image based on the registration matrix for registering the first image and the second image.

[0071] The technical solution of the navigation image registration method provided in this embodiment is to establish a corresponding reference coordinate system based on the facial information of the target object in the corresponding image, and register the intermediate image in the reference coordinate system to achieve navigation image registration of the target object. This reduces the influence of individual differences in facial features on navigation image registration and improves the speed and accuracy of navigation image registration.

[0072] Figure 2This is a flowchart of another navigation image registration method provided by an embodiment of the present invention. This embodiment belongs to the same inventive concept as the navigation image registration method in the above embodiments. Based on the above embodiments, before determining the registration matrix of the first intermediate image and the second intermediate image, the following steps are added: constructing a first binary function of the first coordinate information of the first intermediate image, and constructing a second binary function of the second coordinate information of the second intermediate image; performing a two-dimensional Fourier transform on the first binary function to obtain first frequency domain data, and performing a two-dimensional Fourier transform on the second binary function to obtain second frequency domain data; performing a high-pass filter on the first frequency domain data to obtain first target frequency domain data, and performing a high-pass filter on the second frequency domain data to obtain second target frequency domain data; performing a two-dimensional inverse Fourier transform on the first target frequency domain data to update the first intermediate image, and performing a two-dimensional inverse Fourier transform on the second target frequency domain data to update the second intermediate image.

[0073] like Figure 2 As shown, the navigation image registration method includes the following steps:

[0074] S210. Perform image reconstruction on medical image point cloud data to obtain a first image, and perform image reconstruction on navigation point cloud data to obtain a second image. The first image includes first facial information of the target object, and the second image includes second facial information of the target object. The first facial information includes the position of the first nose tip and the position of the first eye, and the second facial information includes the position of the second nose tip and the position of the second eye.

[0075] S220. Determine the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and map the first image to the first reference coordinate system to obtain the first transformation matrix and the first intermediate image.

[0076] S230. Determine the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, and map the second image to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image.

[0077] Furthermore, before determining the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and before determining the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, the process includes:

[0078] Step c1: Extract a first region image from the first image, including the defined face region, and update the first image using the first region image. The defined face region includes the nose and eyes of the target object.

[0079] Since the facial features of the target subject above the lower edge of the nose and below the forehead do not change significantly during surgery, the facial area is defined to include the tip of the nose and both eyes.

[0080] For example, an existing image processing algorithm or a corresponding model is used to extract an image from the first image that includes the portion above the lower edge of the nose and below the forehead of the target object, as the first region image, and the first image is updated using the first region image.

[0081] Step c2: Extract a second region image, including the defined face region, from the second image, and update the second image using the second region image.

[0082] In one embodiment, the facial region is defined as including the tip of the nose and two eyes of the target object. An existing image processing algorithm or a corresponding model is used to extract an image from the second image that includes the portion above the tip of the nose and below the forehead of the target object, which is then used as the second region image. The second image is then updated using the second region image.

[0083] S2401. Construct a first binary function for the first coordinate information of the first intermediate image, and construct a second binary function for the second coordinate information of the second intermediate image.

[0084] Taking three-dimensional coordinate information as an example, in the first reference coordinate system, a binary function corresponding to the first coordinate information of the first intermediate image is constructed with the coordinates of two coordinate axes as independent variables and the coordinates of the remaining coordinate axis as dependent variables. This function serves as the first binary function. Correspondingly, in the second reference coordinate system, a binary function corresponding to the second coordinate information of the second intermediate image is constructed with the coordinates of the corresponding two coordinate axes as independent variables and the coordinates of the remaining coordinate axis as dependent variables. This function serves as the second binary function.

[0085] For example, using the X-axis and Y-axis coordinates corresponding to the first coordinate information of the first intermediate image in the first reference coordinate system as independent variables and the Z-axis coordinate as dependent variables, a corresponding binary function is constructed as the first binary function; correspondingly, using the X-axis and Y-axis coordinates corresponding to the second coordinate information of the second intermediate image in the second reference coordinate system as independent variables and the Z-axis coordinate as dependent variables, a corresponding binary function is constructed as the second binary function.

[0086] S2402. Perform a two-dimensional Fourier transform on the first binary function to obtain the first frequency domain data, and perform a two-dimensional Fourier transform on the second binary function to obtain the second frequency domain data.

[0087] A two-dimensional Fourier transform is performed on the first binary function in the first reference coordinate system to obtain the frequency domain data corresponding to the first intermediate image, which is used as the first frequency domain data; a two-dimensional Fourier transform is performed on the second binary function in the second reference coordinate system to obtain the frequency domain data corresponding to the second intermediate image, which is used as the second frequency domain data.

[0088] The advantage of doing this is that it transforms the image from the image space to the frequency space, allowing for further frequency domain processing, such as edge enhancement, image sharpening, image smoothing, noise suppression, and / or spectrum analysis. Since signals are often simpler and more intuitive in the frequency domain than in the time domain, and mature existing frequency domain techniques can be applied in the spatial frequency domain.

[0089] S2403. Perform high-pass filtering on the first frequency domain data to obtain the first target frequency domain data, and perform high-pass filtering on the second frequency domain data to obtain the second target frequency domain data.

[0090] The high-frequency components of the first frequency domain data are used as the first target frequency domain data, and the high-frequency components of the second frequency domain data are used as the second target frequency domain data. This approach further reduces the amount of data required for navigation image registration, thus accelerating the registration process.

[0091] For example, the first frequency domain data is input into a high-pass filter to obtain high-frequency frequency domain data, which corresponds to the facial edges and other features of the target object that show significant changes in the first intermediate image.

[0092] S2404. Perform a two-dimensional inverse Fourier transform on the frequency domain data of the first target to update the first intermediate image, and perform a two-dimensional inverse Fourier transform on the frequency domain data of the second target to update the second intermediate image.

[0093] Specifically, a two-dimensional inverse Fourier transform is performed on the first target frequency domain data corresponding to the target frequency band to obtain the corresponding image, which is used as the updated first intermediate image; a two-dimensional inverse Fourier transform is performed on the second target frequency domain data corresponding to the target frequency band to obtain the corresponding image, which is used as the updated second intermediate image.

[0094] S250. The rotation and translation matrices of the first and second intermediate images are determined by the iterative nearest point algorithm, and the rotation and translation matrices are concatenated to form the registration matrix.

[0095] The Iterative Closest Point (ICP) algorithm is a point set matching algorithm with a point set coordinate system. To find the closest pair of points between two sets, the algorithm calculates the transformation error of the closest point pair based on the estimated registration matrix, continuing this process until the iteration error is less than a set threshold or a set number of iterations is reached to determine the final registration matrix.

[0096] For example, the ICP algorithm is used to register the point cloud data corresponding to the first intermediate image and the point cloud data of the second intermediate image to obtain the rotation matrix and translation matrix of the first intermediate image and the second intermediate image. The rotation matrix and translation matrix are then concatenated to form the registration matrix.

[0097] S260. Complete the registration of the first image and the second image based on the first transformation matrix, the second transformation matrix and the registration matrix.

[0098] The technical solution of the navigation image registration method provided in this embodiment is to construct a binary function to perform Fourier transform and filter the set face region image; and to use the ICP algorithm to accurately register the filtered set face region image, thereby reducing the amount of data of the ICP algorithm, speeding up the operation of the ICP algorithm, and thus improving the speed of navigation image registration.

[0099] Figure 3A This is a structural block diagram of a navigation image registration device provided in an embodiment of the present invention. This embodiment is applicable to scenarios of navigation image registration based on facial features. The device can be implemented in hardware and / or software and integrated into the processor of an electronic device with application development capabilities.

[0100] like Figure 3A As shown, the navigation image registration device includes:

[0101] Image reconstruction module 301 is used to perform image reconstruction on medical image point cloud data to obtain a first image and to perform image reconstruction on navigation point cloud data to obtain a second image. The first image includes first facial information of the target object and the second image includes second facial information of the target object. The first facial information includes the first nose tip position and the first eye position, and the second facial information includes the second nose tip position and the second eye position.

[0102] The first coordinate transformation module 302 is used to determine the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and to map the first image to the first reference coordinate system to obtain the first transformation matrix and the first intermediate image.

[0103] The second coordinate transformation module 303 is used to determine the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, and to map the second image to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image;

[0104] The registration matrix determination module 304 is used to determine the registration matrix between the first intermediate image and the second intermediate image;

[0105] The image registration module 305 is used to register the first image and the second image according to the first transformation matrix, the second transformation matrix and the registration matrix.

[0106] Optional, such as Figure 3B As shown, the device also includes a frequency domain transformation module 306, which is used for:

[0107] Construct a first binary function for the first coordinate information of the first intermediate image, and construct a second binary function for the second coordinate information of the second intermediate image;

[0108] Perform a two-dimensional Fourier transform on the first binary function to obtain the first frequency domain data, and perform a two-dimensional Fourier transform on the second binary function to obtain the second frequency domain data.

[0109] High-pass filtering is applied to the first frequency domain data to obtain the first target frequency domain data, and high-pass filtering is applied to the second frequency domain data to obtain the second target frequency domain data.

[0110] A two-dimensional inverse Fourier transform is performed on the frequency domain data of the first target to update the first intermediate image, and a two-dimensional inverse Fourier transform is performed on the frequency domain data of the second target to update the second intermediate image.

[0111] Optionally, the first coordinate transformation module 302 is also used for:

[0112] Determine the first line connecting the inner canthi of the two eyes and the first perpendicular line passing through the first tip of the nose;

[0113] Determine the first normal vector that passes through the first tip of the nose, is perpendicular to the plane containing the first vertical line and the first connecting line, and points to the outer side of the convex surface of the face;

[0114] A first reference coordinate system is constructed with the first nose tip position as the origin and the first perpendicular line, the first target line, and the first normal vector as the coordinate axes. The first target line passes through the first nose tip position and is parallel to the first line.

[0115] Optionally, the second coordinate transformation module 303 is also used for:

[0116] Determine the second line connecting the inner canthi of the two eyes and the second perpendicular line passing through the second tip of the nose;

[0117] Determine the second normal vector that passes through the second tip of the nose, is perpendicular to the plane containing the second vertical line and the second line connecting them, and points to the outer side of the convex surface of the face;

[0118] A second reference coordinate system is constructed with the second nose tip position as the origin and the second perpendicular line, the second target line, and the second normal vector as the coordinate axes. The second target line passes through the second nose tip position and is parallel to the second line.

[0119] Optional, such as Figure 3C As shown, the device also includes a point cloud conversion module 307, which is used for:

[0120] Acquire CT image data of the target object;

[0121] Convert CT image data into point cloud format to obtain medical image point cloud data;

[0122] Obtain navigation image data of the target object;

[0123] The navigation image data is converted into a point cloud format to obtain navigation point cloud data.

[0124] Optional, such as Figure 3D As shown, the device also includes an image update module 308, which is used for:

[0125] Extract a first region image from the first image, including the defined face region, and update the first image using the first region image. The defined face region includes the nose and eyes of the target object.

[0126] Extract a second region image from the second image, including the defined facial region, and update the second image using the second region image.

[0127] Optionally, the registration matrix determination module 304 is used to: determine the rotation matrix and translation matrix of the first intermediate image and the second intermediate image through an iterative nearest point algorithm, and concatenate the rotation matrix and translation matrix to form a registration matrix.

[0128] The technical solution of the navigation image registration device provided in this embodiment of the invention establishes a reference coordinate system to register intermediate images, thereby achieving navigation image registration of the target object and improving the speed and accuracy of navigation image registration.

[0129] The navigation image registration device provided in this embodiment of the invention can execute the navigation image registration method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0130] Figure 4 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0131] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as navigation image registration methods.

[0134] In some embodiments, the navigation image registration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the navigation image registration method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the navigation image registration method by any other suitable means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0140] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0141] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A navigation image registration method, characterized in that, include: A first image is obtained by reconstructing the medical image point cloud data, and a second image is obtained by reconstructing the navigation point cloud data. The first image includes first facial information of the target object, and the second image includes second facial information of the target object. The first facial information includes the first nose tip position and the first eye position, and the second facial information includes the second nose tip position and the second eye position. Based on the first nose tip position and the first eye position, a first reference coordinate system corresponding to the first image is determined, and the first image is mapped to the first reference coordinate system to obtain a first transformation matrix and a first intermediate image; The second reference coordinate system corresponding to the second image is determined based on the second nose tip position and the second eye position, and the second image is mapped to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image; Determine the registration matrix between the first intermediate image and the second intermediate image; The registration of the first image and the second image is completed based on the first transformation matrix, the second transformation matrix, and the registration matrix.

2. The method according to claim 1, characterized in that, Before determining the registration matrix between the first intermediate image and the second intermediate image, the following steps are included: Construct a first binary function for the first coordinate information of the first intermediate image, and construct a second binary function for the second coordinate information of the second intermediate image; Perform a two-dimensional Fourier transform on the first binary function to obtain the first frequency domain data, and perform a two-dimensional Fourier transform on the second binary function to obtain the second frequency domain data. The first frequency domain data is high-pass filtered to obtain the first target frequency domain data, and the second frequency domain data is high-pass filtered to obtain the second target frequency domain data. A two-dimensional inverse Fourier transform is performed on the first target frequency domain data to update the first intermediate image, and a two-dimensional inverse Fourier transform is performed on the second target frequency domain data to update the second intermediate image.

3. The method according to claim 1, characterized in that, The first eye position includes the inner canthus positions of the first two eyes of the target object. The step of determining the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position includes: Determine the first line connecting the inner canthi of the first two eyes and the first perpendicular line passing through the first tip of the nose. Determine the first normal vector that passes through the first tip of the nose, is perpendicular to the plane containing the first vertical line and the first line connecting them, and points to the outer side of the convex surface of the face. The first reference coordinate system is constructed with the first nose tip position as the origin and the first perpendicular line, the first target line and the first normal vector as the coordinate axes, wherein the first target line passes through the first nose tip position and is parallel to the first line.

4. The method according to claim 1, characterized in that, The second eye position includes the second inner canthus position of the target object's two eyes. The step of determining the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position includes: Determine the second line connecting the inner canthi of the two eyes and the second perpendicular line passing through the second tip of the nose; Determine the second normal vector that passes through the second tip of the nose, is perpendicular to the plane containing the second vertical line and the second line connecting them, and points to the outer side of the convex surface of the face; The second reference coordinate system is constructed with the second nose tip position as the origin and the second perpendicular line, the second target line and the second normal vector as the coordinate axes, wherein the second target line passes through the second nose tip position and is parallel to the second line.

5. The method according to claim 1, characterized in that, The medical image point cloud data is determined through the following steps: Acquire CT image data of the target object; The CT image data is converted into point cloud format to obtain medical image point cloud data; The navigation point cloud data is determined through the following steps: Obtain navigation image data of the target object; The navigation image data is converted into point cloud format to obtain navigation point cloud data.

6. The method according to any one of claims 1-5, characterized in that, Before determining the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and before determining the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, the process includes: Extract a first region image including a defined facial region from the first image, and update the first image using the first region image, wherein the defined facial region includes the nose and eyes of the target object; Extract a second region image, including the defined facial region, from the second image, and update the second image using the second region image.

7. The method according to claim 6, characterized in that, Determining the registration matrix between the first intermediate image and the second intermediate image includes: The rotation and translation matrices of the first and second intermediate images are determined by the iterative nearest-point algorithm, and the rotation and translation matrices are concatenated to form the registration matrix.

8. A navigation image registration device, characterized in that, include: The image reconstruction module is used to reconstruct the medical image point cloud data to obtain a first image and to reconstruct the navigation point cloud data to obtain a second image. The first image includes first facial information of the target object and the second image includes second facial information of the target object. The first facial information includes a first nose tip position and a first eye position, and the second facial information includes a second nose tip position and a second eye position. The first coordinate transformation module is used to determine the first reference coordinate system corresponding to the first image based on the first nose tip position and the first eye position, and to map the first image to the first reference coordinate system to obtain the first transformation matrix and the first intermediate image. The second coordinate transformation module is used to determine the second reference coordinate system corresponding to the second image based on the second nose tip position and the second eye position, and to map the second image to the second reference coordinate system to obtain the second transformation matrix and the second intermediate image; The registration matrix determination module is used to determine the registration matrix between the first intermediate image and the second intermediate image; The image registration module is used to register the first image and the second image based on the first transformation matrix, the second transformation matrix, and the registration matrix.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the navigation image registration method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the navigation image registration method according to any one of claims 1-7.

Images