Image data registration method, device and computer program product

Abstract

The embodiment of the invention is suitable for the technical field of computer-aided medical treatment and image processing, and provides an image data registration method and device and a computer program product, and the method comprises the steps: obtaining a plurality of two-dimensional images, and determining a first direction based on the plurality of two-dimensional images; sorting the plurality of two-dimensional images along a first direction; a space coordinate system is constructed based on the two-dimensional image sequence obtained after sorting, and the direction of the z axis of the space coordinate system is the same as the first direction; the intersection between each two-dimensional image arranged along the z axis of the space coordinate system and the to-be-registered three-dimensional data is calculated, a plurality of two-dimensional display windows are obtained through rendering based on the intersection, and any two-dimensional display window is used for displaying intersection grids of the two-dimensional images and the three-dimensional data; and in response to an adjustment instruction for any intersecting grid, performing position adjustment on the three-dimensional data so as to realize registration of the two-dimensional image and the three-dimensional data. By adopting the method, the registration precision of the two-dimensional image and the three-dimensional data can be improved.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of computer-assisted medical technology and image processing technology, and in particular, to an image data registration method, device and computer program product. Background Art

[0002] In modern medical diagnosis and surgical navigation, computed tomography (CT) and ultrasound imaging are two widely used imaging technologies, each with its own unique advantages. CT technology can provide high-resolution three-dimensional bone structure, while ultrasound technology has the characteristics of real-time, radiation-free, and portable. Spatial registration between these two types of images has always been a difficult problem in the industry, especially in the intraoperative environment, which is greatly increased by the influence of time and data scanning.

[0003] Traditional registration between ultrasound images and CT data usually relies on manually selected landmarks or a one-time automatic registration algorithm based on deep neural networks. However, the manual selection of landmarks is easily affected by the operator's subjective factors, resulting in low registration accuracy. The automatic registration algorithm may also fail when the ultrasound image does not scan the established anatomical structure or when the patient's posture is significantly deformed. In addition, due to the lack of flexibility, automatic registration cannot adjust the registration results when errors occur in the registration, nor can it be adjusted in real time according to the specific situation during the operation. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an image data registration method, device and computer program product to reduce the difficulty of registration between two-dimensional images and three-dimensional data and improve the registration accuracy. For example, by using the image data registration method provided in the embodiments of the present application, accurate registration between preoperative three-dimensional CT data and intraoperative two-dimensional ultrasound images can be achieved in an intraoperative scene.

[0005] A first aspect of an embodiment of the present application provides an image data registration method, comprising: Acquire a plurality of two-dimensional images, and determine a first direction based on the plurality of two-dimensional images; sorting the plurality of the two-dimensional images along the first direction; Based on the sorted two-dimensional image sequence, constructing a spatial coordinate system, wherein the direction of the z-axis of the spatial coordinate system is the same as the first direction; respectively calculating the intersections between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and rendering a plurality of two-dimensional display windows based on the intersections, wherein any of the two-dimensional display windows is used to display the intersection grids of the two-dimensional image and the three-dimensional data; In response to an adjustment instruction for the intersecting grid displayed in any one of the two-dimensional display windows, the three-dimensional data is positionally adjusted to achieve registration between the two-dimensional image and the three-dimensional data.

[0006] Optionally, determining the first direction based on the multiple two-dimensional images includes: Determine a capture position corresponding to each of the two-dimensional images, wherein the capture position includes a spatial position of a capture device used to capture each of the two-dimensional images when capturing each of the two-dimensional images; A first direction is determined based on a straight line obtained by fitting the plurality of acquisition positions.

[0007] Optionally, the sorting the plurality of two-dimensional images along the first direction includes: According to the spatial position of the acquisition device when each of the two-dimensional images is acquired, the two-dimensional images acquired at the corresponding position are sorted along the first direction to obtain a two-dimensional image sequence.

[0008] Optionally, constructing a spatial coordinate system based on the sorted two-dimensional image sequence includes: Calculating the bounding box of the sorted two-dimensional image sequence, and determining the bounding box coordinate system corresponding to the bounding box; The z-axis of the bounding box coordinate system is rotated to be the same as the first direction to construct a spatial coordinate system, wherein the x-axis, y-axis and z-axis of the spatial coordinate system coincide with the x-axis, y-axis and z-axis of the bounding box coordinate system respectively.

[0009] Optionally, the obtaining of a plurality of two-dimensional display windows based on the intersection rendering includes: Determine, from the plurality of the two-dimensional images, a plurality of candidate images whose corresponding intersections are greater than a threshold; According to a preset distance value, a plurality of target images are determined from the plurality of candidate images, and the intersection between each of the target images and the three-dimensional data is rendered to obtain a plurality of intersecting grids; The plurality of intersecting grids are displayed through a plurality of two-dimensional display windows.

[0010] Optionally, displaying the plurality of intersecting grids through a plurality of two-dimensional display windows comprises: For any of the target images, focusing the camera on the center of the target image; Control the visual axis of the camera to align with the plane normal vector of the target image, and adjust the upper end of the visual axis to align in the opposite direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image.

[0011] Optionally, before determining from the plurality of two-dimensional images a plurality of candidate images whose corresponding intersections are greater than a threshold, the method further includes: If the intersection between any of the two-dimensional images and the three-dimensional data is an empty set, after moving the three-dimensional data so that the center of the three-dimensional data coincides with the center of the bounding box of the two-dimensional image sequence, the intersection between each of the two-dimensional images and the three-dimensional data is recalculated.

[0012] Optionally, the adjustment instruction includes a translation instruction or a rotation instruction, the translation instruction includes a positive translation instruction or a negative translation instruction, the rotation instruction includes a positive rotation instruction or a negative rotation instruction, the adjustment method corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis or z-axis of the spatial coordinate system, and the adjustment method corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis or z-axis of the spatial coordinate system, and the position adjustment of the three-dimensional data in response to the adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows includes: In response to an adjustment instruction for the intersecting grids displayed in any of the two-dimensional display windows, the three-dimensional data is positionally adjusted according to an adjustment method corresponding to the adjustment instruction.

[0013] Optionally, the adjusting the position of the three-dimensional data according to the adjustment method corresponding to the adjustment instruction includes: generating a rotation matrix based on an image coordinate system of a target image corresponding to the adjustment instruction and an adjustment method corresponding to the adjustment instruction; The position of the three-dimensional data is adjusted according to the rotation matrix.

[0014] Optionally, it also includes: Determine the top pixel of each intersection line constituting the intersection grid in each of the two-dimensional display windows, and expand the expanded intersection grid based on each of the top pixels; The degree of overlap between the expanded intersecting grid and the corresponding two-dimensional image in the two-dimensional display window is calculated, and the degree of overlap is used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

[0015] Optionally, the two-dimensional image includes an ultrasound image, and the three-dimensional data includes CT data.

[0016] A second aspect of an embodiment of the present application provides an image data registration device, comprising: A determination module, configured to acquire a plurality of two-dimensional images and determine a first direction based on the plurality of two-dimensional images; A sorting module, used for sorting the plurality of the two-dimensional images along the first direction; A construction module, used to construct a spatial coordinate system based on the sorted two-dimensional image sequence, wherein the direction of the z-axis of the spatial coordinate system is the same as the first direction; A rendering module, used to respectively calculate the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and render a plurality of two-dimensional display windows based on the intersection, wherein any of the two-dimensional display windows is used to display the intersection grid of the two-dimensional image and the three-dimensional data; A registration module is used to adjust the position of the three-dimensional data in response to an adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows, so as to achieve registration between the two-dimensional image and the three-dimensional data.

[0017] A third aspect of an embodiment of the present application provides a computer-assisted medical device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer-assisted medical device implements the method described in the first aspect above.

[0018] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the method described in the first aspect above is implemented.

[0019] A fifth aspect of the embodiments of the present application provides a computer program product, including a computer program, which, when executed, enables the method described in the first aspect to be executed.

[0020] Compared with the prior art, the embodiments of the present application have the following beneficial effects: The embodiments of the present application can realize accurate interactive registration between two-dimensional images and three-dimensional data, for example, accurate interactive registration between intraoperative two-dimensional ultrasound images and preoperative CT bone models, overcoming the limitations of fully automatic algorithms in complex situations. In addition, the embodiments of the present application provide an intuitive and interactive registration method, allowing doctors to make real-time adjustments based on professional judgments, and through observation of multiple interfaces, reduce the requirements of the registration process for obvious features of the scanned image, improve the registration efficiency, reduce the complexity of manual operations, and maintain the accuracy of the registration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of an image data registration method provided in an embodiment of the present application; Figure 2 It is a schematic diagram of the overall process of an image data registration method provided in an embodiment of the present application; Figure 3 It is a schematic diagram of a process of constructing a three-dimensional skeleton model provided in an embodiment of the present application; Figure 4 is a schematic diagram of an ultrasonic probe provided in an embodiment of the present application; Figure 5 It is a schematic diagram of a straight line fitting process provided by an embodiment of the present application; Figure 6 It is a schematic diagram of a process of sorting two-dimensional images provided in an embodiment of the present application; Figure 7 is a schematic diagram of constructing a spatial coordinate system provided by an embodiment of the present application; Figure 8 is a schematic diagram of a process for determining a candidate image provided by an embodiment of the present application; Fig. 9 It is a flowchart of a rendering display window provided in an embodiment of the present application; Fig.10 It is a schematic diagram of the display effect of a two-dimensional rendering window provided in an embodiment of the present application; Fig.11 It is a schematic diagram of a process of updating a rendering window provided in an embodiment of the present application; Fig.12 It is a schematic diagram of a registration confidence evaluation process provided in an embodiment of the present application; Fig.13 is a schematic diagram of an image data registration device provided in an embodiment of the present application; Fig.14 It is a schematic diagram of a computer-assisted medical device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0024] The technical solution of the present application is described below through specific embodiments.

[0025] Reference Figure 1 , shows a schematic diagram of an image data registration method provided in an embodiment of the present application, which may specifically include the following steps: S101. Acquire a plurality of two-dimensional images, and determine a first direction based on the plurality of two-dimensional images.

[0026] It should be noted that the method can be applied to a computer device, which can be various types of computer-assisted medical devices. That is, the computer device or computer-assisted medical device can achieve the registration of image data by executing the various steps of the method provided in the embodiment of the present application. In one example, the above-mentioned computer device can be a device including a display unit, which can display relevant data or pictures during the registration process for the operator to understand the registration status, so as to facilitate the operator to adjust the registration process based on the intuitively displayed picture, so as to achieve the purpose of accurate registration.

[0027] The registration performed by applying this method can be the registration of two-dimensional images and three-dimensional data. For example, the above-mentioned two-dimensional image can be a two-dimensional ultrasound image collected during the intraoperative stage, and the above-mentioned three-dimensional data can be three-dimensional CT data collected during the preoperative stage. For ease of understanding, the subsequent introduction of the embodiments of the present application takes the registration of two-dimensional ultrasound images and three-dimensional CT data during the intraoperative stage as an example to illustrate the specific implementation process of the present method. It should be understood that this method is not limited to the registration between ultrasound images and CT data, but is also applicable to the registration between other types of two-dimensional images and other types of three-dimensional data, and the embodiments of the present application do not limit this.

[0028] like Figure 2 FIG. 1 is a schematic diagram of the overall process of an image data registration method provided by an embodiment of the present application. Figure 2As shown, during the registration process, the input data of the computer device includes three types. One is the current target CT image data X, that is, the CT data of the patient collected before the operation; the other is the ultrasound image I' in space, that is, the ultrasound image of the patient collected during the operation. The registration process of this method is to register the above-mentioned preoperative CT data with the intraoperative ultrasound image. In this process, the computer device can also receive input data from the user, such as Figure 2 The user keyboard input shown in , as the third input data of the computer device, can be used as a user control flow to adjust the registration process in real time, so as to achieve the purpose of accurate registration and obtain the corresponding registration matrix.

[0029] like Figure 2 As shown, the registration matrix obtained after the registration is completed is also the output data of the computer device, that is, Figure 2 The registration matrix obtained by registering the preoperative CT data to the intraoperative ultrasound space is shown in , and the registration matrix can be expressed as M=[R|T]∈R^(4*4).

[0030] As mentioned above, the three-dimensional CT data can be obtained by collecting CT images of the patient in the preoperative stage. For example, in the preoperative stage, the CT data of the patient can be collected by using a CT device. Figure 1 As shown, the acquired CT data can be processed by a deep neural network to generate a grid S suitable for registration during the intraoperative stage.

[0031] Specifically, Figure 3 FIG. 1 is a flow chart of a method for constructing a three-dimensional skeleton model according to an embodiment of the present application. Figure 1 and Figure 3 As shown in , for the current target CT image data X, i.e., preoperative CT data, a deep neural network can be used to segment it into label data of the same size as the CT. The label data can specifically indicate the patient's body parts and the foreground and background areas in the image. Figure 3 As shown in the figure, the segmentation result of the preoperative CT data can be expressed as Y∈[0,1]^(w*h*d). On this basis, the MC (marching cube) algorithm and the patch filtering algorithm can be used to generate a grid S suitable for intraoperative registration. The grid S is a three-dimensional bone model obtained based on CT data, which consists of a point cloud and the line segments connecting the point clouds. Each point in the point cloud p∈R^3.

[0032] In the embodiment of the present application, the plurality of two-dimensional images may be acquired using a calibrated ultrasound probe under the guidance of a navigation system. For example, N1 intraoperative ultrasound images I={I0, ..., I N1}, where N1≥3.

[0033] like Figure 4 , which is a schematic diagram of an ultrasonic probe provided in an embodiment of the present application, a tracer 401 may be installed on the ultrasonic probe 400, and the calibration of the ultrasonic probe 400 may be completed by combining the tracer 401 with a navigation system (not shown in the figure). When the calibrated ultrasonic probe 400 acquires each ultrasonic image, its pixel point set Ut and spatial matrix Mt may be obtained at the same time. That is, each ultrasonic image acquired may not only obtain the relevant information of its image coordinate system, but also include the relevant information of its inherent spatial coordinate system.

[0034] In a possible implementation of the embodiment of the present application, based on the acquired multiple ultrasound images, the computer device may first determine a direction, such as a first direction. The first direction may be used to assist in constructing a spatial coordinate system required for registration.

[0035] In a specific implementation, a straight line may be fitted based on a plurality of acquired ultrasound images, and the first direction may be determined based on the fitted straight line.

[0036] like Figure 5 FIG. 1 is a schematic diagram of a straight line fitting process provided by an embodiment of the present application. Figure 5 The fitting process shown is the process of fitting a straight line based on multiple ultrasound images acquired by the ultrasound probe. Figure 5 As shown, each ultrasound image acquired by the ultrasound probe can be expressed as I i ={U i , M i}, multiple ultrasound images are Figure 5 The image sequence I={I0, ..., I N1 The computer device may determine the first direction based on a straight line fitted from multiple acquisition positions by determining the acquisition position corresponding to each two-dimensional image.

[0037] In the embodiment of the present application, the acquisition position may include the spatial position of the acquisition device used to acquire each two-dimensional image when acquiring each two-dimensional image. For example, the acquisition device is an ultrasonic probe, so the straight line Gz can be obtained by fitting according to the spatial position of the endpoint P of the ultrasonic probe.

[0038] In a specific implementation, the spatial position P = {p0, ..., p N1}, the least squares method (LSM) is used for fitting to obtain the straight line Gz=LSM(p).

[0039] In the embodiment of the present application, for the fitted straight line Gz, the first direction can be determined by the directions of the first image and the last image in the plurality of ultrasound images. Exemplarily, the first direction can be approximately from the first image in the ultrasound image to the last image.

[0040] S102: Sort the multiple two-dimensional images along the first direction.

[0041] In actual operation, the positions of multiple ultrasound images acquired by the ultrasound probe in space are not strictly arranged in the order of acquisition time. Therefore, after obtaining multiple images acquired by the ultrasound probe, the computer device can sort the multiple images along a certain direction, for example, sort the multiple two-dimensional ultrasound images along the first direction determined above, so that the spatial position corresponding to each image after sorting is arranged along a certain direction.

[0042] Specifically, according to the spatial position of the acquisition device when each two-dimensional image is acquired, the two-dimensional images acquired at the corresponding position may be sorted along the first direction to obtain a two-dimensional image sequence.

[0043] like Figure 6 FIG. 1 is a schematic diagram of a process for sorting two-dimensional images provided in an embodiment of the present application. Figure 6 2 shows a process of sorting the image sequence I along the first direction Gz to obtain a two-dimensional image sequence I′.

[0044] Specifically, refer to Figure 6 The computer device can sort the position P of the ultrasound probe endpoint when each image is acquired along the first direction Gz, for example, sort the position P of the ultrasound probe endpoint according to the distance D, thereby outputting the sorted ultrasound image sequence I'.

[0045] S103 . Construct a spatial coordinate system based on the sorted two-dimensional image sequence, wherein the direction of the z-axis of the spatial coordinate system is the same as the first direction.

[0046] In an embodiment of the present application, the computer device can construct a spatial coordinate system based on the sorted two-dimensional image sequence, so that the direction of the z-axis of the spatial coordinate system can be the same as the first direction determined above.

[0047] like Figure 7 As shown, it is a schematic diagram of constructing a spatial coordinate system provided by an embodiment of the present application. Figure 7In the process shown, after the images are sorted to obtain a two-dimensional image sequence I', the computer device can calculate the bounding box of the two-dimensional image sequence I' and construct a spatial coordinate system based on the bounding box. In one example, an OBB (oriented bounding box) bounding box can be used to construct the spatial coordinate system.

[0048] Reference Figure 7 As shown, after the computer device completes the sorting of the ultrasound images and obtains the image sequence I', it can calculate the OBB bounding box of the sorted two-dimensional image sequence I' in three dimensions, and determine the bounding box coordinate system corresponding to the OBB bounding box, and obtain the x-axis, y-axis and z-axis {Bx, By, Bz} of the bounding box coordinate system, as well as the bounding box center c, the length, width and height (w, h, l) of the bounding box coordinate system. The computer device can construct a spatial coordinate system by rotating the z-axis of the bounding box coordinate system to the same direction as the first direction, so that the x-axis, y-axis and z-axis of the spatial coordinate system coincide with the x-axis, y-axis and z-axis of the bounding box coordinate system, respectively. Therefore, the spatial coordinate system constructed above can be expressed as {Gx, Gy, Gz}, and the two-dimensional image sequence I' includes multiple two-dimensional images arranged along the z-axis of the spatial coordinate system.

[0049] S104, respectively calculating the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and rendering based on the intersection to obtain multiple two-dimensional display windows, any of the two-dimensional display windows is used to display the intersection grid of the two-dimensional image and the three-dimensional data.

[0050] After completing the aforementioned steps, a plurality of two-dimensional ultrasound images are acquired, and a spatial coordinate system is constructed based on the two-dimensional ultrasound images, the computer device can perform initial registration.

[0051] like Figure 1 As shown, in the initial registration stage, the computer device can calculate the intersection between each two-dimensional image arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and the intersection can specifically refer to the intersection between the two-dimensional ultrasound image and the grid S (three-dimensional skeletal model S). The computer device can determine multiple candidate images I2 whose corresponding intersections are greater than the threshold T2 from multiple two-dimensional images. Exemplarily, the computer device can select N2 two-dimensional images with the largest intersection as candidate images.

[0052] In a possible implementation of an embodiment of the present application, when N2 two-dimensional images with the largest intersection are selected as candidate images, if the intersection between any two-dimensional image and the three-dimensional data is an empty set, for example, the intersection between any ultrasound image and the model S is empty, the computer device can move the three-dimensional data so that the center of the three-dimensional data coincides with the center c of the bounding box of the two-dimensional image sequence I', and then recalculate the intersection between each two-dimensional image and the three-dimensional data.

[0053] like Figure 8 FIG. 1 is a schematic diagram of a process for determining a candidate image provided by an embodiment of the present application. Figure 8 The figure shows the process of selecting N2 two-dimensional images with the largest intersection as candidate images from the two-dimensional image sequence after sorting. Specifically, the computer device can calculate the intersection of the model S and each two-dimensional image. If there is no intersection between all two-dimensional images and the model S, for example, the intersection between all two-dimensional images and the model S is less than the threshold T1, the computer device can first move the model S to a position where the center of the model coincides with the center c of the OBB bounding box, and then calculate the intersection between the two. And select N2 images with an intersection greater than T2 as candidate images.

[0054] In the embodiment of the present application, based on the selected multiple candidate images, the computer device can render them, and display the registration status of the candidate images with the model S through the multiple two-dimensional display serial ports obtained by rendering.

[0055] In a possible implementation of an embodiment of the present application, multiple target images can be determined from multiple candidate images according to a preset distance value. Since each candidate image is arranged along the z-axis of the spatial coordinate system, multiple target images can be selected from multiple candidate images as evenly as possible according to a certain distance. Then, the computer device can render the intersection between the target image and the three-dimensional data to obtain multiple intersecting grids, and display the rendered multiple intersecting grids through multiple two-dimensional display windows. The above-mentioned intersecting grids can be used to characterize the situation of the two-dimensional target image and the model S being registered, that is, multiple two-dimensional sections in the registration process. In addition, the computer device can also render a three-dimensional display window for displaying the three-dimensional skeletal model S.

[0056] like Fig. 9 FIG. 1 is a flow chart of a rendering display window provided in an embodiment of the present application. Fig. 9 The display window shown includes a plurality of two-dimensional display windows {w0, …, w N2} and the 3D display window w 3dAfter rendering the above display window, the computer device can construct a registration controller A, which maps the user's operation, such as the key operation of the user on the keyboard, to a corresponding control action, thereby providing the user with a keyboard operation to guide the rendered display window to be adjusted synchronously, thereby achieving a more accurate registration purpose.

[0057] In the embodiment of the present application, multiple intersecting grids are displayed through multiple two-dimensional display windows, that is, for any target image, the camera is focused on the center of the target image, thereby controlling the camera's visual axis to align with the plane normal vector of the target image, and adjusting the upper end of the visual axis to align with the y-axis of the image coordinate system corresponding to the target image in the opposite direction, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image. Fig.10 FIG. 1 is a schematic diagram of a display effect of a two-dimensional rendering window provided in an embodiment of the present application. Fig.10 Four two-dimensional display windows are shown in FIG. 1 , each of which displays an example of a two-dimensional section formed by tangent between a selected ultrasound image (target image) and the model S based on the aforementioned steps. The line segments in these two-dimensional sections represent the intersecting grids obtained by the intersection of the ultrasound image and the model S. Fig.10 The intersecting grids shown allow users to visually observe the registration effect.

[0058] In a possible implementation of an embodiment of the present application, the registration controller constructed by the computer device may include multiple control actions corresponding to multiple keys on the keyboard. These actions may include the action of translating along the x-axis, y-axis and z-axis of the spatial coordinate system, and may also include the action of rotating the x-axis, y-axis and z-axis of the spatial coordinate system. Since translation or rotation both include translating or rotating along the positive or negative direction of the corresponding axis, there are 12 actions in total. These 12 actions may correspond to 12 keys on the keyboard, and when the user presses any one of the 12 keys on the keyboard, the computer device may select the corresponding action for it. The step size of each action may be defaulted to 1, and the step size may be adjusted according to actual needs, and the embodiment of the present application is not limited to this.

[0059] The user can adjust the registration process according to the above configured actions.

[0060] S105 . In response to an adjustment instruction for the intersecting grid displayed in any one of the two-dimensional display windows, positionally adjust the three-dimensional data to achieve registration between the two-dimensional image and the three-dimensional data.

[0061] In the embodiment of the present application, the user can make adjustments to the displayed intersecting grids, thereby adjusting the registration result to achieve more accurate registration.

[0062] As mentioned above, the control actions configured by the computer device include positive or negative translation or rotation along the x-axis, y-axis or z-axis of the spatial coordinate system. Therefore, the adjustment instructions of the user when adjusting the alignment results also include translation instructions or rotation instructions. The computer device can respond to the adjustment instructions and adjust the position of the three-dimensional data according to the adjustment method corresponding to the adjustment instructions.

[0063] Specifically, the translation instruction includes a positive translation instruction or a negative translation instruction, and the rotation instruction includes a positive rotation instruction or a negative rotation instruction. Therefore, the adjustment method corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis or z-axis of the spatial coordinate system, and the adjustment method corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis or z-axis of the spatial coordinate system.

[0064] like Fig.11 FIG. 1 is a flow chart of updating a rendering window provided in an embodiment of the present application. Fig.11 The process of updating the rendering display window based on the user's adjustment instruction is shown. Figure 1 and Fig.11 As shown, for a certain rendering display window, after receiving an instruction from the user to adjust the intersecting grids displayed in the window, the computer device can generate a rotation matrix based on the image coordinate system of the target image corresponding to the adjustment instruction and the adjustment method corresponding to the adjustment instruction, and then adjust the position of the model S according to the rotation matrix to obtain an updated model S'. The computer device can re-render the corresponding picture based on the updated model S' and display it through the display window to show the adjusted registration result.

[0065] Specifically, after determining the window that needs to be adjusted, the computer device can use the image coordinate system corresponding to the ultrasound image in the window as the standard coordinate system, and generate a rotation example and apply it to the three-dimensional model S according to the action corresponding to the user adjustment instruction, so that the action input by the user is orthogonal to the operation on the view displayed in the current window. Exemplarily, the action corresponding to the user adjustment instruction is an action on a certain axis, and the computer device will not affect the display angle of the view on the other two axes during the execution of the action. For example, based on the adjustment instruction input by the user, the computer device translates the displayed view along the x-axis, and does not affect the display angle of the view on the y-axis and z-axis.

[0066] The user can make multiple adjustments to ensure that the final registration result meets the requirements of the surgical operation and achieve the purpose of accurate registration.

[0067] In the embodiment of the present application, based on the intersecting grids displayed in each display window, the computer device can also evaluate the registration confidence and objectively evaluate the registration accuracy or registration quality.

[0068] In a possible implementation of the embodiment of the present application, as Fig.12 FIG. 1 is a schematic diagram of a registration confidence evaluation process provided by an embodiment of the present application. Fig.12 As shown, for the rendered intersecting grid, that is, the picture displayed in each display window, the computer device can determine the top pixels of each intersecting line that constitutes the intersecting grid in each two-dimensional display window, and expand based on each top pixel to obtain the expanded intersecting grid. By calculating the overlap between the expanded intersecting grid and the corresponding two-dimensional image in the two-dimensional display window, it can be used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

[0069] Specifically, each display window includes ultrasonic image Ii and its intersection line li with model S, and this intersection line is the intersection line of the whole model s, including the area that ultrasonic image has not scanned, so computer equipment can take top pixel to intersection line, be denoted as F(li), namely for each column pixel, only retain a certain amount of non-zero data. Then, for the top pixel obtained, morphological expansion algorithm can be used to obtain the expanded boundary, and then the boundary is used to sample ultrasonic image, calculate pixel value, and obtain bone surface evaluation value g. Because bone surface is white and strongly reflected, the larger the bone surface evaluation value g is, the better. By using the pixel number of top pixel to normalize the evaluation value g obtained, after averaging the g value corresponding to each display window rendered, as the registration confidence of evaluating the actual registration effect, feedback is given to the user.

[0070] In the embodiments of the present application, by combining computer automatic algorithms and manual professional judgment during the intraoperative registration stage, doctors are allowed to fine-tune the registration results in real time based on clinical experience, thereby significantly improving the registration accuracy between preoperative three-dimensional data and intraoperative two-dimensional images, overcoming the deviations that may occur when simply relying on automatic algorithms to deal with complex anatomical structures, so that the registration process can adapt to various complex situations. For example, when the standard plane is not scanned during ultrasound acquisition, this method can also be used for registration. The registration method provided in the embodiments of the present application can significantly improve the accuracy of diagnosis and the effectiveness of treatment, especially in minimally invasive surgery that requires precise positioning, and has extremely high application value.

[0071] Secondly, the registration method provided in the embodiment of the present application gives the operator a highly flexible control logic through interactive design, so that the doctor can adjust the registration parameters in real time according to the patient's specific situation, clinical needs and professional judgment. This flexibility can cope with various non-standard or emergency situations, so that the registration process can quickly adapt to the individual differences and special anatomical structures of different patients, thereby improving the overall medical effect.

[0072] Third, by applying the registration method provided in the embodiment of the present application, GPU accelerated computing technology can be used in the intraoperative registration process, which helps to significantly reduce the calculation time in the registration process. In particular, when the calculation model intersects the grid with the image plane, the processing speed of the GPU is 20 times faster than that of the traditional CPU. This efficient calculation can save waiting time during surgery, and this efficiency improvement is particularly important when the case is complicated.

[0073] Fourth, in the process of applying the registration method provided by the embodiment of the present application, the operation interface can be rendered intuitively and clearly through the display unit of the computer device, and multiple two-dimensional sections and three-dimensional models can be displayed at the same time to help doctors comprehensively observe the registration effect and quickly identify potential problems. The multi-window display and interactive adjustment method adopted by this method not only reduces the difficulty of operation through an intuitive and visual operation interface, but also reduces the risk of misoperation, so that even less experienced doctors can quickly master the specific usage methods.

[0074] Fifth, the embodiment of the present application introduces an objective evaluation mechanism based on image coincidence, which provides a quantifiable confidence index for the registration result. This objective evaluation mechanism can not only help doctors quickly judge the quality of registration, but also provide a reliable basis for subsequent diagnosis and treatment decisions. At the same time, this objective evaluation also provides valuable data support for medical quality control and subsequent research.

[0075] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0076] Reference Fig.13 , shows a schematic diagram of an image data registration device provided in an embodiment of the present application, which may specifically include a determination module 1301, a sorting module 1302, a construction module 1303, a rendering module 1304 and a registration module 1305, wherein: A determination module 1301 is used to acquire a plurality of two-dimensional images and determine a first direction based on the plurality of two-dimensional images; A sorting module 1302, configured to sort the plurality of two-dimensional images along the first direction; A construction module 1303 is used to construct a spatial coordinate system based on the sorted two-dimensional image sequence, wherein the direction of the z-axis of the spatial coordinate system is the same as the first direction; A rendering module 1304, used to respectively calculate the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and render a plurality of two-dimensional display windows based on the intersection, wherein any of the two-dimensional display windows is used to display the intersection grid of the two-dimensional image and the three-dimensional data; The registration module 1305 is used to adjust the position of the three-dimensional data in response to an adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows, so as to achieve registration between the two-dimensional image and the three-dimensional data.

[0077] In a possible implementation of the embodiment of the present application, the determining module 1301 may be specifically used to: Determine a capture position corresponding to each of the two-dimensional images, wherein the capture position includes a spatial position of a capture device used to capture each of the two-dimensional images when capturing each of the two-dimensional images; A first direction is determined based on a straight line obtained by fitting the plurality of acquisition positions.

[0078] In a possible implementation of the embodiment of the present application, the sorting module 1302 may be specifically used to: According to the spatial position of the acquisition device when each of the two-dimensional images is acquired, the two-dimensional images acquired at the corresponding position are sorted along the first direction to obtain a two-dimensional image sequence.

[0079] In a possible implementation of the embodiment of the present application, the construction module 1303 may be specifically used to: Calculating the bounding box of the sorted two-dimensional image sequence, and determining the bounding box coordinate system corresponding to the bounding box; The z-axis of the bounding box coordinate system is rotated to be the same as the first direction to construct a spatial coordinate system, wherein the x-axis, y-axis and z-axis of the spatial coordinate system coincide with the x-axis, y-axis and z-axis of the bounding box coordinate system respectively.

[0080] In a possible implementation of the embodiment of the present application, the rendering module 1304 may be specifically used for: Determine, from the plurality of the two-dimensional images, a plurality of candidate images whose corresponding intersections are greater than a threshold; According to a preset distance value, a plurality of target images are determined from the plurality of candidate images, and the intersection between each of the target images and the three-dimensional data is rendered to obtain a plurality of intersecting grids; The plurality of intersecting grids are displayed through a plurality of two-dimensional display windows.

[0081] In the embodiment of the present application, the rendering module 1304 may also be used for: For any of the target images, focusing the camera on the center of the target image; Control the visual axis of the camera to align with the plane normal vector of the target image, and adjust the upper end of the visual axis to align in the opposite direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image.

[0082] In the embodiment of the present application, the rendering module 1304 may also be used for: If the intersection between any of the two-dimensional images and the three-dimensional data is an empty set, after moving the three-dimensional data so that the center of the three-dimensional data coincides with the center of the bounding box of the two-dimensional image sequence, the rendering module 1304 is called to recalculate the intersection between each of the two-dimensional images and the three-dimensional data.

[0083] In a possible implementation manner of the embodiment of the present application, the adjustment instruction includes a translation instruction or a rotation instruction, the translation instruction includes a positive translation instruction or a negative translation instruction, the rotation instruction includes a positive rotation instruction or a negative rotation instruction, the adjustment method corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis or z-axis of the spatial coordinate system, and the adjustment method corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis or z-axis of the spatial coordinate system. The registration module 1305 can be specifically used to: In response to an adjustment instruction for the intersecting grids displayed in any of the two-dimensional display windows, the three-dimensional data is positionally adjusted according to an adjustment method corresponding to the adjustment instruction.

[0084] In the embodiment of the present application, the registration module 1305 may also be used for: generating a rotation matrix based on an image coordinate system of a target image corresponding to the adjustment instruction and an adjustment method corresponding to the adjustment instruction; The position of the three-dimensional data is adjusted according to the rotation matrix.

[0085] In a possible implementation of the embodiment of the present application, the registration module 1305 may also be used to: Determine the top pixel of each intersection line constituting the intersection grid in each of the two-dimensional display windows, and expand the expanded intersection grid based on each of the top pixels; The degree of overlap between the expanded intersecting grid and the corresponding two-dimensional image in the two-dimensional display window is calculated, and the degree of overlap is used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

[0086] In an embodiment of the present application, the two-dimensional image may include an ultrasound image, and the three-dimensional data may include CT data.

[0087] The present application provides an image data registration device, which can be used to implement the various steps in the aforementioned method embodiments to achieve accurate registration between two-dimensional images and three-dimensional data. For example, registration between two-dimensional ultrasound images and three-dimensional CT images can be achieved in an intraoperative scene.

[0088] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0089] Reference Fig.14 , shows a schematic diagram of a computer-assisted medical device provided in an embodiment of the present application. Fig.14 As shown, the computer-assisted medical device 1400 in the embodiment of the present application includes: a processor 1410, a memory 1420, and a computer program 1421 stored in the memory 1420 and executable on the processor 1410. When the processor 1410 executes the computer program 1421, the steps in each embodiment of the above-mentioned image data registration method are implemented, such as Figure 1 Alternatively, when the processor 1410 executes the computer program 1421, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig.13 Functions of modules 1301 to 1305 are shown.

[0090] Exemplarily, the computer program 1421 may be divided into one or more modules / units, which are stored in the memory 1420 and executed by the processor 1410 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 1421 in the computer-assisted medical device 1400. For example, the computer program 1421 may be divided into a determination module, a sorting module, a construction module, a rendering module, and a registration module, and the specific functions of each module are as follows: A determination module, configured to acquire a plurality of two-dimensional images and determine a first direction based on the plurality of two-dimensional images; A sorting module, used for sorting the plurality of the two-dimensional images along the first direction; A construction module, used to construct a spatial coordinate system based on the sorted two-dimensional image sequence, wherein the direction of the z-axis of the spatial coordinate system is the same as the first direction; A rendering module, used to respectively calculate the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and render a plurality of two-dimensional display windows based on the intersection, wherein any of the two-dimensional display windows is used to display the intersection grid of the two-dimensional image and the three-dimensional data; A registration module is used to adjust the position of the three-dimensional data in response to an adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows, so as to achieve registration between the two-dimensional image and the three-dimensional data.

[0091] The computer-assisted medical device 1400 may be a computer device capable of implementing the functions of the corresponding steps in the aforementioned method embodiments, and the computer device may be a desktop computer, a cloud server, etc. The computer-assisted medical device 1400 may include, but is not limited to, a processor 1410 and a memory 1420. Those skilled in the art will appreciate that Fig.14 It is only an example of the computer-assisted medical device 1400 and does not constitute a limitation of the computer-assisted medical device 1400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer-assisted medical device 1400 may also include input and output devices, network access devices, buses, etc.

[0092] The processor 1410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0093] The memory 1420 may be an internal storage unit of the computer-assisted medical device 1400, such as a hard disk or memory of the computer-assisted medical device 1400. The memory 1420 may also be an external storage device of the computer-assisted medical device 1400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer-assisted medical device 1400. Further, the memory 1420 may also include both an internal storage unit and an external storage device of the computer-assisted medical device 1400. The memory 1420 is used to store the computer program 1421 and other programs and data required by the computer-assisted medical device 1400. The memory 1420 may also be used to temporarily store data that has been output or is to be output.

[0094] An embodiment of the present application also discloses a computer-assisted medical device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the methods described in the aforementioned embodiments are implemented.

[0095] The embodiment of the present application also discloses a registration system, including an ultrasound probe and a computer device connected to the ultrasound probe. The ultrasound probe can be used to collect multiple two-dimensional images, such as two-dimensional ultrasound images; the computer device can process the multiple two-dimensional images collected by the ultrasound probe in accordance with the relevant steps described in the above-mentioned method embodiments to achieve registration between the intraoperative two-dimensional ultrasound image and the preoperative three-dimensional data, such as the preoperative three-dimensional CT data.

[0096] The embodiments of the present application further disclose a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the methods described in the above embodiments are implemented.

[0097] The embodiments of the present application further disclose a computer program product, including a computer program. When the computer program is run on a computer, the computer is enabled to execute the methods described in the aforementioned embodiments.

[0098] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image data registration method, characterized in that: include: Acquire a plurality of two-dimensional images, and determine a first direction based on the plurality of two-dimensional images; sorting the plurality of the two-dimensional images along the first direction; Based on the sorted two-dimensional image sequence, constructing a spatial coordinate system, wherein the direction of the z-axis of the spatial coordinate system is the same as the first direction; respectively calculating the intersections between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and rendering a plurality of two-dimensional display windows based on the intersections, wherein any of the two-dimensional display windows is used to display the intersection grids of the two-dimensional image and the three-dimensional data; In response to an adjustment instruction for the intersecting grid displayed in any one of the two-dimensional display windows, the three-dimensional data is positionally adjusted to achieve registration between the two-dimensional image and the three-dimensional data.

2. The method according to claim 1, characterized in that The determining the first direction based on the plurality of two-dimensional images comprises: Determine a capture position corresponding to each of the two-dimensional images, wherein the capture position includes a spatial position of a capture device used to capture each of the two-dimensional images when capturing each of the two-dimensional images; A first direction is determined based on a straight line obtained by fitting the plurality of acquisition positions.

3. The method according to claim 2, characterized in that The step of sorting the plurality of two-dimensional images along the first direction comprises: According to the spatial position of the acquisition device when each of the two-dimensional images is acquired, the two-dimensional images acquired at the corresponding position are sorted along the first direction to obtain a two-dimensional image sequence.

4. The method according to claim 1, characterized in that: The step of constructing a spatial coordinate system based on the sorted two-dimensional image sequence comprises: Calculating the bounding box of the sorted two-dimensional image sequence, and determining the bounding box coordinate system corresponding to the bounding box; The z-axis of the bounding box coordinate system is rotated to be the same as the first direction to construct a spatial coordinate system, wherein the x-axis, y-axis and z-axis of the spatial coordinate system coincide with the x-axis, y-axis and z-axis of the bounding box coordinate system respectively.

5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a plurality of two-dimensional display windows based on the intersection rendering includes: Determine, from the plurality of the two-dimensional images, a plurality of candidate images whose corresponding intersections are greater than a threshold; According to a preset distance value, a plurality of target images are determined from the plurality of candidate images, and the intersection between each of the target images and the three-dimensional data is rendered to obtain a plurality of intersecting grids; The plurality of intersecting grids are displayed through a plurality of two-dimensional display windows.

6. The method according to claim 5, characterized in that The displaying of the plurality of intersecting grids through a plurality of two-dimensional display windows comprises: For any of the target images, focusing the camera on the center of the target image; Control the visual axis of the camera to align with the plane normal vector of the target image, and adjust the upper end of the visual axis to align in the opposite direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image.

7. The method according to claim 5, characterized in that Before determining, from the plurality of two-dimensional images, a plurality of candidate images whose corresponding intersections are greater than a threshold, the method further includes: If the intersection between any of the two-dimensional images and the three-dimensional data is an empty set, after moving the three-dimensional data so that the center of the three-dimensional data coincides with the center of the bounding box of the two-dimensional image sequence, the intersection between each of the two-dimensional images and the three-dimensional data is recalculated.

8. The method according to any one of claims 1 to 4 or 6 or 7, characterized in that: The adjustment instruction includes a translation instruction or a rotation instruction, the translation instruction includes a positive translation instruction or a negative translation instruction, the rotation instruction includes a positive rotation instruction or a negative rotation instruction, the adjustment method corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis or z-axis of the spatial coordinate system, and the adjustment method corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis or z-axis of the spatial coordinate system, and the position adjustment of the three-dimensional data in response to the adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows includes: In response to an adjustment instruction for the intersecting grids displayed in any of the two-dimensional display windows, the three-dimensional data is positionally adjusted according to an adjustment method corresponding to the adjustment instruction.

9. The method according to claim 8, characterized in that The step of adjusting the position of the three-dimensional data according to the adjustment method corresponding to the adjustment instruction includes: generating a rotation matrix based on an image coordinate system of a target image corresponding to the adjustment instruction and an adjustment method corresponding to the adjustment instruction; The position of the three-dimensional data is adjusted according to the rotation matrix.

10. The method according to any one of claims 1 to 4 or 6 or 7 or 9, characterized in that: Also includes: Determine the top pixel of each intersection line constituting the intersection grid in each of the two-dimensional display windows, and expand the expanded intersection grid based on each of the top pixels; The degree of overlap between the expanded intersecting grid and the corresponding two-dimensional image in the two-dimensional display window is calculated, and the degree of overlap is used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

11. A computer-assisted medical device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the computer-assisted medical device is caused to implement the method according to any one of claims 1 to 10.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the method according to any one of claims 1 to 10 is executed.

Images