Image data registration method, device and computer program product

Abstract

Embodiments of the present application are applicable to the fields of computer-aided medical technology and image processing technology, and provide an image data registration method, device, and computer program product. The method includes: obtaining multiple two-dimensional images, and determining a first direction based on the multiple two-dimensional images; sorting the multiple two-dimensional images along the first direction; constructing a spatial coordinate system based on the sorted two-dimensional image sequence, where the direction of the z-axis of the spatial coordinate system is the same as the first direction; respectively calculating the intersection between each two-dimensional image arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and rendering multiple two-dimensional display windows based on the intersection, where any two-dimensional display window is used to display the intersection grid of the two-dimensional image and the three-dimensional data; in response to an adjustment instruction for any intersection grid, adjusting the position of the three-dimensional data to achieve the registration of the two-dimensional image and the three-dimensional data. Using the above method can improve the registration accuracy of the two-dimensional image and the three-dimensional data.

Description

Technical Field

[0001] The embodiments of the present application belong to the fields of computer-aided medical technology and image processing technology, and particularly relate to an image data registration method, device, and computer program product. Background Art

[0002] In modern medical diagnosis and surgical navigation, computed tomography (CT) and ultrasound imaging are two widely used imaging technologies, each with unique advantages. CT technology can provide high-resolution three-dimensional bone structures, while ultrasound technology has the characteristics of real-time, radiation-free, and portable. The spatial registration between these two types of images has always been a difficult problem in the industry, especially in the intraoperative environment, where the registration difficulty increases significantly due to factors such as time and data scanning.

[0003] Traditional registration between ultrasound images and CT data usually relies on manually selecting landmarks or on a one-time automatic registration algorithm based on a deep neural network. However, manually selecting landmarks is easily affected by the subjective factors of the operator, resulting in low registration accuracy. The automatic registration algorithm may also fail when the ultrasound image does not scan the established anatomical structure or when there is a large deformation in the patient's posture. In addition, due to the lack of flexibility, automatic registration cannot adjust the registration result when there is an error in registration, nor can it be adjusted in real time according to the specific intraoperative situation. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide an image data registration method, device, and computer program product to reduce the registration difficulty between two-dimensional images and three-dimensional data and improve the registration accuracy. For example, by applying the image data registration method provided by the embodiments of the present application, accurate registration between preoperative three-dimensional CT data and intraoperative two-dimensional ultrasound images can be achieved in the intraoperative scenario.

[0005] The first aspect of the embodiments of the present application provides an image data registration method, including:

[0006] Obtain multiple two-dimensional images, and determine a first direction based on the multiple two-dimensional images;

[0007] Sort the multiple two-dimensional images along the first direction;

[0008] Based on the sorted two-dimensional image sequence, construct a spatial coordinate system, and the direction of the z-axis of the spatial coordinate system is the same as the first direction;

[0009] Calculate the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered respectively, and render multiple two-dimensional display windows based on the intersection. Any one of the two-dimensional display windows is used to display the intersection grid between the two-dimensional image and the three-dimensional data;

[0010] In response to an adjustment instruction for the intersection grid displayed in any one of the two-dimensional display windows, adjust the position of the three-dimensional data to achieve registration between the two-dimensional image and the three-dimensional data.

[0011] Optionally, the determining the first direction based on multiple two-dimensional images includes:

[0012] Determine the acquisition position corresponding to each two-dimensional image, where the acquisition position includes the spatial position where the acquisition device used to acquire each two-dimensional image is located when acquiring each two-dimensional image;

[0013] Determine the first direction based on the straight line fitted from multiple acquisition positions.

[0014] Optionally, the sorting multiple two-dimensional images along the first direction includes:

[0015] Sort the two-dimensional images acquired at the corresponding positions along the first direction according to the spatial position where the acquisition device is located when acquiring each two-dimensional image to obtain a two-dimensional image sequence.

[0016] Optionally, the constructing a spatial coordinate system based on the sorted two-dimensional image sequence includes:

[0017] Calculate the bounding box of the sorted two-dimensional image sequence and determine the corresponding bounding box coordinate system;

[0018] Rotate the z-axis of the bounding box coordinate system to be the same as the first direction to construct a spatial coordinate system, where the x-axis, y-axis, and z-axis of the spatial coordinate system coincide with the x-axis, y-axis, and z-axis of the bounding box coordinate system respectively.

[0019] Optionally, the rendering multiple two-dimensional display windows based on the intersection includes:

[0020] Determine multiple candidate images from multiple two-dimensional images whose corresponding intersections are greater than a threshold;

[0021] Determine multiple target images from multiple candidate images according to a preset distance value, and render the intersection between each target image and the three-dimensional data to obtain multiple intersection grids;

[0022] Display multiple intersection grids through multiple two-dimensional display windows.

[0023] Optionally, the display of the multiple intersecting meshes through multiple two-dimensional display windows includes:

[0024] For any one of the target images, focus the camera on the center of the target image;

[0025] Control the optical axis of the camera to align with the plane normal vector of the target image, and adjust the upper end of the optical axis to align with the opposite direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image.

[0026] Optionally, before determining multiple candidate images with the intersection greater than the threshold from multiple two-dimensional images, it further includes:

[0027] If the intersection between any two-dimensional image and the three-dimensional data is an empty set, after moving the three-dimensional data to make the center of the three-dimensional data coincide with the center of the bounding box of the two-dimensional image sequence, recalculate the intersection between each two-dimensional image and the three-dimensional data.

[0028] Optionally, the adjustment instruction includes a translation instruction or a rotation instruction. The translation instruction includes a positive translation instruction or a negative translation instruction. The rotation instruction includes a positive rotation instruction or a negative rotation instruction. The adjustment method corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis, or z-axis of the space coordinate system. The adjustment method corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis, or z-axis of the space coordinate system. Responding to the adjustment instruction for the intersecting mesh displayed in any one of the two-dimensional display windows and performing position adjustment on the three-dimensional data includes:

[0029] Respond to the adjustment instruction for the intersecting mesh displayed in any one of the two-dimensional display windows, and perform position adjustment on the three-dimensional data according to the adjustment method corresponding to the adjustment instruction.

[0030] Optionally, the performing position adjustment on the three-dimensional data according to the adjustment method corresponding to the adjustment instruction includes:

[0031] Generate a rotation matrix based on the image coordinate system of the target image corresponding to the adjustment instruction and the adjustment method corresponding to the adjustment instruction;

[0032] Perform position adjustment on the three-dimensional data according to the rotation matrix.

[0033] Optionally, it further includes:

[0034] Determine the top pixels of each intersecting line that composes the intersecting grid in each of the two-dimensional display windows, and perform dilation based on each of the top pixels to obtain the dilated intersecting grid;

[0035] Calculate the coincidence degree between the dilated intersecting grid and the corresponding two-dimensional image in the two-dimensional display window, where the coincidence degree is used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

[0036] Optionally, the two-dimensional image includes an ultrasound image, and the three-dimensional data includes CT data.

[0037] A second aspect of the embodiments of the present application provides an image data registration device, including:

[0038] A determination module, configured to obtain multiple two-dimensional images and determine a first direction based on the multiple two-dimensional images;

[0039] A sorting module, configured to sort the multiple two-dimensional images along the first direction;

[0040] A construction module, configured to construct a spatial coordinate system based on the two-dimensional image sequence obtained after sorting, where the direction of the z-axis of the spatial coordinate system is the same as the first direction;

[0041] A rendering module, configured to calculate the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered respectively, and render multiple two-dimensional display windows based on the intersection, where any one of the two-dimensional display windows is used to display the intersecting grid of the two-dimensional image and the three-dimensional data;

[0042] A registration module, configured to, in response to an adjustment instruction for the intersecting grid displayed in any one of the two-dimensional display windows, adjust the position of the three-dimensional data to achieve the registration between the two-dimensional image and the three-dimensional data.

[0043] A third aspect of the embodiments of the present application provides a computer-aided medical device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the computer-aided medical device implements the method described in the first aspect above.

[0044] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the method described in the first aspect above is implemented.

[0045] A fifth aspect of the embodiments of the present application provides a computer program product, including a computer program which, when running, causes the method described in the first aspect above to be executed.

[0046] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0047] The embodiments of the present application can achieve accurate interactive registration between two-dimensional images and three-dimensional data. For example, accurate interactive registration between intraoperative two-dimensional ultrasound images and preoperative CT bone models can be achieved, overcoming the limitations of fully automatic algorithms in complex situations. Moreover, by providing an intuitive and interactive registration method, the embodiments of the present application allow doctors to make real-time adjustments based on professional judgment. Through multi-interface observation, the requirements for obvious features of the scanned images in the registration process are reduced, the registration efficiency is improved, the complexity of manual operations is reduced, and the accuracy of the registration results is maintained at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 It is a schematic diagram of an image data registration method provided by an embodiment of the present application;

[0050] Figure 2 It is an overall flowchart of an image data registration method provided by an embodiment of the present application;

[0051] Figure 3 It is a flowchart of a process for constructing a three-dimensional bone model provided by an embodiment of the present application;

[0052] Figure 4 It is a schematic diagram of an ultrasound probe provided by an embodiment of the present application;

[0053] Figure 5 It is a flowchart of a process for linear fitting provided by an embodiment of the present application;

[0054] Figure 6 It is a flowchart of a process for sorting two-dimensional images provided by an embodiment of the present application;

[0055] Figure 7 It is a schematic diagram of constructing a space coordinate system provided by an embodiment of the present application;

[0056] Figure 8 It is a flowchart of a process for determining candidate images provided by an embodiment of the present application;

[0057] Figure 9 is a schematic flow chart of rendering a display window provided by an embodiment of the present application;

[0058] Figure 10 is a schematic diagram of the display effect of a two-dimensional rendering window provided by an embodiment of the present application;

[0059] Figure 11 is a schematic flow chart of updating a rendering window provided by an embodiment of the present application;

[0060] Figure 12 is a schematic flow chart of registration confidence evaluation provided by an embodiment of the present application;

[0061] Figure 13 is a schematic diagram of an image data registration device provided by an embodiment of the present application;

[0062] Figure 14 is a schematic diagram of a computer-aided medical device provided by an embodiment of the present application. Detailed implementation manners

[0063] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from hindering the description of the present application.

[0064] The technical solution of the present application will be described below through specific embodiments.

[0065] Referring to Figure 1 , a schematic diagram of an image data registration method provided by an embodiment of the present application is shown, which may specifically include the following steps:

[0066] S101. Obtain multiple two-dimensional images, and determine a first direction based on the multiple two-dimensional images.

[0067] It should be noted that this method can be applied to a computer device, and the computer device can be various types of computer-aided medical devices. That is, the computer device or the computer-aided medical device can realize the registration of image data by executing each step of the method provided by the embodiment of the present application. In one example, the above computer device can be a device including a display unit, and the display unit can display relevant data or pictures during the registration process for the operator to understand the registration situation, facilitating the operator to adjust the registration process based on the intuitively displayed pictures to achieve the purpose of accurate registration.

[0068] The registration performed using this method can be the registration of a two-dimensional image and three-dimensional data. Exemplarily, the above two-dimensional image can be a two-dimensional ultrasound image acquired during the intraoperative stage, and the above three-dimensional data can be three-dimensional CT data acquired during the preoperative stage. For ease of understanding, the subsequent introduction in the embodiments of this application will take the registration of a two-dimensional ultrasound image and three-dimensional CT data during the intraoperative stage as an example to illustrate the specific implementation process of this method. It should be understood that this method is not limited to the registration between ultrasound images and CT data, and is also applicable to the registration between other types of two-dimensional images and other types of three-dimensional data. The embodiments of this application do not make any limitations in this regard.

[0069] As Figure 2 shown, it is a schematic diagram of the overall process of an image data registration method provided by an embodiment of this application. As Figure 2 shown, during the registration process, the input data of the computer device includes three types. One is the current target CT image data X, that is, the CT data of the patient acquired preoperatively; the other is the ultrasound image I' in space, that is, the ultrasound image of the patient acquired during the intraoperative stage. The registration process of this method is the registration of the above preoperative CT data and the intraoperative ultrasound image. During this process, the computer device can also receive the input data of the user. For example, Figure 2 the user keyboard input shown in can be used as the third input data of the computer device, and can be used to adjust the registration process in real time as the user control flow, so as to achieve the purpose of accurate registration and obtain the corresponding registration matrix.

[0070] As Figure 2 shown, the registration matrix obtained after the registration is also the output data of the computer device, that is, Figure 2 the registration matrix obtained by registering the preoperative CT data to the intraoperative ultrasound space shown in , and this registration matrix can be expressed as M = [R|T] ∈ R^(4*4).

[0071] As mentioned above, the three-dimensional CT data can be obtained by performing CT image acquisition on the patient during the preoperative stage. Exemplarily, during the preoperative stage, the CT data of the patient can be acquired using a CT device. As Figure 1 shown, the acquired CT data can be processed through a deep neural network to generate a grid S suitable for registration during the intraoperative stage.

[0072] Specifically, as Figure 3 shown, it is a schematic diagram of the process of constructing a three-dimensional bone model provided by an embodiment of this application. Combining Figure 1 and Figure 3As shown, for the current target CT image data X, i.e., the preoperative CT data, the deep neural network can be used to segment it into label data of the same size as the CT. The above label data can specifically indicate the patient's body parts and foreground and background regions in the image, etc. As Figure 3 shown, the segmentation result of the preoperative CT data can be expressed as Y ∈ [0,1]^(w*h*d). On this basis, the MC (marching cube) algorithm and the patch filtering algorithm can be used to generate a mesh S suitable for intraoperative registration. This mesh S is also the three-dimensional bone model obtained based on the CT data, which is composed of a point cloud and each line segment connecting the point cloud. Each point p in the point cloud ∈ R^3.

[0073] In the embodiments of the present application, multiple two-dimensional images can be collected using a calibrated ultrasound probe under the guidance of a navigation system. For example, N1 intraoperative ultrasound images I = {I0, ..., I N1} can be collected, where N1 ≥ 3.

[0074] As Figure 4 shown, it is a schematic diagram of an ultrasound probe provided by an embodiment of the present application. A tracer 401 can be installed on the ultrasound probe 400. The calibration of the ultrasound probe 400 can be completed in combination with the tracer 401 and a navigation system (not shown in the figure). When the calibrated ultrasound probe 400 collects each ultrasound image, its pixel point set Ut and spatial matrix Mt can be obtained at the same time. That is, for each collected ultrasound image, not only the relevant information of its image coordinate system can be obtained, but also the relevant information of its inherent spatial coordinate system can be included.

[0075] In a possible implementation manner of the embodiments of the present application, based on the multiple collected ultrasound images, the computer device can first determine a direction, for example, the first direction. This first direction can be used to assist in constructing the spatial coordinate system required for registration in the subsequent process.

[0076] In a specific implementation, a straight line can be fitted based on the multiple collected ultrasound images, and thus, based on the fitted straight line, the first direction can be determined.

[0077] As Figure 5 shown, it is a schematic flowchart of a straight line fitting provided by an embodiment of the present application. Figure 5 The fitting process shown is the process of fitting a straight line based on multiple ultrasound images collected by the ultrasound probe. As Figure 5 shown, each ultrasound image collected by the ultrasound probe can be expressed as I i ={U i , M i}, and multiple ultrasound images are Figure 5The image sequence I = {I0, ..., I N1} in it. The computer device can determine the acquisition position corresponding to each two-dimensional image, and thus determine the first direction based on the straight line obtained by fitting multiple acquisition positions.

[0078] In the embodiments of the present application, the above-mentioned acquisition position may include the spatial position where the acquisition device used to acquire each two-dimensional image is located when acquiring each two-dimensional image. For example, the acquisition device is an ultrasonic probe, so the straight line Gz can be obtained by fitting according to the spatial position of the end point P of the ultrasonic probe.

[0079] In a specific implementation, based on the self-spatial coordinate system of each ultrasonic image and the internally calibrated matrix, the spatial position P = {p0, ..., p N1} where the end point of the ultrasonic probe is located when each ultrasonic image is acquired can be calculated, and the least squares method (LSM) is used for fitting, so as to obtain the straight line Gz = LSM(p).

[0080] In the embodiments of the present application, for the fitted straight line Gz, the first direction can be determined by the directions of the first image and the last image among multiple ultrasonic images. Exemplarily, the above-mentioned first direction can be approximately from the first image to the last image in the ultrasonic image.

[0081] S102. Sort the multiple two-dimensional images along the first direction.

[0082] In the actual operation process, the positions of multiple ultrasonic images acquired by the ultrasonic probe in space are not strictly arranged in the order of acquisition time. Therefore, after the computer device obtains multiple images acquired by the ultrasonic probe, it can sort the multiple images along a certain direction, for example, sort the multiple two-dimensional ultrasonic images along the previously determined first direction, so that the spatial positions corresponding to each sorted image are arranged along a certain direction.

[0083] Specifically, according to the spatial position where the acquisition device is located when each two-dimensional image is acquired, the two-dimensional images acquired at the corresponding positions can be sorted along the first direction to obtain a two-dimensional image sequence.

[0084] As Figure 6 shown, it is a schematic flowchart of a method for sorting two-dimensional images provided by the embodiments of the present application, Figure 6 which shows the process of sorting the image sequence I along the first direction Gz to obtain the two-dimensional image sequence I'.

[0085] Specifically, referring to Figure 6, the computer device can sort the positions P of the ultrasonic probe tip when collecting each image along the first direction Gz. For example, sort the positions P of the ultrasonic probe tip according to the distance D, so as to output the sorted ultrasonic image sequence I'.

[0086] S103. Based on the obtained two-dimensional image sequence after sorting, construct a spatial coordinate system, and the direction of the z-axis of the spatial coordinate system is the same as the first direction.

[0087] In the embodiment of the present application, the computer device can construct a spatial coordinate system based on the obtained two-dimensional image sequence after sorting, so that the direction of the z-axis of the spatial coordinate system can be the same as the previously determined first direction.

[0088] As Figure 7 shown, it is a schematic diagram of constructing a spatial coordinate system provided by the embodiment of the present application. According to the Figure 7 shown process, after sorting the images to obtain the two-dimensional image sequence I', the computer device can calculate the bounding box of the two-dimensional image sequence I', and construct a spatial coordinate system based on the bounding box. In one example, an OBB (oriented bounding box) can be used to construct the spatial coordinate system.

[0089] Referring to Figure 7 shown, after the computer device completes the sorting of the ultrasonic images and obtains the image sequence I', it can calculate the OBB bounding box of the obtained two-dimensional image sequence I' in three dimensions, and determine the bounding box coordinate system corresponding to the OBB bounding box, to obtain the x-axis, y-axis and z-axis {Bx, By, Bz} of the bounding box coordinate system, as well as information such as the center c of the bounding box, the length, width and height (w, h, l) of the bounding box, etc. The computer device can construct a spatial coordinate system by rotating the z-axis of the bounding box coordinate system to be the same as the first direction, so that the x-axis, y-axis and z-axis of the spatial coordinate system coincide with the x-axis, y-axis and z-axis of the bounding box coordinate system respectively. Therefore, the previously constructed spatial coordinate system can be expressed as {Gx, Gy, Gz}, and the two-dimensional image sequence I' includes multiple two-dimensional images arranged along the z-axis of the spatial coordinate system.

[0090] S104. Calculate the intersection between each of the two-dimensional images arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered respectively, and render multiple two-dimensional display windows based on the intersection. Any one of the two-dimensional display windows is used to display the intersection grid of the two-dimensional image and the three-dimensional data.

[0091] After completing the above steps, collecting multiple two-dimensional ultrasonic images, and constructing a spatial coordinate system based on the two-dimensional ultrasonic images, the computer device can perform initial registration.

[0092] As Figure 1 shown, in the initial registration stage, the computer device may calculate the intersection between each two-dimensional image arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered. Specifically, the intersection may refer to the intersection between the two-dimensional ultrasound image and the mesh S (three-dimensional bone model S). The computer device may determine multiple candidate images I2 from multiple two-dimensional images, where the corresponding intersection of each candidate image is greater than the threshold T2. Exemplarily, the computer device may select N2 two-dimensional images with the largest intersections as candidate images.

[0093] In a possible implementation manner of the embodiment of the present application, when selecting N2 two-dimensional images with the largest intersections as candidate images, if the intersection between any two-dimensional image and the three-dimensional data is an empty set, for example, the intersection between any ultrasound image and the model S is an empty set, the computer device may move the three-dimensional data to make the center of the three-dimensional data coincide with the center c of the bounding box of the two-dimensional image sequence I'. Then, recalculate the intersection between each two-dimensional image and the three-dimensional data.

[0094] As Figure 8 shown, it is a schematic flowchart of a process for determining candidate images provided by an embodiment of the present application. Figure 8 As shown, it is the process of selecting N2 two-dimensional images with the largest intersections from the two-dimensional image sequence after sorting. Specifically, the computer device may calculate the intersection between the model S and each two-dimensional image. If there is no intersection between all two-dimensional images and the model S, for example, the intersection between all two-dimensional images and the model S is less than the threshold T1, the computer device may first move the model S to a position where the model center coincides with the center c of the OBB bounding box, and then calculate the intersection between the two. And select N2 images with intersections greater than T2 as candidate images.

[0095] In the embodiment of the present application, based on the selected multiple candidate images, the computer device may render them and display the registration situation with the model S through multiple two-dimensional display ports obtained by rendering.

[0096] In a possible implementation manner of the embodiment of the present application, multiple target images may be determined from multiple candidate images according to a preset distance value. Since each candidate image is arranged along the z-axis of the spatial coordinate system, multiple target images may be selected from multiple candidate images as evenly as possible at a certain distance. Then, the computer device may render the intersection between the target images and the three-dimensional data to obtain multiple intersecting meshes, and display the multiple intersecting meshes obtained by rendering through multiple two-dimensional display windows. The above intersecting meshes may be used to represent the registration situation between the two-dimensional target images and the model S, that is, multiple two-dimensional sections during the registration process. In addition, the computer device may also render a three-dimensional display window for displaying the three-dimensional bone model S.

[0097] As shown Figure 9 in the figure, it is a schematic flowchart of a process for rendering a display window provided by an embodiment of the present application. Figure 9 The shown display window includes multiple two-dimensional display windows {w0, …, w N2} and a three-dimensional display window w 3d . After rendering the above display window, the computer device can construct a registration controller A. By mapping the user's operations, such as key operations on the keyboard, to corresponding control actions, the user can operate on the keyboard to guide the rendered display window to be adjusted synchronously, achieving the purpose of more accurate registration.

[0098] In the embodiment of the present application, when multiple two-dimensional display windows are used to display multiple intersecting meshes, for any target image, the camera is focused on the center of the target image, so as to control the optical axis of the camera to be aligned with the plane normal vector of the target image, and adjust the upper end of the optical axis to be aligned with the reverse direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image. As Figure 10 shown in the figure, it is a schematic diagram of the display effect of a two-dimensional rendering window provided by an embodiment of the present application. Figure 10 4 two-dimensional display windows are shown in the figure. Each two-dimensional display window shows an example of a two-dimensional section formed by a selected ultrasound image (target image) tangent to the model S based on the foregoing steps. The line segments in these two-dimensional sections all represent the intersecting meshes obtained by the intersection of the ultrasound image and the model S. Based Figure 10 on the intersecting meshes shown in the figure, the user can intuitively observe the registration effect.

[0099] In a possible implementation manner of the embodiment of the present application, the registration controller constructed by the computer device may include multiple control actions corresponding one-to-one to multiple keys on the keyboard. These actions may include actions of translating along the x-axis, y-axis, and z-axis of the spatial coordinate system, and may also include actions of rotating around the x-axis, y-axis, and z-axis of the spatial coordinate system. Since translation or rotation includes translation or rotation along the positive or negative direction of the corresponding axis, there are a total of 12 such actions. These 12 actions can correspond to 12 keys on the keyboard. When the user presses any one of the 12 keys on the keyboard, the computer device can select a corresponding action for it. The step size of each action can be defaulted to 1, and this step size can be adjusted according to actual needs, which is not limited in the embodiment of the present application.

[0100] The user can adjust the registration process according to the above-configured actions.

[0101] S105. In response to an adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows, adjust the position of the three-dimensional data to achieve registration between the two-dimensional image and the three-dimensional data.

[0102] In the embodiment of the present application, the user can adjust the displayed intersecting grid, thereby adjusting the registration result to achieve more accurate registration.

[0103] As described above, the control actions configured by the computer device include positive or negative translation or rotation along the x-axis, y-axis, or z-axis of the space coordinate system. Therefore, the adjustment instructions when the user adjusts the registration result include translation instructions or rotation instructions, and the computer device can respond to the adjustment instructions and adjust the position of the three-dimensional data according to the adjustment method corresponding to the adjustment instructions.

[0104] Specifically, the translation instruction includes a positive translation instruction or a negative translation instruction, and the rotation instruction includes a positive rotation instruction or a negative rotation instruction. Therefore, the adjustment methods corresponding to the positive translation instruction or the negative translation instruction include positive translation or negative translation along the x-axis, y-axis, or z-axis of the space coordinate system, and the adjustment methods corresponding to the positive rotation instruction or the negative rotation instruction include positive rotation or negative rotation along the x-axis, y-axis, or z-axis of the space coordinate system.

[0105] As Figure 11 shown, it is a schematic flowchart of a process for updating a rendering window provided by an embodiment of the present application, Figure 11 showing the process of updating the rendering display window based on the user's adjustment instruction. Combining Figure 1 and Figure 11 shown, for a certain rendering display window, after receiving the instruction from the user to adjust the intersecting grid displayed in the window, the computer device can generate a rotation matrix based on the image coordinate system of the target image corresponding to the adjustment instruction and the adjustment method corresponding to the adjustment instruction, and then adjust the position of the model S according to the rotation matrix to obtain an updated model S'. The computer device can re-render the corresponding picture based on the updated model S' and display it through the display window to show the adjusted registration result.

[0106] Specifically, after determining the window that needs to be adjusted, the computer device can use the image coordinate system corresponding to the ultrasonic image in that window as the standard coordinate system, and generate a rotation matrix according to the action corresponding to the user adjustment instruction and apply it to the three-dimensional model S, so that the operation of the action input by the user on the view displayed in the current window is orthogonal. Exemplarily, if the action corresponding to the user adjustment instruction is an action on a certain axis, the computer device will not affect the display angles of the view on the other two axes during the execution of this action. For example, based on the adjustment instruction input by the user, the computer device translates the displayed view along the x-axis, which will not affect the display angles of the view on the y-axis and z-axis.

[0107] The user can make multiple adjustments so that the final registration result meets the requirements of the surgical operation, achieving the purpose of accurate registration.

[0108] In the embodiments of the present application, based on the intersecting grids displayed in each display window, the computer device can also evaluate the registration confidence level to objectively evaluate the registration accuracy or registration quality.

[0109] In a possible implementation manner of the embodiments of the present application, as Figure 12 shown, it is a schematic diagram of a registration confidence level evaluation process provided by the embodiments of the present application. As Figure 12 shown, for the rendered intersecting grids, that is, the pictures displayed in each display window, the computer device can determine the top pixels of each intersecting line that makes up the intersecting grid in each two-dimensional display window, and perform dilation based on each top pixel to obtain the dilated intersecting grid. By calculating the coincidence degree between the dilated intersecting grid and the corresponding two-dimensional image in the two-dimensional display window, it can be used to evaluate the registration confidence level between the two-dimensional image and the three-dimensional data.

[0110] Specifically, each display window includes an ultrasonic image Ii and its intersecting line li with the model S, and this intersecting line is the intersecting line of the entire model s, including the area not scanned by the ultrasonic image. Therefore, the computer device can take the top pixels of the intersecting line, denoted as F(li), that is, for each column of pixels, only a certain number of non-0 data are retained. Then, for the obtained top pixels, a morphological dilation algorithm can be used on them to obtain the dilated boundary, and then the ultrasonic image is sampled using this boundary to calculate the pixel values to obtain the bone surface evaluation value g. Since the bone surface shows strong white reflection, the larger the bone surface evaluation value g, the better. By normalizing the evaluation value g obtained by using the number of pixels of the top pixels, the average value of the g values corresponding to each display window obtained by rendering can be used as the registration confidence level for evaluating the actual registration effect and fed back to the user.

[0111] In the embodiments of the present application, by combining computer automatic algorithms and manual professional judgment during the intraoperative registration stage, doctors are allowed to micro-adjust the registration results in real time according to clinical experience, thereby significantly improving the registration accuracy between preoperative three-dimensional data and intraoperative two-dimensional images, overcoming the deviations that may occur when simply relying on automatic algorithms in dealing with complex anatomical structures, and enabling the registration process to adapt to various complex situations. For example, when the standard plane is not scanned during ultrasound acquisition, this method can also be used for registration. The registration method provided by the embodiments of the present application can significantly improve the accuracy of diagnosis and the effectiveness of treatment, especially in minimally invasive surgeries that require precise positioning, and has extremely high application value.

[0112] Secondly, the registration method provided by the embodiments of the present application endows the operator with a highly flexible control logic through interactive design, enabling doctors to adjust the registration parameters in real time according to the specific conditions of the patient, clinical needs, and professional judgment. This flexibility can cope with various non-standard or emergency situations, enabling the registration process to quickly adapt to the individual differences and special anatomical structures of different patients, thereby improving the overall medical effect.

[0113] Thirdly, when applying the registration method provided by the embodiments of the present application, GPU acceleration computing technology can be adopted during the intraoperative registration process, which helps to significantly reduce the computing time during the registration process. Especially when calculating the intersection grid of the computational model and the image plane, the processing speed of the GPU is 20 times faster than that of the traditional CPU. This high-efficiency computing can save the waiting time during the operation, and this efficiency improvement is particularly important when the case is complex.

[0114] Fourthly, during the process of applying the registration method provided by the embodiments of the present application, the operation interface can be intuitively and clearly rendered through the display unit of the computer device, while displaying multiple two-dimensional sections and three-dimensional models, helping doctors comprehensively observe the registration effect and quickly identify potential problems. The multi-window display and interactive adjustment methods adopted by this method, through an intuitive and visual operation interface, not only reduce the operation difficulty but also reduce the risk of misoperation, enabling even doctors with less experience to quickly master the specific usage method.

[0115] Fifthly, the embodiments of the present application introduce an objective evaluation mechanism based on image coincidence degree, providing a quantifiable confidence index for the registration results. This objective evaluation mechanism can not only help doctors quickly judge the registration quality but also provide a reliable basis for subsequent diagnosis and treatment decisions. At the same time, this objective evaluation also provides valuable data support for medical quality control and subsequent research.

[0116] It should be noted that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0117] Reference Figure 13 , which shows a schematic diagram of an image data registration device provided in an embodiment of the present application. Specifically, it may include a determination module 1301, a sorting module 1302, a construction module 1303, a rendering module 1304, and a registration module 1305, where:

[0118] The determination module 1301 is configured to obtain multiple two-dimensional images and determine a first direction based on the multiple two-dimensional images;

[0119] The sorting module 1302 is configured to sort the multiple two-dimensional images along the first direction;

[0120] The construction module 1303 is configured to construct a spatial coordinate system based on the sequence of two-dimensional images obtained after sorting. The direction of the z-axis of the spatial coordinate system is the same as the first direction;

[0121] The rendering module 1304 is configured to calculate the intersection between each two-dimensional image arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered, and render multiple two-dimensional display windows based on the intersection. Any one of the two-dimensional display windows is used to display the intersection grid between the two-dimensional image and the three-dimensional data;

[0122] The registration module 1305 is configured to, in response to an adjustment instruction for the intersection grid displayed in any one of the two-dimensional display windows, adjust the position of the three-dimensional data to achieve the registration between the two-dimensional image and the three-dimensional data.

[0123] In a possible implementation manner of the embodiment of the present application, the determination module 1301 may specifically be configured to:

[0124] Determine the acquisition position corresponding to each two-dimensional image, where the acquisition position includes the spatial position where the acquisition device used to acquire each two-dimensional image is located when acquiring each two-dimensional image;

[0125] Determine the first direction based on the straight line fitted from the multiple acquisition positions.

[0126] In a possible implementation manner of the embodiment of the present application, the sorting module 1302 may specifically be configured to:

[0127] Sort the two-dimensional images acquired at the corresponding positions along the first direction according to the spatial position where the acquisition device is located when acquiring each two-dimensional image to obtain a sequence of two-dimensional images.

[0128] In a possible implementation manner of the embodiment of the present application, the construction module 1303 may specifically be configured to:

[0129] Calculate the bounding box of the sorted two-dimensional image sequence, and determine the coordinate system of the bounding box corresponding to the bounding box;

[0130] Rotate the z-axis of the bounding box coordinate system to be the same as the first direction to construct a space coordinate system, where the x-axis, y-axis, and z-axis of the space coordinate system coincide with the x-axis, y-axis, and z-axis of the bounding box coordinate system respectively.

[0131] In a possible implementation manner of the embodiment of the present application, the rendering module 1304 may specifically be used for:

[0132] Determine multiple candidate images with an intersection greater than a threshold from multiple of the two-dimensional images;

[0133] Determine multiple target images from multiple of the candidate images according to a preset distance value, and render the intersection between each target image and the three-dimensional data to obtain multiple intersecting meshes;

[0134] Display the multiple intersecting meshes through multiple two-dimensional display windows.

[0135] In the embodiment of the present application, the rendering module 1304 may also be used for:

[0136] For any one of the target images, focus the camera on the center of the target image;

[0137] Control the optical axis of the camera to align with the normal vector of the plane of the target image, and adjust the upper end of the optical axis to align with the reverse direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image.

[0138] In the embodiment of the present application, the rendering module 1304 may also be used for:

[0139] If the intersection between any one of the two-dimensional images and the three-dimensional data is an empty set, after moving the three-dimensional data to make the center of the three-dimensional data coincide with the center of the bounding box of the two-dimensional image sequence, call the rendering module 1304 to recalculate the intersection between each two-dimensional image and the three-dimensional data.

[0140] In a possible implementation manner of the embodiment of the present application, the adjustment instruction includes a translation instruction or a rotation instruction. The translation instruction includes a positive translation instruction or a negative translation instruction. The rotation instruction includes a positive rotation instruction or a negative rotation instruction. The adjustment manner corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis, or z-axis of the space coordinate system. The adjustment manner corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis, or z-axis of the space coordinate system. Specifically, the registration module 1305 may be configured to:

[0141] In response to an adjustment instruction for the intersecting grid displayed in any of the two-dimensional display windows, adjust the position of the three-dimensional data according to the adjustment manner corresponding to the adjustment instruction.

[0142] In the embodiment of the present application, the registration module 1305 may further be configured to:

[0143] Generate a rotation matrix based on the image coordinate system of the target image corresponding to the adjustment instruction and the adjustment manner corresponding to the adjustment instruction;

[0144] Adjust the position of the three-dimensional data according to the rotation matrix.

[0145] In a possible implementation manner of the embodiment of the present application, the registration module 1305 may further be configured to:

[0146] Determine the top pixels of each intersecting line that composes the intersecting grid in each two-dimensional display window, and perform dilation based on each top pixel to obtain a dilated intersecting grid;

[0147] Calculate the coincidence degree between the dilated intersecting grid and the corresponding two-dimensional image in the two-dimensional display window. The coincidence degree is used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

[0148] In the embodiment of the present application, the two-dimensional image may include an ultrasound image, and the three-dimensional data may include CT data.

[0149] An image data registration device provided by an embodiment of the present application can implement each step in the foregoing method embodiments when applied, so as to achieve precise registration between a two-dimensional image and three-dimensional data. For example, registration between a two-dimensional ultrasound image and a three-dimensional CT image can be achieved in an intraoperative scenario.

[0150] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, refer to the description in the method embodiment part.

[0151] Refer to Figure 14, showing a schematic diagram of a computer-aided medical device provided by an embodiment of the present application. As Figure 14 shown, the computer-aided medical device 1400 in the embodiment of the present application includes: a processor 1410, a memory 1420, and a computer program 1421 stored in the memory 1420 and executable on the processor 1410. When the processor 1410 executes the computer program 1421, the steps in each embodiment of the above image data registration method are implemented, such as Figure 1 the steps S101 to S105 shown. Alternatively, when the processor 1410 executes the computer program 1421, the functions of each module / unit in each device embodiment above are implemented, such as Figure 13 the functions of the modules 1301 to 1305 shown.

[0152] Exemplarily, the computer program 1421 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 1420 and executed by the processor 1410 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments can be used to describe the execution process of the computer program 1421 in the computer-aided medical device 1400. For example, the computer program 1421 can be divided into a determination module, a sorting module, a construction module, a rendering module, and a registration module. The specific functions of each module are as follows:

[0153] The determination module is used to obtain multiple two-dimensional images and determine a first direction based on the multiple two-dimensional images;

[0154] The sorting module is used to sort the multiple two-dimensional images along the first direction;

[0155] The construction module is used to construct a spatial coordinate system based on the two-dimensional image sequence obtained after sorting, and the direction of the z-axis of the spatial coordinate system is the same as the first direction;

[0156] The rendering module is used to calculate the intersection between each two-dimensional image arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered respectively, and render multiple two-dimensional display windows based on the intersection. Any one of the two-dimensional display windows is used to display the intersection grid between the two-dimensional image and the three-dimensional data;

[0157] The registration module is used to adjust the position of the three-dimensional data in response to an adjustment instruction for the intersection grid displayed in any one of the two-dimensional display windows, so as to realize the registration between the two-dimensional image and the three-dimensional data.

[0158] The computer-aided medical device 1400 can be a computer device capable of implementing the functions of the corresponding steps in the foregoing various method embodiments. Such a computer device can be a desktop computer, a cloud server, etc. The computer-aided medical device 1400 may include, but is not limited to, a processor 1410 and a memory 1420. Those skilled in the art can understand that Figure 14 This is merely an example of the computer-aided medical device 1400 and does not constitute a limitation on the computer-aided medical device 1400. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, the computer-aided medical device 1400 may further include input / output devices, network access devices, a bus, etc.

[0159] The processor 1410 can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0160] The memory 1420 can be an internal storage unit of the computer-aided medical device 1400, such as the hard disk or memory of the computer-aided medical device 1400. The memory 1420 can also be an external storage device of the computer-aided medical device 1400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer-aided medical device 1400. Further, the memory 1420 can also include both the internal storage unit and the external storage device of the computer-aided medical device 1400. The memory 1420 is used to store the computer program 1421 and other programs and data required by the computer-aided medical device 1400. The memory 1420 can also be used to temporarily store data that has been output or is to be output.

[0161] The embodiments of the present application also disclose a computer-aided medical device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the methods described in the foregoing various embodiments are implemented.

[0162] The embodiments of the present application also disclose a registration system, including an ultrasound probe and a computer device communicatively connected to the ultrasound probe. Among them, the ultrasound probe can be used to collect multiple two-dimensional images, such as two-dimensional ultrasound images; the computer device can perform processing related to the steps described in the foregoing various method embodiments based on the multiple two-dimensional images collected by the ultrasound probe, and realize the registration between the intraoperative two-dimensional ultrasound images and the preoperative three-dimensional data, such as preoperative three-dimensional CT data.

[0163] The embodiments of the present application also disclose a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the methods described in the foregoing various embodiments are implemented.

[0164] The embodiments of the present application also disclose a computer program product, including a computer program. When the computer program runs on a computer, the computer is caused to execute the methods described in the foregoing various embodiments.

[0165] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image data registration method, characterized in that, Including: Obtaining multiple two-dimensional images, and determining a first direction based on the multiple two-dimensional images, where the multiple two-dimensional images include ultrasonic images; Sorting the multiple two-dimensional images along the first direction; Based on the sorted two-dimensional image sequence, constructing a spatial coordinate system, where the direction of the z-axis of the spatial coordinate system is the same as the first direction; Calculating the intersection between each two-dimensional image arranged along the z-axis of the spatial coordinate system and the three-dimensional data to be registered respectively, and rendering multiple two-dimensional display windows based on the intersection, and rendering a three-dimensional display window, where the three-dimensional display window is used to display a three-dimensional bone model, and any one of the two-dimensional display windows is used to display the intersection grid between the two-dimensional image and the three-dimensional data, and the intersection grid is represented by a line segment in a two-dimensional section formed by the two-dimensional image being tangent to the three-dimensional bone model; In response to an adjustment instruction for the intersection grid displayed in any one of the two-dimensional display windows, adjusting the position of the three-dimensional data to achieve registration between the two-dimensional image and the three-dimensional data.

2. The method according to claim 1, wherein The determining the first direction based on the multiple two-dimensional images includes: Determining the acquisition position corresponding to each two-dimensional image, where the acquisition position includes the spatial position where the acquisition device used to acquire each two-dimensional image is located when acquiring each two-dimensional image; Determining the first direction based on the straight line fitted from the multiple acquisition positions.

3. The method according to claim 2, wherein The sorting the multiple two-dimensional images along the first direction includes: Sorting the two-dimensional images acquired at the corresponding positions along the first direction according to the spatial position where the acquisition device is located when acquiring each two-dimensional image to obtain a two-dimensional image sequence.

4. The method according to claim 1, wherein The constructing a spatial coordinate system based on the sorted two-dimensional image sequence includes: Calculating the bounding box of the sorted two-dimensional image sequence, and determining the bounding box coordinate system corresponding to the bounding box; Rotating the z-axis of the bounding box coordinate system to be the same as the first direction to construct a spatial coordinate system, where the x-axis, y-axis, and z-axis of the spatial coordinate system coincide with the x-axis, y-axis, and z-axis of the bounding box coordinate system respectively.

5. The method according to any one of claims 1 to 4, characterized in that The rendering multiple two-dimensional display windows based on the intersection includes: Determining multiple candidate images from the multiple two-dimensional images whose corresponding intersections are greater than a threshold; Determining multiple target images from the multiple candidate images according to a preset distance value, and rendering the intersection between each target image and the three-dimensional data to obtain multiple intersection grids; Displaying the multiple intersection grids through multiple two-dimensional display windows.

6. The method according to claim 5, wherein The displaying the multiple intersection grids through multiple two-dimensional display windows includes: For any one of the target images, focusing the camera on the center of the target image; Controlling the optical axis of the camera to be aligned with the normal vector of the plane of the target image, and adjusting the upper end of the optical axis to be aligned with the reverse direction of the y-axis of the image coordinate system corresponding to the target image, so that the rendering result of the target image in the two-dimensional display window corresponding to the camera is the same as the result output by the acquisition device used to acquire the target image.

7. The method according to claim 5, characterized in that Before determining multiple candidate images whose corresponding intersections among the multiple two-dimensional images are greater than a threshold, the following steps are further included: If the intersection between any one of the two-dimensional images and the three-dimensional data is an empty set, after moving the three-dimensional data to make the center of the three-dimensional data coincide with the center of the bounding box of the two-dimensional image sequence, recalculate the intersection between each two-dimensional image and the three-dimensional data.

8. The method according to any one of claims 1 to 4, 6 or 7, characterized in that The adjustment instruction includes a translation instruction or a rotation instruction. The translation instruction includes a positive translation instruction or a negative translation instruction. The rotation instruction includes a positive rotation instruction or a negative rotation instruction. The adjustment method corresponding to the positive translation instruction or the negative translation instruction includes positive translation or negative translation along the x-axis, y-axis, or z-axis of the spatial coordinate system. The adjustment method corresponding to the positive rotation instruction or the negative rotation instruction includes positive rotation or negative rotation along the x-axis, y-axis, or z-axis of the spatial coordinate system. The position adjustment of the three-dimensional data in response to the adjustment instruction for the intersecting grid displayed in any one of the two-dimensional display windows includes: In response to the adjustment instruction for the intersecting grid displayed in any one of the two-dimensional display windows, perform position adjustment on the three-dimensional data according to the adjustment method corresponding to the adjustment instruction.

9. The method according to claim 8, wherein The performing position adjustment on the three-dimensional data according to the adjustment method corresponding to the adjustment instruction includes: Generate a rotation matrix based on the image coordinate system of the target image corresponding to the adjustment instruction and the adjustment method corresponding to the adjustment instruction; Perform position adjustment on the three-dimensional data according to the rotation matrix.

10. The method according to any one of claims 1 to 4 or 6 or 7 or 9, characterized in that The following steps are further included: Determine the top pixels of each intersecting line that makes up the intersecting grid in each two-dimensional display window, and perform dilation based on each top pixel to obtain the dilated intersecting grid; Calculate the coincidence degree between the dilated intersecting grid and the corresponding two-dimensional image in the two-dimensional display window. The coincidence degree is used to evaluate the registration confidence between the two-dimensional image and the three-dimensional data.

11. A computer-aided medical device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the computer-aided medical device implements the method according to any one of claims 1 to 10.

12. A computer program product, comprising a computer program, characterized in that, When the computer program runs, the method according to any one of claims 1 to 10 is executed.

Images