A method, device and storage medium for displaying point cloud registration results

Abstract

Embodiments of the present invention provide a method, device, and storage medium for displaying point cloud registration results, relating to the field of point cloud registration. The method comprises: acquiring first and second point cloud data of a target object; performing point cloud registration on the first and second point cloud data to obtain registered first and second point cloud data; converting the registered first and second point cloud data into first and second depth images, respectively; and performing image channel conversion on the first and second depth images to obtain three-channel depth images, which are used to display the point cloud registration results of the target object. Using the present invention, point cloud registration results can be quickly judged in clinical practice, more closely matching the display method familiar to traditional Chinese medicine practitioners in radiotherapy clinics, and enhancing the user's sensory experience.

Description

Technical Field

[0001] The present invention relates to the field of point cloud registration, and in particular to a method, device and storage medium for displaying point cloud registration results. Background Art

[0002] With the development of technology, doctors can now combine 2D images and 3D models of lesions for comprehensive analysis during diagnosis and treatment. The 3D model can be obtained by registering point cloud data from different angles of 3D scanning.

[0003] However, since point cloud data lacks boundary and domain information, it is difficult to directly determine whether the point cloud image after registration is successful. Related technologies can use meshing of point cloud images to determine the registration results, but the MESH display effect shows the rough surface texture of the lesion and deep grooves, resulting in poor display quality. Summary of the Invention

[0004] The present invention provides a method, device and storage medium for displaying point cloud registration results, which can quickly judge the registration results in clinical practice and provide support for the rapid and accurate formulation of radiotherapy plans.

[0005] In a first aspect, an embodiment of the present invention provides a method for displaying point cloud registration results, the method comprising:

[0006] Acquire first point cloud data and second point cloud data of the target object;

[0007] Performing point cloud registration on the first point cloud data and the second point cloud data to obtain registered first point cloud data and second point cloud data;

[0008] Converting the registered first point cloud data and the second point cloud data into a first depth image and a second depth image respectively;

[0009] Performing image channel conversion on the first depth image and the second depth image to obtain a three-channel depth image, and the three-channel depth image is used to display the point cloud registration result of the target object.

[0010] In one embodiment, the first point cloud data is a reference image in the point cloud registration process, point cloud data generated according to the outer contour of the planned image of the target object, or point cloud data of the target object stored after positioning is completed.

[0011] In one embodiment, the first depth image and the second depth image are both single-channel depth images; and performing image channel conversion on the first depth image and the second depth image to obtain a three-channel depth image includes:

[0012] Perform image channel conversion on the first depth image and the second depth image to obtain a first three-channel depth image and a second three-channel depth image;

[0013] The image colors of the first three-channel depth image and the image colors of the second three-channel depth image form complementary colors.

[0014] In one embodiment, the method further comprises:

[0015] If the three-channel depth image has color cast, determining that the first point cloud data and the first point cloud data registration are not aligned;

[0016] If the three-channel depth image does not have the color cast, it is determined that the first point cloud data and the first point cloud data registration are aligned.

[0017] In one embodiment, the method further comprises:

[0018] In response to an adjustment operation on any pose of any of the first point cloud data and the second point cloud data, a three-channel depth image corresponding to the adjusted pose is generated.

[0019] In one embodiment, the method further comprises:

[0020] Display the three-channel depth image corresponding to the adjusted posture.

[0021] In one embodiment, if the display interface includes a first area and a second area, the method further includes:

[0022] The registered first point cloud data and the second point cloud data are displayed in the first area, and the three-channel depth image is displayed in the second area.

[0023] In one embodiment, the first point cloud data and the second point cloud data are subjected to point cloud registration, and a relative offset between the first point cloud data and the second point cloud data is obtained; the method includes:

[0024] The relative offset between the first point cloud data and the second point cloud data is sent to a radiotherapy control device to control a treatment head of the radiotherapy device to stop beam emission or adjust the position of a treatment bed.

[0025] In a second aspect, an embodiment of the present invention further provides a device for displaying point cloud registration results, comprising: a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to execute a method for displaying point cloud registration results as described in any one of the first aspects.

[0026] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for displaying point cloud registration results as described in any one of the first aspects is executed.

[0027] The present invention provides a method, device, and storage medium for displaying point cloud registration results. By performing depth map conversion on the two registered point cloud data of the target object, and then performing image channel conversion on the converted depth map again, a three-channel depth image is finally used to display the point cloud registration results of the target object. This makes the point cloud registration results clearer and more in line with the display method familiar to radiotherapy clinicians, making it easier for physicians to quickly judge the registration results in clinical practice. Furthermore, because the three-channel depth image is smooth and uniform, similar to the two-dimensional slice display commonly used in clinical practice, it can enhance the user's sensory experience compared to the rough and grainy traditional point cloud display or MESH display. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A schematic structural diagram of a radiotherapy system provided by the present invention;

[0030] Figure 2 A schematic diagram of a process for displaying point cloud registration results provided by the present invention;

[0031] Figure 3 A schematic diagram of a flow chart of another method for displaying point cloud registration results provided by the present invention;

[0032] Figure 4 A schematic structural diagram of a display interface provided by the present invention;

[0033] Figure 5 A schematic diagram of a virtual device for displaying point cloud registration results provided by the present invention;

[0034] Figure 6 This is a schematic diagram of a device for displaying point cloud registration results provided by the present invention.

[0035] Icons: 1, radiotherapy equipment; 11, treatment bed; 12, radiotherapy device; 2, point cloud camera; 3, device for displaying point cloud registration results; 31, display device; 4, radiotherapy control device; 30, processor; 40, memory; 1000, acquisition module; 2000, registration module; 3000, first conversion module; 4000, second conversion module. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0038] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0039] Before explaining the present invention in detail, a possible application scenario of the present invention is first introduced.

[0040] During radiotherapy of a target object, such as a lesion within a patient's brain, high-energy radiation is usually used to kill tumor cells. Therefore, whether the lesion is accurately positioned at the radiotherapy site is related to the patient's radiotherapy effect.

[0041] In order to accurately locate the target object, it is usually necessary to align the point cloud data acquired in real time and the reference point cloud data, and finally obtain the point cloud data of the target object after registration. However, since the point cloud data is a set of discrete points in a three-dimensional coordinate system, there is no topological information and domain information between the points, and there is no boundary information for the entire set of point cloud data. Ideally, the two sets of point cloud data should overlap with each other after registration. However, due to the characteristics of the point cloud, the positional relationship between the points cannot be displayed. Even after registration, the unaligned point cloud data will still overlap with each other in the display effect. This makes it difficult to directly observe the registered point cloud data to determine whether the registration is aligned.

[0042] In related technologies, point cloud data can be MESHed (wireless mesh networked) so that in the three-dimensional object reconstructed by the point cloud, there is topological information and domain information between the points, and the result can be intuitively judged in the process of judging whether the registration is aligned. However, the MESHed point cloud data has a serious sense of graininess in the display effect. Although the registration result can be judged, it is not suitable for clinical diagnosis and treatment. Especially after the patient's head point cloud data is MESHed, the display effect is poor because it presents a realistic three-dimensional stereoscopic image of the head, and the three-dimensional head has a dense, uneven granular effect. At the same time, this three-dimensional display effect is different from the traditional display method used by physicians in clinical practice, which increases the difficulty of clinical cognition and the user experience is poor.

[0043] Therefore, the embodiments of the present application provide a method, device and storage medium for displaying point cloud registration results, so that the point cloud registration results are clear and can be quickly judged. At the same time, it is closer to the display method familiar to clinical Chinese medicine physicians in radiotherapy, thereby enhancing the user's sensory experience.

[0044] The method for displaying point cloud registration results provided by the present application will be explained below through multiple embodiments in conjunction with the accompanying drawings. The device for displaying point cloud registration results that executes the method for displaying point cloud registration results can be a computer device with processing capabilities, which can be a laptop, desktop computer, server, or other computer device.

[0045] In order to clearly illustrate the actual scenario of using the method for displaying point cloud registration results provided by the present application, an embodiment of the present invention provides a possible implementation of an optical positioning system for a radiotherapy body surface. Figure 1 This is a schematic diagram of the structure of an optical positioning system for radiotherapy provided by the present invention, such as Figure 1As shown, the radiotherapy body surface optical positioning system includes: a radiotherapy device 1, a point cloud camera 2, a device for displaying point cloud registration results 3, and a radiotherapy control device 4, wherein the radiotherapy device 1 includes a treatment bed 11 for carrying and positioning a patient, and a radiotherapy device 12 having a radiotherapy space; the point cloud camera 2 is connected to the device for displaying point cloud registration results 3, and the point cloud camera 2 can obtain first point cloud data of a target object (such as a patient's head or body) located on the treatment bed 11, and the device for displaying point cloud registration results 3 can obtain the first point cloud data and second point cloud data of the target object and perform registration, and then send the registration result to the radiotherapy control device 4; the radiotherapy control device 4 controls the movement of the treatment bed 11 according to the registration result, so that the lesion of the target object moves to the target radiotherapy position of the radiotherapy device 12 in the radiotherapy space, such as the isocenter of the radiotherapy device 12; the radiotherapy control device 4 can also control the radioactive source in the treatment head of the radiotherapy device 12 to be turned off according to the registration result, so that when the lesion of the target object deviates from the isocenter of the radiotherapy device 12 during radiotherapy, the treatment head is controlled to stop beam emission.

[0046] Here, the first point cloud data can be real-time point cloud data of the target object; the second point cloud data can be a reference image, which is point cloud data generated based on the outer contour of a planned image of the target object, such as a CT image and / or MRI image, or point cloud data of the target object stored after positioning is completed.

[0047] Before performing radiotherapy on a target patient, the target patient must be positioned. Prior to positioning, the patient is secured to the treatment couch 11. The point cloud camera 2 can be used to image the target patient's surface, generating real-time point cloud data of the target patient. Alternatively, a non-invasive method, such as a radiotherapy positioning film, can be used to secure the target patient to the treatment couch 11. The radiotherapy positioning film can be a head mask, a body phantom, or a head, neck, and shoulder mask.

[0048] It should be noted that the target object can be the part of the patient where the lesion is located, such as the patient's head, the patient's body, etc. If the target object is the patient's head, the target object photographed by the point cloud camera 2 can include the entire or partial head of the patient, such as the area where the patient's eyes and nose are located. Similarly, if the target object is the patient's body, the target object photographed by the point cloud camera 2 can include the entire or partial body where the lesion is located, such as the chest, abdomen, or other parts of the patient where the lesion is located.

[0049] The point cloud camera 2 can be installed on both sides of the treatment bed 11, or on the ceiling of the treatment room or other locations to capture the target object. If the shooting range of the point cloud camera 2 is wide enough, the point cloud camera 2 can be one, or more than two, to capture point cloud data from multiple angles.

[0050] After capturing the first point cloud data of the target object, the point cloud camera 2 may communicate the captured first point cloud data to the device 3 for displaying the point cloud registration result. Here, communication transmission includes wireless transmission or wired transmission. Wireless transmission may include WIFI, Bluetooth, NFC, and other transmission methods. Wired transmission may include cable transmission, etc.

[0051] The device 3 for displaying the point cloud registration results can also perform any of the following methods for displaying the point cloud registration results. Accordingly, the device 3 for displaying the point cloud registration results includes a display device 31 for displaying the point cloud registration results, such as a three-channel depth image obtained by performing the method for displaying the point cloud registration results. The device 3 for displaying the point cloud registration results can be a display screen having only a display function, or a display screen having both a touch screen function and a display function, and this application does not impose any restrictions on this.

[0052] The radiotherapy control device 4 may be a computer device with processing capabilities, which may be a laptop computer, a desktop computer, a server, a PLC circuit, etc.

[0053] In the above Figure 1 Based on the provided radiotherapy body surface optical positioning system, in order to clearly introduce the process of displaying the point cloud registration result, the present invention also provides a possible implementation method for displaying the point cloud registration result. Figure 2 This is a flow chart of a method for displaying point cloud registration results provided by the present invention. Figure 2 As shown, the method for displaying point cloud registration results includes:

[0054] S110: Acquire first point cloud data and second point cloud data of the target object.

[0055] The target object can be a part of the patient's head or body, or a portion of the head or body, or a pre-set phantom object, such as a head phantom or body phantom. For example, if treating an intracranial lesion, the target object is the patient's head; if treating an intrathoracic lesion, the target object is the patient's chest.

[0056] It should be noted that, with respect to the first point cloud data and the second point cloud data, during the process of positioning the target object, the first point cloud data may be the point cloud data of the target object acquired in real time by the point cloud camera 2, and the second point cloud data may be the point cloud data generated based on planning images such as CT images and / or MRI images; during the process of radiotherapy of the target object, the first point cloud data may be the point cloud data of the target object acquired in real time by the point cloud camera 2, and the second point cloud data may be the point cloud data generated based on planning images such as CT images and / or MRI images, or it may be the point cloud data of the target object acquired by the point cloud camera 2 after the positioning of the target object is completed.

[0057] Optionally, the first point cloud data of the target object acquired in real time by the point cloud camera 2 can be captured at a single angle or at multiple angles. A single point cloud camera 2 can capture data at a single angle, or at multiple angles. Furthermore, multiple point cloud cameras 2 can capture data at fixed angles, or at multiple angles. This embodiment does not impose any specific restrictions on the number or placement of point cloud cameras 2 in the radiotherapy body surface optical positioning system.

[0058] Optionally, if a plurality of point cloud data captured by one or more point cloud cameras 2 are obtained for the target object, the captured point cloud data need to be spliced ​​and fused to obtain first point cloud data of the target object acquired in real time.

[0059] Optionally, since planning images such as CT images and MRI images all have outer contours, the second point cloud data of the target object can be obtained according to the outer contours of the planning images such as CT images or MRI images.

[0060] In one possible implementation, taking a CT image as an example, if the target object is the patient's head, the patient wears a mask to scan the CT image. Since the HU value (Hounsfield unit, a dimensionless unit commonly used in computed tomography (CT) to express CT numerical values) of the mask material is different from the HU value of the human face skin or air, a suitable HU threshold can be selected to perform binary segmentation on the original CT image, and edge detection and contour extraction can be performed on the segmented binary image. For example, the Canny operator is used to extract the outer contour to obtain the outer contour information of the patient's skin. The outer contour information is composed of three-dimensional coordinates, and the second point cloud data of the target object can be obtained. The same method is used to obtain the second point cloud data of the target object based on the outer contour of the MRI image, which will not be repeated here.

[0061] Optionally, before performing contour extraction on a planning image such as a CT image or an MRI image, the image may be preprocessed, for example, by reducing noise in the image through image filtering.

[0062] In another possible implementation method, the obtained planned images such as CT images or MRI images can be directly imported into Mimics (medical image data processing) software, and the Segmentation tool can be used to segment the area of ​​interest of the target object. Then, the Calculate 3D (three-dimensional graphics calculator) tool can be used to perform three-dimensional reconstruction of the target object to obtain a three-dimensional reconstructed model of the target object. The model is then converted and saved as a point cloud format model to obtain the second point cloud data of the target object.

[0063] Optionally, if the target object is photographed not only by CT but also by MRI, the CT image and the MRI image may be fused first, and second point cloud data of the target object may be generated according to the outer contour of the fused image.

[0064] S120: Perform point cloud registration on the first point cloud data and the second point cloud data to obtain registered first point cloud data and second point cloud data.

[0065] In order to determine whether the current target object is positioned, or whether the target object moves during radiotherapy, it is necessary to perform point cloud registration on the first point cloud data and the second point cloud data to obtain the registered first point cloud data and the second point cloud data.

[0066] It should be noted that point cloud data refers to a set of vectors in a three-dimensional coordinate system, represented by (X, Y, Z) three-dimensional coordinates. Point cloud registration involves transforming the coordinates of all points in the floating image to the coordinate system of the reference image, using a rotation matrix based on the point-to-point correspondence between the two point cloud data. In this embodiment, the point cloud data of the target object acquired in real time by the point cloud camera 2, i.e., the first point cloud data, serves as the floating image; and the point cloud data of the target object generated from planning images such as CT images and / or MRI images, or acquired by the point cloud camera 2 after the target object has been positioned, i.e., the second point cloud data, serves as the reference image. By registering the floating image with the reference image, it is possible to determine whether the current target object has been positioned correctly or whether the target object has moved during radiotherapy.

[0067] In one possible implementation, a preset point cloud registration algorithm, such as an iterative closest point algorithm or a feature matching-based registration algorithm, can be used to calculate the rotation matrix. Alternatively, a point cloud registration algorithm compiled on an open source program platform can be used to automatically register two images. Simply inputting the point cloud data of the two images automatically generates a registered image.

[0068] In another possible implementation, after obtaining the first and second point cloud data of the target object, the point cloud data serving as the reference image can be cropped according to a preset body surface template to obtain a region of interest in the reference image. The entire floating image is then registered with the region of interest in the reference image to ultimately obtain the registered first and second point cloud data.

[0069] In another possible implementation, since the amount of 3D point cloud data typically acquired is enormous and accompanied by a lot of noise, inputting the original 3D point cloud data will increase the time required for the point cloud registration algorithm, and the presence of noise will affect the accuracy of the registration. Therefore, after obtaining the first and second point cloud data of the target object, in order to reduce the order of magnitude of the 3D point cloud data and reduce the noise in the data, a preset filtering algorithm, such as voxel grid filtering, can be first used to filter the first and second point cloud data separately, and then use the filtered first and second point cloud data for registration.

[0070] S130 , converting the registered first point cloud data and the second point cloud data into a first depth image and a second depth image respectively.

[0071] Since the depth image can use the different grayscale values ​​of each pixel to reflect the distance of a certain point from the camera, it can make up for the problem that the point cloud data cannot reflect the front-to-back relationship between points. Therefore, the first point cloud data and the second point cloud data after registration can be converted into a first depth image and a second depth image respectively. Since the depth image is a two-dimensional image, any pixel can only represent information in a two-dimensional direction, such as the (X, Y) direction, that is, only one observation angle, that is, an image of one posture, can be displayed in a depth image. Therefore, according to the preset posture, the first point cloud data and the second point cloud data after registration can be converted into the first depth image and the second depth image under the corresponding posture.

[0072] Optionally, during the conversion process, the projection distance between each 3D point cloud and the projection plane can be calculated, and this distance can be used as the grayscale value of the corresponding pixel in the point cloud to convert the entire point cloud data into a depth image. Optionally, during the calculation process, in order to avoid noise caused by some 3D point clouds being too far from the projection plane when converted to a depth map, a preset distance can be set. When the 3D point cloud is greater than this distance from the projection plane, the point cloud is discarded and does not participate in the depth map conversion.

[0073] In one possible implementation, an open-source conversion algorithm can be used to directly convert point cloud data into a depth map. By inputting the registered first and second point cloud data, as well as a preset pose, a first depth image and a second depth image can be generated.

[0074] S140 , performing image channel conversion on the first depth image and the second depth image to obtain a three-channel depth image, where the three-channel depth image is used to display a point cloud registration result of the target object.

[0075] In order to facilitate the observation of the results of point cloud registration, it is necessary to perform image channel conversion on the first and second depth images to obtain a three-channel depth image. A three-channel depth image means that the image is in RGB color mode, and each pixel in the image has three color channels: R (Red), G (Green), and B (Blue).

[0076] Optionally, during the image channel conversion process, the first depth image and the second depth image can be copied to the R, G, and B channels respectively, and the two images can be colored according to color requirements. For example, the RGB value of the required color can be input to obtain a three-channel depth image composed of two three-channel depth images, wherein the colors of the two three-channel depth images are different.

[0077] Since the depth image can show the front-to-back relationship of the position distance projection surface between pixels, and the two three-channel depth images after coloring can have color differences, the point cloud registration result of the target object can be determined by observing the two three-channel depth images. If only a single color can be displayed in the three-channel depth image screen, it means that the registration is aligned; if there is more than one color in the three-channel depth image screen, it means that the registration is not aligned. Based on this, a quick judgment of the point cloud registration results is achieved. At the same time, the final three-channel depth image is a two-dimensional image, which is more in line with the display method familiar in radiotherapy clinics, that is, it is closer to the two-dimensional slice display method commonly used in clinical practice, which reduces the adaptation process for physicians and enhances the sensory experience.

[0078] This embodiment converts the two registered point cloud data of the target object into depth images, and then performs image channel conversion on the converted depth images. Finally, a three-channel depth image is used to display the point cloud registration results of the target object. This makes the point cloud registration results clearer and more consistent with the display method familiar to radiotherapy clinicians, facilitating rapid clinical evaluation of the registration results. Furthermore, because the three-channel depth images are smooth and uniform, similar to the two-dimensional slice displays commonly used in clinical practice, they can enhance the user's sensory experience compared to the rough and grainy traditional point cloud or MESH displays.

[0079] Optionally, based on the above embodiment, the first point cloud data is the real-time point cloud data of the target object, and the second point cloud data is the reference image in the point cloud registration process, which is the point cloud data generated according to the outer contour of the planned image of the target object, or the point cloud data of the target object stored after the positioning is completed.

[0080] In this implementation, the first point cloud data is defined as the real-time point cloud data of the target object. Specifically, the first point cloud data is defined as the floating image during point cloud registration, while the second point cloud data is defined as the reference image during point cloud registration. During the registration process, the coordinates of all points in the first point cloud data are transformed using a rotation matrix to the coordinate system of the second point cloud data.

[0081] The first point cloud data as the floating image may be the point cloud data of the target object acquired in real time by the point cloud camera 2 during the positioning of the target object, or the point cloud data of the target object acquired in real time by the point cloud camera 2 during radiotherapy.

[0082] It should be noted that the planning image is a CT image or MRI image captured of the target object, and second point cloud data of the target object can be obtained based on the outer contour of the planning image. Alternatively, when the target object is fixed on the treatment couch 11 and the target object is positioned, that is, the target point / target area center of the target object lesion is aligned with the isocenter of the radiotherapy device 12, the point cloud data captured by the point cloud camera 2 for the target object is the point cloud data used as the reference image in the registration process, that is, the second point cloud data of the target object is obtained.

[0083] In this embodiment, by using point cloud data acquired in real time as a floating image, and using point cloud data generated by the outer contour of the planned image or the target object stored after the positioning is completed as a reference image, the point cloud data used as the reference image contains all point cloud data of the target object required for use during radiotherapy, which makes it easy to determine whether the current target object is positioned from the alignment results of the first point cloud image and the second point cloud image, or to determine whether the target object moves during radiotherapy.

[0084] In the above Figure 2 Based on the provided method for displaying point cloud registration results, in order to explain in detail the process of image channel conversion, the present invention also provides a possible implementation method for image channel conversion. In this embodiment, the first depth image and the second depth image are both single-channel depth images; in the above S140, the first depth image and the second depth image are converted into a three-channel depth image, including:

[0085] S142 , performing image channel conversion on the first depth image and the second depth image respectively to obtain a first three-channel depth image and a second three-channel depth image.

[0086] The image colors of the first three-channel depth image and the image colors of the second three-channel depth image form complementary colors.

[0087] A single-channel depth image is gray. Since the human eye is much more sensitive to color changes than grayscale changes, to facilitate quick observation of the point cloud registration results of the target object, the image colors of the first three-channel depth image and the second three-channel depth image can be set to complementary colors during the image channel conversion process. When the two images form complementary colors, it is very effective for observing details at the edge of the image where the original grayscale changes are very small.

[0088] In one possible implementation, after a single-channel depth image undergoes image channel conversion, two three-channel depth images are obtained. When coloring these two three-channel depth images, the input RGB1 and RGB2 must form complementary colors. For example, when RGB1 = (255, 255, 0), RGB2 = (0, 0, 255) can be input to make the image colors of the two three-channel depth images form complementary colors. RGB1 represents the RGB values ​​of the first three-channel depth image, and RGB2 represents the RGB values ​​of the second three-channel depth image.

[0089] In this embodiment, by converting two single-channel depth images into two three-channel depth images with complementary colors, when observing the three-channel depth image formed by the superposition of the two three-channel depth images, the details at the edge of the image can be captured more quickly, which is conducive to quickly observing the point cloud registration results of the target object.

[0090] In the above Figure 2 On the basis of the provided method for displaying point cloud registration results, in order to explain in detail the process of determining the point cloud registration result of the target object, the present invention also provides a possible implementation method for determining the point cloud registration result. Figure 3 This is a flow chart of a method for determining point cloud registration results provided by the present invention. Figure 3 As shown, the method includes:

[0091] S210: If the three-channel depth image has color cast, determine that the first point cloud data and the first point cloud data registration are not aligned.

[0092] S220: If the three-channel depth image does not have color cast, determine that the first point cloud data and the first point cloud data registration are aligned.

[0093] Because for two pixels that form complementary colors, if the two pixels are overlapped and aligned, gray will be displayed; if the two pixels are not overlapped and aligned, color cast will be displayed, that is, color will be displayed in the picture. In order to quickly judge the result of the point cloud registration from the three-channel depth image, the huge visual difference between gray and color is used to convert the first and second point cloud data after registration into a three-channel depth image through a single-channel depth image. Then, the superimposed display effect of two three-channel depth images with complementary colors will be displayed. That is, for the final displayed image, if there is color cast in the current image and color appears, it means that the first and second point cloud data after registration before conversion are not aligned after registration; if there is no color cast in the current image and the whole image is gray, it means that the first and second point cloud data after registration before conversion are aligned after registration.

[0094] Then, for the three-channel depth image finally displayed, we can use the principle that the superposition of complementary color pixels is gray, and the superposition of non-aligned complementary color pixels is color cast to quickly determine whether the point cloud registration result of the target object is aligned from the final three-channel depth image. While saving the registration judgment time, the pure gray image is more in line with the display method familiar to clinical physicians, thereby enhancing the user's sensory experience.

[0095] In the above Figure 2 Based on the method for displaying point cloud registration results provided, the present invention also provides another possible implementation method for displaying point cloud registration results. In this embodiment, the above method further includes:

[0096] S310 , in response to an adjustment operation on any pose of any point cloud data of the first point cloud data and the second point cloud data, generating a three-channel depth image corresponding to the adjusted pose.

[0097] It should be noted that since both the first and second point cloud data are three-dimensional data, and the depth image is a two-dimensional image, in the process of generating the depth image, it is necessary to select the perspective from which the human eye observes the point cloud data, that is, to select the pose from which the point cloud data is observed, and then the depth image can be generated at the corresponding pose.

[0098] In this embodiment, after the point cloud registration is completed, the posture of the human eye observing the point cloud data can be adjusted interactively using the mouse, and the posture adjustment operation can be performed on any point cloud data in the first point cloud data and the second point cloud data. The device 3 for displaying the point cloud registration results can respond to the adjustment operation of any posture of any point cloud data in the first point cloud data and the second point cloud data, generate a depth image under the corresponding posture, and then convert and generate a three-channel depth image corresponding to the adjusted posture.

[0099] In one possible implementation, the display device 31 has a touch screen function, and the posture of any point cloud data among the first point cloud data and the second point cloud data can be adjusted directly by touching the screen with a finger, and then the point cloud registration result display device 3 responds to the operation to generate a three-channel depth image corresponding to the adjusted posture.

[0100] Since the two point cloud data have been aligned and have the same coordinate system, no matter which point cloud data's pose is adjusted, when generating a depth image, the other point cloud data will also generate a depth image at the corresponding pose. Therefore, based on the three-channel depth images corresponding to the two adjusted poses, a three-channel depth image that can directly determine the alignment result is constructed.

[0101] In this embodiment, a three-channel depth image of the corresponding posture can be obtained by adjusting the arbitrary posture of the response point cloud data, which makes it convenient to observe the target object from multiple angles when formulating a radiotherapy plan, avoids the information error caused by a single-angle depth image, and facilitates doctors to quickly judge the alignment results in clinical practice.

[0102] Based on the above method, this embodiment further includes:

[0103] S312: Display the three-channel depth image corresponding to the adjusted posture.

[0104] The device 3 for displaying the point cloud registration results includes a display device 31, which can directly display the three-channel depth image corresponding to the adjusted pose on the display device 31. In other words, by simply adjusting the pose of either the first or second point cloud data using a mouse or touch screen, the three-channel depth image corresponding to the adjusted pose can be instantly displayed, enhancing the clinical physician's sensory experience.

[0105] In the above Figure 2 Based on the provided method for displaying point cloud registration results, the present invention also provides another possible implementation method for displaying point cloud registration results. Figure 4 This is a schematic diagram of the structure of a display interface provided by the present invention. Figure 4 As shown, if the display interface includes a first area and a second area, the above method further includes:

[0106] S410: Display the registered first point cloud data and the second point cloud data in the first area, and display the three-channel depth image in the second area.

[0107] If the display interface on the display device in the device for displaying the point cloud registration result 3 includes a first area and a second area, the shapes and sizes of the first area and the second area are not limited in this application.

[0108] Optionally, the registered first point cloud data and the second point cloud data can be displayed in the first area, and the three-channel depth image can be displayed in the second area. In one possible implementation, the position of any point cloud data in the registered first point cloud data and the second point cloud data can be adjusted in the first area using a mouse or touch screen. After the point cloud registration result displaying device 3 responds to the adjustment operation, the three-channel depth image corresponding to the adjusted position can be displayed in the second area.

[0109] Through this display method, the information contained in each posture of the point cloud registration can be displayed more comprehensively and instantly during the clinical diagnosis and treatment process, which is close to the display method commonly used by clinical physicians and saves physicians' time in judging the registration results.

[0110] In the above Figure 2 Based on the provided method for displaying point cloud registration results, the present invention also provides a possible implementation method for controlling radiotherapy plans. In the above method, the first point cloud data and the second point cloud data are point cloud registered, and the relative offset between the first point cloud data and the second point cloud data is obtained. The above method also includes:

[0111] S510: Send the relative offset between the first point cloud data and the second point cloud data to a radiotherapy control device to control the radiotherapy device treatment head to stop beam emission or adjust the position of the treatment bed.

[0112] During point cloud registration, the coordinates of all point clouds in the floating image are transformed to the coordinate system of the reference image using the rotation matrix between the two point clouds based on the correspondence between the points. This eliminates the relative offset between the two point clouds and completes the registration. In the ideal state of perfect alignment, the relative offset between the two point clouds is 0. However, in practice, a relative offset within a preset relative offset threshold (e.g., 0.1 mm) is allowed between the two registered point clouds.

[0113] Optionally, after performing point cloud registration of the first and second point cloud data, a relative offset between the point cloud coordinate positions of the first and second point cloud data can also be obtained. The relative offset between the point cloud coordinate positions of the first and second point cloud data is transmitted to the radiotherapy control device 4, which then determines the obtained relative offset according to a preset relative offset threshold. If the relative offset between the first and second point cloud data is within the preset relative offset threshold, the radiotherapy control device 4 does not need to take any action. If the relative offset between the first and second point cloud data is not within the preset relative offset threshold, and if the first point cloud data is point cloud data of the target object acquired in real time by the point cloud camera 2 during radiotherapy, the radiotherapy control device 4 controls the treatment head in the radiotherapy device 12 to terminate beam delivery. If the first point cloud data is point cloud data of the target object acquired in real time by the point cloud camera 2 during positioning of the target object, the radiotherapy control device 4 controls the position of the treatment couch 11 to adjust the position of the target object's lesion to the isocenter of the radiotherapy device 12.

[0114] In one possible implementation, the radiotherapy control device 4 may include an alarm device with a three-color indicator light. When the radiotherapy treatment head needs to be instructed to stop beam delivery, the radiotherapy control device 15 displays a red light; when the treatment couch needs to be adjusted, the radiotherapy control device 15 displays a yellow light; and when no adjustment is required, the radiotherapy control device 15 displays a green light. Furthermore, when the radiotherapy control device 15 displays a red or yellow light, a voice alarm of a preset decibel level may be emitted to alert staff. Optionally, the radiotherapy control device 4 may also include a display device, such as a screen, to facilitate prompting staff. If control by the radiotherapy control device 4 is required, the text "Stop Treatment" or "Adjust Treatment Couch" may be displayed directly on the screen, although this is not a limitation in this embodiment.

[0115] In this embodiment, by sending the point cloud registration result to the radiotherapy control device in a timely manner, the radiotherapy control device can promptly instruct the staff to adjust the radiotherapy plan, thereby ensuring the reliability and safety of the radiotherapy plan.

[0116] Figure 5 A schematic diagram of a virtual device for displaying point cloud registration results provided by the present invention, such as Figure 5 As shown, the virtual device for displaying the point cloud registration result includes:

[0117] The acquisition module 1000 is used to acquire first point cloud data and second point cloud data of a target object.

[0118] The registration module 2000 is used to perform point cloud registration on the first point cloud data and the second point cloud data to obtain the registered first point cloud data and the second point cloud data.

[0119] A first conversion module 3000, configured to convert the registered first point cloud data and the second point cloud data into a first depth image and a second depth image respectively;

[0120] The second conversion module 4000 is used to perform image channel conversion on the first depth image and the second depth image to obtain a three-channel depth image. The three-channel depth image is used to display the point cloud registration result of the target object.

[0121] In one embodiment, the first point cloud data is real-time point cloud data of the target object, and the second point cloud data is a reference image in the point cloud registration process, which is point cloud data generated based on the outer contour of the planned image of the target object, or point cloud data of the target object stored after the positioning is completed.

[0122] In one embodiment, the second conversion module 4000 is further configured to perform image channel conversion on the first depth image and the second depth image, respectively, to obtain a first three-channel depth image and a second three-channel depth image; wherein the image color of the first three-channel depth image forms a complementary color with the image color of the second three-channel depth image.

[0123] In one embodiment, the virtual device displaying the point cloud registration result also includes a determination module for determining that the first point cloud data and the first point cloud data registration are not aligned if there is color cast in the three-channel depth image; and for determining that the first point cloud data and the first point cloud data registration are aligned if there is no color cast in the three-channel depth image.

[0124] In one embodiment, the virtual device displaying the point cloud registration result further includes an adjustment module for responding to an adjustment operation on any pose of any point cloud data among the first point cloud data and the second point cloud data, and generating a three-channel depth image corresponding to the adjusted pose.

[0125] In one embodiment, the virtual device for displaying the point cloud registration result further includes a display module for displaying the three-channel depth image corresponding to the adjusted posture.

[0126] In one embodiment, the display module is further configured to display the registered first point cloud data and the second point cloud data in the first area, and display the three-channel depth image in the second area if the display interface includes a first area and a second area.

[0127] In one embodiment, the registration module 2000 is further configured to perform point cloud registration on the first point cloud data and the second point cloud data to obtain a relative offset between the first point cloud data and the second point cloud data.

[0128] In one embodiment, the virtual device for displaying the point cloud registration result further includes a control module for sending the relative offset between the first point cloud data and the second point cloud data to a radiotherapy control device to control the radiotherapy device treatment head to stop beam emission or adjust the position of the treatment bed.

[0129] The above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0130] Figure 6 This is a schematic diagram of a device for displaying point cloud registration results provided by the present invention. The device can be a computing device or server with computing and processing capabilities.

[0131] The device 3 for displaying point cloud registration results includes a processor 30 and a memory 40. The memory 40 stores a computer program executable by the processor 30. The processor 30 executes the computer program to perform the above method embodiment. The specific implementation method and technical effects are similar and will not be repeated here.

[0132] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for displaying the point cloud registration result in the above embodiment is executed.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional units.

[0136] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor (English: Processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated: ROM), a random access memory (English: Random Access Memory, abbreviated: RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0137] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for displaying point cloud registration results, characterized in that: The method comprises: Acquire first point cloud data and second point cloud data of the target object; Performing point cloud registration on the first point cloud data and the second point cloud data to obtain registered first point cloud data and second point cloud data; Converting the registered first point cloud data and the second point cloud data into a first depth image and a second depth image respectively; Performing image channel conversion on the first depth image and the second depth image to obtain a three-channel depth image, wherein the three-channel depth image is used to display a point cloud registration result of the target object; The first depth image and the second depth image are both single-channel depth images; and performing image channel conversion on the first depth image and the second depth image to obtain a three-channel depth image includes: Performing image channel conversion on the first depth image and the second depth image respectively to obtain a first three-channel depth image and a second three-channel depth image; wherein the image color of the first three-channel depth image and the image color of the second three-channel depth image form a complementary color; Displaying the three-channel depth image formed by superimposing the first three-channel depth image and the second three-channel depth image; If color exists in the three-channel depth image formed by the superposition, determining that the first point cloud data and the second point cloud data are not aligned; If the three-channel depth image formed by the superposition is displayed only in gray, it is determined that the first point cloud data and the second point cloud data are aligned.

2. The method according to claim 1, characterized in that The first point cloud data is the real-time point cloud data of the target object, and the second point cloud data is the reference image in the point cloud registration process, which is the point cloud data generated according to the outer contour of the planned image of the target object, or the point cloud data of the target object stored after the positioning is completed.

3. The method according to claim 1, characterized in that The method further comprises: In response to an adjustment operation on any pose of any of the first point cloud data and the second point cloud data, a three-channel depth image consisting of the superposition corresponding to the adjusted pose is generated.

4. The method according to claim 3, characterized in that The method further comprises: The depth image of the three channels formed by the superposition corresponding to the adjusted posture is displayed.

5. The method according to claim 1, wherein If the display interface includes a first area and a second area, the method further includes: The registered first point cloud data and the second point cloud data are displayed in the first area, and the three-channel depth image formed by the superposition is displayed in the second area.

6. The method according to claim 1, characterized in that Performing point cloud registration on the first point cloud data and the second point cloud data, and obtaining a relative offset between the first point cloud data and the second point cloud data; the method comprises: The relative offset between the first point cloud data and the second point cloud data is sent to a radiotherapy control device to control a treatment head of the radiotherapy device to stop beam emission or adjust the position of a treatment bed.

7. A device for displaying point cloud registration results, characterized in that: include: A processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to perform the method for displaying point cloud registration results according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, executes the method for displaying point cloud registration results according to any one of claims 1 to 5.

Images