Point cloud data processing method and device, computer device, and radiotherapy system

Abstract

This application relates to a point cloud data processing method, apparatus, computer equipment, and radiotherapy system. The method includes: acquiring first point cloud data captured by each camera device; there are at least two camera devices positioned at different locations; determining matching point pairs in the first point cloud data of every two camera devices; removing a first matching point from the matching point pairs and retaining a second matching point; and fusing the second matching point with the non-matching points in each of the first point cloud data sets to obtain a fused point cloud image; wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal to the first point cloud data and the line containing the depth direction of the corresponding camera device. Applied in radiotherapy scenarios, this method can accelerate radiotherapy workflow and throughput, improving the effectiveness and safety of radiotherapy.

Description

Technical Field

[0001] This application relates to the field of point cloud fusion processing technology, and in particular to a point cloud data processing method, apparatus, computer equipment, and radiotherapy system. Background Technology

[0002] In optically guided radiotherapy systems, a camera is needed to provide simulated positioning information and develop a radiotherapy plan. The calculated required radiation dose is then transmitted to the tumor center in the form of a radiation beam.

[0003] To ensure that the radiation dose reaches the tumor accurately, it is necessary to maintain consistency in the patient's body posture during the imaging and treatment processes, accurately position the patient before treatment, and monitor the patient's physiological and non-physiological movements in real time during treatment.

[0004] To improve system stability, multiple cameras are often used to collect point cloud information from the patient's body surface, avoiding the impact of occlusion during radiotherapy. The fusion of point cloud information from multiple cameras plays a crucial role; however, current point cloud fusion methods mostly involve simply overlaying the point clouds collected from multiple cameras or converting the point clouds into surfaces for overlay. This process is time-consuming and significantly impacts real-time tracking in radiotherapy systems. Summary of the Invention

[0005] Therefore, it is necessary to provide a point cloud data processing method, device, computer equipment, and radiotherapy system that can effectively improve the speed of point cloud fusion computing in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a point cloud data processing method, which includes:

[0007] Acquire the first point cloud data captured by each camera device; there are at least two camera devices, and they are set up in different locations;

[0008] Identify matching point pairs in the first point cloud data of every two camera devices;

[0009] Remove the first matching point from the matching point pair and keep the second matching point, then fuse the second matching point with the non-matching points in each of the first point cloud data to obtain a point cloud fused image;

[0010] Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0011] In one embodiment, the method further includes, prior to the step of determining matching point pairs in the first point cloud data of every two camera devices:

[0012] The first target point cloud data is removed from the first point cloud data to obtain the second point cloud data of each camera device; wherein the first target point cloud data is the point cloud data with a first included angle greater than a first preset included angle, or the point cloud data with a cosine value of the first included angle less than a first preset value;

[0013] Determine matching point pairs in the first point cloud data of every two camera devices, including:

[0014] Identify matching point pairs in the second point cloud data of every two camera devices;

[0015] The first matching point is removed from the matching point pairs, and the second matching point is retained. The second matching point is then fused with the non-matching points in each of the first point cloud data to obtain a fused point cloud image, including:

[0016] Remove the first matching point from the matching point pair and keep the second matching point, then fuse the second matching point with the non-matching points in each second point cloud data to obtain a point cloud fused image.

[0017] In one embodiment, prior to the step of removing the first matching point from the matching point pair and retaining the second matching point, the method includes:

[0018] Remove the target matching point pairs from the matching point pairs to obtain the filtered matching point pairs; the target matching point pairs are matching point pairs whose second included angle is greater than the second preset angle, or matching point pairs whose absolute value of the cosine of the second included angle is less than the second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair;

[0019] Remove the first matching point from the matching point pair and keep the second matching point, including:

[0020] Remove the first matching point from the filtered matching point pairs and keep the second matching point.

[0021] In one embodiment, before the step of removing the first matching point from the matching point pair and retaining the second matching point, the method further includes:

[0022] Deduplication is performed on the first point cloud data from different camera devices to obtain the third point cloud data;

[0023] The first matching point is removed from the matching point pairs, and the second matching point is retained. The second matching point is then fused with the non-matching points in each of the first point cloud data to obtain a fused point cloud image, including:

[0024] The first matching point is removed from the matching point pair, the second matching point is retained, and the second matching point is fused with the non-matching points in each third point cloud data to obtain the point cloud fused image.

[0025] In one embodiment, deduplication is performed between the first point cloud data from different camera devices to obtain third point cloud data, including:

[0026] Calculate the average distance between nearest points of the first point cloud data for each camera device;

[0027] A distance threshold is determined based on the average distance of each camera device;

[0028] The second target point cloud data is removed from the first point cloud data to obtain the third point cloud data; the second target point cloud data is the point cloud data in the first point cloud data of each camera device whose distance from the first point cloud data of other camera devices is less than a distance threshold.

[0029] In one embodiment, the first point cloud data is point cloud data obtained by the camera device from the object to be treated. After obtaining the point cloud fused image, the method further includes:

[0030] Send the point cloud fusion image to the workstation, which is connected to the radiotherapy equipment.

[0031] Secondly, a point cloud data processing device is provided, the device comprising:

[0032] The first point cloud acquisition module is used to acquire the first point cloud data captured by each camera device; there are at least two camera devices, which are set in different locations;

[0033] The matching point pair determination module is used to determine the matching point pairs in the first point cloud data of every two camera devices;

[0034] The point cloud fusion image acquisition module is used to remove the first matching point from the matching point pair and retain the second matching point, and fuse the second matching point with the non-matching points in each of the first point cloud data to obtain the point cloud fusion image;

[0035] Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0036] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0037] Fourthly, a radiation therapy system is provided, comprising:

[0038] At least two camera devices are installed in different locations in the treatment room. The camera devices are used to photograph the patients to be treated in the treatment room to obtain the first point cloud data.

[0039] The aforementioned computer equipment is communicatively connected to each camera device;

[0040] A workstation, connected to a computer device, is used to determine the target area of ​​the object to be treated based on point cloud fusion images;

[0041] A radiation therapy device, connected to a workstation, used to perform targeted radiation therapy on a target area under the control of the workstation.

[0042] Fifthly, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] The above-mentioned point cloud data processing method, apparatus, computer equipment, and radiotherapy system have at least the following beneficial effects:

[0044] The process involves acquiring initial point cloud data from each of at least two cameras positioned at different locations. Matching points are registered from the initial point cloud data captured by each camera to obtain matching point pairs for point cloud fusion. The first matching point in each pair is discarded, and the second matching point is retained. The second matching point, being closer to the depth direction of the camera compared to the first, is used for point cloud data fusion based on the filtered matching points, resulting in a fused point cloud image from multiple cameras. This method avoids overlap after fusion, optimizes data volume, improves the speed of point cloud fusion computation, and retains more high-quality point cloud data, resulting in fast and accurate fusion. In radiotherapy scenarios, this can accelerate the radiotherapy workflow and throughput, improving the effectiveness and safety of radiotherapy. Attached Figure Description

[0045] Figure 1 This is an application environment diagram of a point cloud data processing method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a point cloud data processing method in one embodiment;

[0047] Figure 3a and Figure 3b This is a schematic diagram of the first angle between the normal and the line containing the depth direction according to one or more embodiments;

[0048] Figure 4This is a flowchart illustrating some steps of a point cloud data processing method in yet another embodiment.

[0049] Figure 5 This is a schematic diagram of some steps in the point cloud data processing method in another embodiment;

[0050] Figure 6 This is a partial flowchart of a point cloud data processing method in another embodiment;

[0051] Figure 7 This is a flowchart illustrating the steps of deduplicating first point cloud data from different camera devices to obtain third point cloud data in one embodiment.

[0052] Figure 8a This is a schematic diagram illustrating the effect of fusing point cloud data before deduplication in one embodiment.

[0053] Figure 8b This is a schematic diagram of the point cloud fusion image obtained by performing point cloud data fusion after point cloud deduplication in one embodiment.

[0054] Figure 9a and 9b This is a schematic diagram of the first point cloud data collected by the first two cameras without processing, as shown in one embodiment.

[0055] Figure 9c In one embodiment, the fusion is achieved by performing a point cloud data processing method. Figure 9a and Figure 9b A schematic diagram of a point cloud fusion image obtained from point cloud data;

[0056] Figures 10a-10c These are schematic diagrams of the first point cloud data collected by three camera devices in one embodiment;

[0057] Figure 10d In one embodiment, the fusion is achieved by performing a point cloud data processing method. Figures 10a-10c A schematic diagram of a point cloud fusion image obtained from point cloud data;

[0058] Figure 11 This is a schematic diagram of the point cloud data processing device in one embodiment;

[0059] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The point cloud data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, camera device 102 captures images of the patient in the treatment room to obtain first point cloud data, which is then transmitted to computer device 104 via a network. Computer device 104 processes the first point cloud data to obtain a point cloud fusion image, which is then sent to workstation 105. Based on the point cloud fusion image, workstation 105 uses a point cloud registration algorithm to position the patient and track motion during radiotherapy. Based on this, it generates control commands and sends them to radiotherapy device 106 to guide the pose adjustment of radiotherapy device 106. This enables real-time motion tracking of the radiation projected by radiotherapy device 106 on the target area of ​​the patient, with high real-time performance and accuracy, thus improving the treatment effect during radiotherapy. For example, when radiotherapy device 106 is a six-axis robot, the control commands can include poses for six degrees of freedom to guide the movement of each axis of radiotherapy device 106. Computer device 104 can be, but is not limited to, various personal computers, laptops, and tablets.

[0062] In one embodiment, such as Figure 2 As shown, a point cloud data processing method is provided, which can be applied to... Figure 1 Taking computer device 104 as an example, the following steps are included:

[0063] S202, acquire the first point cloud data captured by each camera device; there are at least two camera devices, positioned in different locations. The camera devices can be depth cameras, etc. Camera devices positioned in different locations can capture images of the patient from different angles, and the shooting ranges of at least two cameras overlap to avoid inaccurate target area localization due to occlusion during treatment. After acquiring point cloud data of the patient's body surface using multiple camera devices, point cloud fusion is required. This process must consider both the increased computation time due to data redundancy and the need to retain and restore as much detail as possible to improve the accuracy of patient motion tracking. Therefore, the point cloud data processing method also includes:

[0064] S204, determine the matching point pairs in the first point cloud data of every two camera devices. A matching point pair refers to a pair of point cloud data that match in two point cloud images. A matching pair of point cloud data can be point cloud data that are at the same coordinate position when the point cloud data of multiple camera devices are transformed to the same coordinate system. The point cloud matching process can be implemented using algorithms such as the point-to-surface ICP algorithm. Other point cloud matching algorithms can also be used. When using the point-to-surface ICP algorithm, its convergence speed is faster than that of the point-to-point ICP algorithm.

[0065] S206, the first matching point is removed from the matching point pair, and the second matching point is retained. The second matching point and the non-matching points in each of the first point cloud data are then fused to obtain a point cloud fused image. The first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point. The first included angle is the angle between the normal of the first point cloud data and the line containing the depth direction of the corresponding camera device. Considering that there may be duplicate points when fusing different point clouds, for each matching point pair, only the higher-quality point cloud data is retained.

[0066] Among them, such as Figure 3a , 3b As shown, the first included angle α is the angle between the normal Z2 of the first point cloud data D1 and the straight line Z1 containing the depth direction S of the corresponding camera device. The angle between the two lines ranges from 0 to 90°. The depth direction of the camera device refers to the main shooting direction S of the camera device. A single point cloud data and its adjacent N (N greater than 1) point cloud data can determine a surface DM. The straight line containing the normal vector F of the tangent plane Q at each point cloud on surface DM can be understood as the normal Z2 of these point clouds. According to Figure 3a and Figure 3b It can be seen that for the first point cloud data that has a convex or concave surface, the first point cloud data in such a way... Figure 3a On the convex surface shown, the angle between its normal vector and the depth direction is obtuse, and the larger this obtuse angle, the better the quality of the first point cloud data; the first point cloud data in... Figure 3b On the concave surface shown, the angle between its normal vector and the depth direction is acute, and the larger this angle, the worse the quality of the first point cloud data. However, in both cases, the angle between the line Z2 containing the normal vector and the line Z1 containing the depth direction is within the range of 0-90°. For example, based on the surface determined by the 16 point cloud data around each point cloud data, the normal vector F of the tangent plane Q of each point cloud data can be calculated, and the line containing the normal vector F is the normal line Z2. It should be understood that the point cloud data removal and deduplication operations performed in this application can be understood as removing and deduplicating the set of points that meet the removal and deduplication conditions.

[0067] Specifically, to avoid overlapping parts after point cloud fusion from multiple camera devices, the weaker point cloud data in a matched point pair can be removed before fusion. The smaller the first included angle in a matched point pair, the closer the point was to the depth direction of the camera device, and the higher its quality. Based on this, the second matched point with the smaller first included angle is retained, while the first matched point with the larger first included angle is removed. Since a larger first included angle corresponds to a smaller cosine value, the matched point can also be filtered by removing the matched point with the smaller cosine value of the first included angle. Point cloud data fusion is then performed based on the deduplication of the matched point pairs. The second matched point pair and the non-matched point pairs in the second point cloud data are merged to obtain a fused point cloud image from multiple camera devices. Non-matched points refer to those point cloud data that did not match when matching two point clouds. It should be noted that the point cloud data processing process described in this application is for the same frame of point clouds from each camera device.

[0068] In one embodiment, such as Figure 4 As shown, prior to step S204, which determines the matching point pairs in the first point cloud data of every two camera devices, the method further includes:

[0069] S402, the first target point cloud data is removed from the first point cloud data to obtain the second point cloud data for each camera device. The first target point cloud data is either point cloud data with a first included angle greater than a first preset angle, or point cloud data with a cosine value of the first included angle less than a first preset value. The second point cloud data can be understood as the first point cloud data after the above removal process. The amount of data in the second point cloud data is less than that in the first point cloud data. The first preset angle can be set based on the requirements of computation time and accuracy in the application scenario. For example, in a radiotherapy scenario, the first preset angle can be configured according to the real-time tracking of the target area and the allowable error distance during radiotherapy. The larger the first preset angle, the more details are retained and restored, and the smaller the error; conversely, the smaller the first preset angle, the more redundant information is eliminated in the process of obtaining the second point cloud data from the first point cloud data, resulting in less computation time required to obtain the point cloud fusion image and higher real-time performance. A balance point can be chosen between the two based on actual needs; for example, 15° can be selected as the first preset angle.

[0070] The removal of the first target point cloud data can be achieved by comparing the cosine of the first included angle with a first preset value. Specifically, two first point cloud data points along the depth direction of the camera device can be selected, and the normal vector in the depth direction can be determined based on the coordinates in the first point cloud data. Similarly, the normal vector of the tangent plane Q can be determined based on the coordinates in the first point cloud data at the tangent point. More conveniently, the unit normal vector of the tangent plane Q and the unit normal vector in the depth direction can be determined. At the same time, the magnitudes (diagonal lengths of the parallelogram) of the normal vector of the tangent plane Q and the normal vector in the depth direction can be calculated based on the coordinates of the first point cloud data. According to the magnitude formula a*b=|a|*|b|*cosθ, where a is the normal vector of the tangent plane Q, |a| is the magnitude of a, b is the normal vector in the depth direction, and |b| is the magnitude of b, a*b, |a|, and |b| can be calculated from the coordinates in the first point cloud data. Based on this, cosθ can be directly determined, and the absolute value of cosθ is the cosine of the first included angle. When both are unit normal vectors, |a| and |b| are both 1, resulting in fast calculation speed.

[0071] The cosine value corresponding to the first included angle ranges from 1 to 0. As the included angle increases, the cosine value decreases. A smaller cosine value indicates that the first point cloud data was captured at an angle further away from the depth direction by the camera device, resulting in poorer point cloud quality. Therefore, a first preset value can be set to filter the first point cloud data. Point cloud data with values ​​smaller than this preset value are discarded, yielding high-quality second point cloud data for subsequent point cloud data fusion. By reasonably setting the first preset value, the computational load during point cloud fusion can be reduced, while preserving as much detail as possible.

[0072] In practical implementation, if the absolute value of cosθ corresponding to the first point cloud data is less than a first preset value, it can be determined that the first point cloud data was obtained by shooting off-center from the depth direction and has poor quality. This data can then be used as the first target point cloud data and discarded. The implementation scheme of discarding the first point cloud data based on the absolute value of cosθ can be directly implemented using the coordinates in the point cloud data. This calculation is simple and beneficial for improving the real-time motion tracking of the target area in radiotherapy scenarios.

[0073] The first preset value can be determined based on the user's desired shooting angle. For example, if the first point cloud data captured within a 15° range based on the depth direction is considered to be of good quality, then cos15° can be determined as the first preset value.

[0074] The removal of the first target point cloud data can also be achieved by comparing the first included angle with a first preset included angle. In this process, the aforementioned modulus formula can be used to determine cosθ, and then an inverse cosine operation can be performed to obtain the angle θ between the normal vector of the tangent plane Q and the normal vector of the depth direction. The first included angle can then be determined based on θ. Specifically, when θ is less than or equal to 90°, the first included angle is equal to θ; when θ is greater than 90°, the first included angle is equal to 180° - θ.

[0075] S204, determine matching point pairs in the first point cloud data of every two camera devices, including:

[0076] S404, determine the matching point pairs in the second point cloud data of every two camera devices;

[0077] S206, Remove the first matching point from the matching point pair and retain the second matching point, then fuse the second matching point with the non-matching points in each of the first point cloud data to obtain a point cloud fused image, including:

[0078] S406, remove the first matching point from the matching point pair and keep the second matching point, and fuse the second matching point with the non-matching points in each second point cloud data to obtain a point cloud fused image.

[0079] Specifically, in the first point cloud data captured by each camera device, point cloud data with a first included angle (the angle between the normal of the first point cloud data and the line containing the depth direction of the corresponding camera device, with the first included angle ranging from 0 to 90°) greater than a first preset angle are discarded, or point cloud data with a cosine value of the first included angle less than the first preset value are discarded. These first point cloud data are point cloud data captured by the camera devices far from the depth direction within the shooting angle range, and their errors are large; that is, these points are shadow points and unstable points. After discarding them, higher-quality second point cloud data is obtained. The second point cloud data has less data volume than the first point cloud data, and the second point cloud data has higher stability. Therefore, the point cloud fusion image obtained by determining matching point pairs based on the second point cloud data of each camera device, and by fusing the point cloud data based on the filtered matching point pairs, is not only fast but also highly accurate. In radiotherapy scenarios, this can accelerate the radiotherapy workflow and throughput, and improve the effectiveness and safety of radiotherapy.

[0080] In one embodiment, before step S206 of removing the first matching point from the matching point pair and retaining the second matching point, as... Figure 5 As shown, it includes:

[0081] S502, remove the target matching point pairs from the matching point pairs to obtain the filtered matching point pairs. The second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair. The target matching point pair is the matching point pair whose second included angle is greater than the second preset angle, or whose absolute value of the cosine of the second included angle is less than the second preset value. The included angle between the two normal vectors ranges from 0 to 180°. The larger the included angle between the two normal vectors corresponding to the matching point pair, the greater the probability of mismatch between the two second point cloud data in the matching point pair. By removing mismatched point pairs and deleting useless point cloud data, the quality of point cloud data used for point cloud data fusion and the efficiency of point cloud data fusion can be effectively improved.

[0082] Step S206, which involves removing the first matching point from the matching point pair and retaining the second matching point, includes:

[0083] S504, remove the first matching point from the filtered matching point pairs and keep the second matching point.

[0084] Specifically, after finding matching points in every two point cloud images from the camera devices, matching point pairs are generated, and the normal vector of the cross-section of each point cloud data is calculated. Then, by calculating the second included angle between the two normal vectors of the matching point pair, mismatched point pairs are eliminated. On the one hand, this avoids accidentally deleting useful point cloud data during subsequent processing such as deleting overlapping points, thus preserving more details of the patient's body surface. On the other hand, it can improve the processing speed by reducing the amount of data used for point cloud data fusion.

[0085] In one embodiment, the normal vector of the tangent plane of each point cloud data can be calculated using 16 points surrounding each point cloud data.

[0086] The specific implementation of removing mismatched point pairs can be achieved using the magnitude of the cosine value. By calculating the cosine value of the angle between the two normal vectors of the matching point pair, matching point pairs whose absolute value of the cosine value is less than a second preset value are removed. For example, matching point pairs whose absolute value of the cosine value is less than cos15° are removed, thus achieving the function of removing mismatched points. This can improve the speed of point cloud data fusion operations. In addition, it can also avoid mistakenly deleting useful first or second point cloud data during subsequent processing such as deleting overlapping points based on matching point pairs (this removal operation can be performed after confirming the matching point pairs based on the first point cloud data, or after confirming the matching point pairs based on the second point cloud data), thus preserving more details of the patient's body surface. The calculation of the cosine value of the angle between the normal vectors can be referred to the description of the calculation of the cosine value of the first angle in the above embodiment. It can be calculated based on the coordinates and modulus formula in the point cloud data, and will not be elaborated here. If the number of mismatched points is too large, for example, if the proportion of mismatched point pairs exceeds a preset ratio, such as when 90% of the matched points are mismatched points, the point cloud data of the current frame can be discarded to avoid affecting the point cloud matching accuracy in subsequent motion tracking and other processing.

[0087] In one embodiment, before step S206 of removing the first matching point from the matching point pair and retaining the second matching point, as follows: Figure 6 As shown, it also includes:

[0088] S602, deduplication is performed between the first point cloud data from different camera devices to obtain third point cloud data. Optionally, deduplication can be performed after filtering matching point pairs to avoid accidentally deleting useful first or second point cloud data. The third point cloud data can be understood as the deduplicated first or second point cloud data, depending on whether deduplication is performed based on the first or second point cloud data in the specific embodiment; both are feasible embodiments of this application. Deduplication is performed between different point cloud data from different camera devices. Deduplication may not be performed between point cloud data from the same camera device to retain more details.

[0089] Step S206, which involves removing the first matching point from the matching point pairs and retaining the second matching point, and fusing the second matching point with the non-matching points in each of the first point cloud data to obtain the point cloud fused image, includes:

[0090] S604, remove the first matching point from the matching point pair and keep the second matching point, and fuse the second matching point with the non-matching points in each third point cloud data to obtain a point cloud fusion image.

[0091] Due to the deduplication process, the amount of data is further optimized and the processing speed is improved, thereby increasing the processing efficiency of point cloud fusion images. During radiotherapy, this is beneficial for improving the real-time and accurate monitoring of the physiological and non-physiological movements of the patient, realizing dynamic tracking and positioning of the target area, and facilitating targeted treatment.

[0092] In one embodiment, step S602, which involves deduplicating the first point cloud data from different camera devices to obtain the third point cloud data, is as follows: Figure 7 As shown, it includes:

[0093] S702, calculate the average distance between neighboring points of the first point cloud data for each camera device. To improve the deduplication effect, the point cloud data acquisition characteristics of a single camera device are fully considered. Based on the point cloud data of that camera device, the average distance between neighboring points of the point cloud data is calculated. This average distance can fully reflect the density of a single point cloud.

[0094] S704, determine the distance threshold based on the average distance of each camera device; there can be many methods for determination. For example, weighted processing can be used. Weighted processing is illustrated using two camera devices as an example, transforming the point cloud data of the first and second camera devices to the same coordinate system. The point cloud data volume of the first camera device is SJ1, and the average distance of the point cloud data of the first camera device in this coordinate system is JL1; the point cloud data volume collected by the second camera device is SJ2, and the average distance of the point cloud data of the second camera device is JL2. Then, the distance threshold JY = (SJ1*JL1 + SJ2*JL2) / (SJ1 + SJ2). Those skilled in the art should understand that for n camera devices, the distance threshold JY = (SJ1*JL1 + SJ2*JL2 + ... + SJ... n *JL n ) / (SJ1+SJ2+…+SJ n ), where n is greater than or equal to 2. SJ n JL represents the point cloud data volume of the nth camera device. n The average distance of the point cloud data of the nth camera device.

[0095] S706, the second target point cloud data is removed from the first point cloud data to obtain the third point cloud data for each camera device; the second target point cloud data consists of point cloud data from the first point cloud data of each camera device whose distance to the first point cloud data of other camera devices is less than a distance threshold. By deduplicating the first point cloud data where the distance between point cloud data of each pair of camera devices is less than the average distance, only one overlapping second point cloud is retained, optimizing the overlap of point clouds between different camera devices and reducing the amount of data in the third point cloud data, thus avoiding increased computational load that could affect the accuracy of positioning and motion tracking during radiotherapy. Compared to the traditional method of deleting overlapping points by directly specifying a distance threshold, the method provided in this application, which uses the average distance between neighboring points in each point cloud as the distance threshold for deduplication, fully considers the data density of a single point cloud to guide the deduplication of point clouds between different camera devices, resulting in better deduplication performance.

[0096] In one embodiment, the first point cloud data is point cloud data obtained by the camera device from the object to be treated. After obtaining the point cloud fused image, the method further includes:

[0097] The point cloud fusion image is sent to the workstation, which is connected to the radiotherapy equipment. The radiotherapy equipment is an instrument that delivers radiation to the surface of the patient's body for treatment. By sending the point cloud fusion image to the workstation, the workstation's computing power can track the target area markers on the patient's body surface in real time. The point cloud fusion image is kept consistent with the patient's current posture. Then, control commands are generated and sent to the radiotherapy equipment to guide its posture adjustment, ensuring that the radiation projected by the radiotherapy equipment is always projected onto the target area, guaranteeing precise radiotherapy during the treatment process.

[0098] In one embodiment, a first matching point can be removed from the matching point pairs while retaining a second matching point, and second target point cloud data can be removed from the second point cloud data of each camera device to obtain third point cloud data for each camera device. Then, the non-matching points in the second matching points and each of the third point cloud data can be fused to obtain a point cloud fused image. This further improves the processing speed of the point cloud fused image while preserving more details.

[0099] To better assist those skilled in the art in understanding the execution process of the point cloud data processing method provided in the embodiments of this application, an example of its application in a radiotherapy scenario is used for illustration, but this does not limit the actual scope of protection of this application.

[0100] Before and during radiotherapy, multiple cameras were used to acquire first point cloud data of the patient's body surface. The normal vectors of all first point cloud data were calculated and compared with the unit normal vector in the depth direction of the cameras to remove shadowed and unstable points. Based on an understanding of the camera hardware and the imaging principle of the depth camera, point cloud data generated along the depth direction of the cameras was found to be of the highest quality. Therefore, the normal vectors of all points in the first point cloud data were calculated, and the cosine of the angle between the normal vector of each point and the unit normal vector in the depth direction was also calculated. First point cloud data with an absolute cosine value greater than cos20° were retained. These point cloud data points were of high quality, and the remaining shadowed and unstable points were removed to obtain the second point cloud data.

[0101] At this point, if point cloud data fusion is performed, the following can be obtained: Figure 8a The point cloud fusion image shown indicates some overlapping points. Then, the second point cloud data acquired simultaneously by each camera device is registered using a point-to-surface ICP algorithm to search for matching points pairwise. To improve registration accuracy, after finding matching points, a normal vector elimination method is used to remove mismatched points. Specifically, after finding matching points, matching point pairs are generated, and the normal vector of the cross-section of each point cloud data is calculated using the 16 point cloud data surrounding each point cloud data. Then, by calculating the cosine of the angle between the normal vectors of the two matching points in the matching point pair, matching point pairs with an absolute value of cos15° are eliminated, thus removing mismatched points. This process avoids accidentally deleting useful point cloud data during subsequent deletion of overlapping points, preserving more details of the patient's body surface.

[0102] Furthermore, the average distance between neighboring points in each point cloud image is calculated. A distance threshold is then calculated based on this average distance, and overlapping points between different camera devices are removed using this threshold to reduce redundant information. Simultaneously, the angle between the normal vector of each matching point and the depth direction of the camera device is calculated. The second matching point with the smaller absolute value of the angle is retained, while the other first matching point is deleted. These two methods effectively remove overlapping points, resulting in the image shown below. Figure 8b The point cloud fusion image shown avoids overlapping parts after point cloud fusion from multiple cameras, and at the same time avoids increasing the amount of computation and affecting the accuracy of subsequent positioning and motion tracking.

[0103] The generated fused point cloud, or point cloud fusion image, can be applied to pre-radiotherapy positioning. During radiotherapy, the fused image maintains consistency with the patient's body posture, facilitating real-time and accurate monitoring of the patient's physiological and non-physiological movements. It can achieve real-time motion tracking of the patient's target area during treatment, improving the accuracy of radiotherapy and reducing the requirements for the patient's posture during radiotherapy, thus improving the user experience.

[0104] In one example, such as Figure 9aThe image shown is a point cloud image captured by a camera device, including 23,451 first point cloud data points, as shown below. Figure 9b Point cloud images acquired by another camera device, including 17,854 first point cloud data points, were obtained by performing the steps of the above method, as shown below. Figure 9c The point cloud fusion image shown includes 35,184 points, which preserves more details and avoids the problem of slow processing speed caused by point cloud overlap.

[0105] In another example, three camera devices were used to acquire the first point cloud data of the body surface of the subject to be treated. The acquired point cloud images are shown below. Figures 10a-10c As shown, by performing the above method steps, the following result is obtained: Figure 10d The point cloud fusion image shown demonstrates that point cloud fusion from multiple cameras avoids incomplete capture caused by occlusion. Furthermore, the fused point cloud image retains more details and has no obvious duplicate points, resulting in high validity of the point cloud data and high computational efficiency.

[0106] In one embodiment, during the process of obtaining the second point cloud data, noise reduction processing such as outlier filtering and Gaussian filtering can be performed on the first point cloud data line, and then the second point cloud data can be obtained based on the size relationship between the first included angle and the first preset included angle.

[0107] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0108] Based on the same inventive concept, this application also provides a point cloud data processing apparatus for implementing the point cloud data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more point cloud data processing apparatus embodiments provided below can be found in the limitations of the point cloud data processing method described above, and will not be repeated here.

[0109] In one embodiment, such as Figure 11As shown, a point cloud data processing device is provided, comprising: a first point cloud acquisition module, a matching point pair determination module, and a point cloud fusion image acquisition module, wherein:

[0110] The first point cloud acquisition module 1102 acquires the first point cloud data captured by each camera device; there are at least two camera devices, and they are set in different locations;

[0111] The matching point pair determination module 1104 determines the matching point pairs in the first point cloud data of every two camera devices;

[0112] The point cloud fusion image acquisition module 1106 removes the first matching point from the matching point pair and retains the second matching point, and fuses the second matching point with the non-matching points in each of the first point cloud data to obtain the point cloud fusion image;

[0113] Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0114] In one embodiment, it also includes:

[0115] The second point cloud data acquisition module is used to remove the first target point cloud data from the first point cloud data to obtain the second point cloud data of each camera device; wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value.

[0116] Matching point pair determination module 1104 includes:

[0117] A matching point pair determination unit is used to determine matching point pairs in the second point cloud data of every two camera devices;

[0118] Point cloud fusion image acquisition module 1106 includes:

[0119] The point cloud fusion image execution unit is used to remove the first matching point from the matching point pair and retain the second matching point, and fuse the second matching point and the non-matching points in each second point cloud data to obtain the point cloud fusion image.

[0120] In one embodiment, the device further includes:

[0121] The matching point pair filtering module is used to remove target matching point pairs from the matching point pairs to obtain filtered matching point pairs. The target matching point pairs are matching point pairs whose second included angle is greater than the second preset angle, or matching point pairs whose absolute value of the cosine of the second included angle is less than the second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair.

[0122] The point cloud fusion image acquisition module 1106 also includes:

[0123] The filtered matching point pair optimization unit is used to remove the first matching point and retain the second matching point from the filtered matching point pairs.

[0124] In one embodiment, the device further includes:

[0125] The deduplication module is used to deduplicatize the first point cloud data from different camera devices to obtain the third point cloud data.

[0126] The point cloud fusion image acquisition module 1106 also includes:

[0127] The point cloud fusion execution unit is used to remove the first matching point from the matching point pair and retain the second matching point, and fuse the second matching point with the non-matching points in each third point cloud data to obtain a point cloud fusion image.

[0128] In one embodiment, the deduplication module includes:

[0129] The average distance calculation unit is used to calculate the average distance between neighboring points of the first point cloud data of each camera device;

[0130] The deduplication execution unit is used to remove the second target point cloud data from the first point cloud data to obtain the third point cloud data; the second target point cloud data is the point cloud data in the first point cloud data of each camera device whose distance from the first point cloud data of other camera devices is less than a distance threshold.

[0131] In one embodiment, the first point cloud data is point cloud data obtained by the camera device from images of the object to be treated, and the device further includes:

[0132] The point cloud fusion image sending module is used to send point cloud fusion images to the workstation, which is connected to the radiotherapy equipment.

[0133] Each module in the aforementioned point cloud data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0134] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a point cloud data processing method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0135] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0137] S202, acquire the first point cloud data captured by each camera device; there are at least two camera devices, and they are set in different locations;

[0138] S204, determine the matching point pairs in the first point cloud data of every two camera devices;

[0139] S206, Remove the first matching point from the matching point pair and keep the second matching point, and fuse the second matching point with the non-matching points in each of the first point cloud data to obtain a point cloud fusion image;

[0140] Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0141] In one embodiment, when the processor executes the computer program, it also implements other steps of the point cloud data processing method described above and achieves corresponding beneficial effects, which will not be elaborated here.

[0142] Fourthly, a radiation therapy system is provided, comprising:

[0143] At least two camera devices are installed in different locations in the treatment room. The camera devices are used to photograph the patient in the treatment room to obtain the first point cloud data. The meaning of the first point cloud data, etc., can be found in the description in the above embodiments, and will not be repeated here.

[0144] The aforementioned computer equipment is communicatively connected to each camera device, and then executes the steps of the aforementioned point cloud data processing method to obtain a point cloud fusion image, which is then sent to the workstation. The workstation determines the target area of ​​the object to be treated based on the point cloud fusion image, and controls the radiotherapy equipment to perform targeted radiotherapy on the target area based on the determined target area.

[0145] Using multiple cameras can avoid the effects of obstruction during radiotherapy. Furthermore, by performing the aforementioned point cloud data processing on the first point cloud data, higher quality point clouds can be selected, which not only preserves and restores more details as much as possible, but also reduces the amount of data that needs to be processed during point cloud data fusion, thereby improving the accuracy and real-time performance of motion tracking of the treated object during radiotherapy.

[0146] Performing the steps of any of the above point cloud processing method embodiments by means of a computer device, including the radiotherapy system of the computer device, has corresponding beneficial effects, which will not be elaborated here.

[0147] Among them, computer equipment and radiation therapy equipment can be integrated into one device or separate devices, and their form is not limited.

[0148] In one embodiment, the computer equipment can be located in the control room, while the radiation therapy equipment and camera device are installed in the treatment room. Operators can configure and modify reference data such as a first preset angle and a second preset angle on the computer equipment from the control room.

[0149] In one embodiment, the computer equipment and the radiation therapy equipment can be wirelessly connected.

[0150] Of course, computer equipment and radiation therapy equipment can also communicate via wired connections. In the case of wired communication, the cables can be routed through the installation space above the ceiling to reduce the impact of wiring on the effective space of the treatment room and operating room.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements any step of the point cloud data processing method described above and achieves the corresponding beneficial effects. See the descriptions in the above embodiments for further details.

[0152] In one embodiment, a computer program product is provided, comprising a computer program that, when executed by a processor, implements any step of the point cloud data processing method described above and achieves the corresponding beneficial effects. See the descriptions in the above embodiments for further details.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to user body surface data, stored data, displayed data, etc. used for target area motion tracking) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A point cloud data processing method, characterized in that, Applied to a radiotherapy system, the method includes: Acquire the first point cloud data of the subject to be treated captured by each camera device; there are at least two camera devices, which are set in different locations in the treatment room; the first point cloud data is the surface point cloud data of the subject to be treated. Determine matching point pairs in the first point cloud data of every two of the aforementioned camera devices; The first matching point is removed from the matching point pair and the second matching point is retained. The second matching point and the non-matching points in each of the first point cloud data are then fused to obtain a point cloud fused image. Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or, the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal of the first point cloud data and the straight line of the depth direction of the corresponding camera device, and the first included angle is used to characterize the degree of deviation between the first point cloud data and the depth direction of the camera device.

2. The method according to claim 1, characterized in that, The method further includes, prior to the step of determining matching point pairs in the first point cloud data of every two of the camera devices: The first target point cloud data is removed from the first point cloud data to obtain the second point cloud data of each camera device; wherein the first target point cloud data is the point cloud data where the first included angle is greater than the first preset included angle, or the point cloud data where the cosine value of the first included angle is less than the first preset value. Determining the matching point pairs in the first point cloud data of every two of the camera devices includes: Determine matching point pairs in the second point cloud data of every two of the camera devices; The step of removing the first matching point from the matching point pair and retaining the second matching point, and fusing the second matching point with the non-matching points in each of the first point cloud data to obtain the point cloud fused image includes: The first matching point is removed from the matching point pair, the second matching point is retained, and the second matching point and the non-matching points in each of the second point cloud data are fused to obtain the point cloud fused image.

3. The method according to claim 2, characterized in that, Before the step of removing the first matching point from the matching point pair and retaining the second matching point, the procedure includes: Remove the target matching point pairs from the matching point pairs to obtain the filtered matching point pairs; the target matching point pairs are the matching point pairs whose second included angle is greater than the second preset angle, or the matching point pairs whose absolute value of the cosine of the second included angle is less than the second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair; The step of removing the first matching point from the matching point pair and retaining the second matching point includes: Remove the first matching point from the filtered matching point pairs and keep the second matching point.

4. The method according to claim 1, characterized in that, Before the step of removing the first matching point from the matching point pair and retaining the second matching point, the method further includes: Deduplication is performed on the first point cloud data from the different camera devices to obtain the third point cloud data; The step of removing the first matching point from the matching point pair and retaining the second matching point, and fusing the second matching point with the non-matching points in each of the first point cloud data to obtain the point cloud fused image includes: The first matching point is removed from the matching point pair, the second matching point is retained, and the second matching point and the non-matching points in each of the third point cloud data are fused to obtain the point cloud fused image.

5. The method according to claim 4, characterized in that, The process of deduplicating the first point cloud data from different camera devices to obtain the third point cloud data includes: Calculate the average distance between neighboring points of the first point cloud data for each of the camera devices; A distance threshold is determined based on the average distance of each of the aforementioned camera devices; The second target point cloud data is removed from the first point cloud data to obtain the third point cloud data; the second target point cloud data is the point cloud data in the first point cloud data of each of the camera devices whose distance from the first point cloud data of other camera devices is less than the distance threshold.

6. The method according to any one of claims 1-5, characterized in that, The first point cloud data is the point cloud data obtained by the camera device from the object to be treated. After obtaining the point cloud fused image, the method further includes: The point cloud fusion image is sent to a workstation connected to a radiotherapy device.

7. A point cloud data processing device, characterized in that, The device is used in a radiotherapy system and includes: The first point cloud acquisition module is used to acquire the first point cloud data of the object to be treated captured by each camera device; there are at least two camera devices, which are set in different locations in the treatment room; the first point cloud data is the point cloud data of the body surface of the object to be treated. A matching point pair determination module is used to determine matching point pairs in the first point cloud data of every two of the camera devices; The point cloud fusion image acquisition module is used to remove the first matching point from the matching point pair and retain the second matching point, and fuse the second matching point and the non-matching points in each of the first point cloud data to obtain a point cloud fusion image; Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or, the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point; the first included angle is the angle between the normal of the first point cloud data and the straight line of the depth direction of the corresponding camera device, and the first included angle is used to characterize the degree of deviation between the first point cloud data and the depth direction of the camera device.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A radiotherapy system, characterized in that, include: At least two camera devices are installed in different locations in the treatment room. The camera devices are used to photograph the object to be treated in the treatment room to obtain the first point cloud data. The computer device of claim 8 is communicatively connected to each of the aforementioned camera devices; A workstation, connected to the computer device, is used to determine the target area of ​​the object to be treated based on the point cloud fusion image; A radiation therapy device, connected to the workstation, is used to perform targeted radiation therapy on the target area under the control of the workstation.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images