Point cloud data processing method and device, computer device, and radiotherapy system

Abstract

The application relates to a point cloud processing method and device, computer equipment and a radiotherapy system, the method comprising the following steps: acquiring first point cloud data shot by each camera device; the camera devices are at least two, and are arranged at different positions; removing first target point cloud data from the first point cloud data to obtain second point cloud data of each camera device; and fusing the second point cloud data of the camera devices to obtain a point cloud fusion image; wherein the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; and the first included angle is an included angle between a normal line of the first point cloud data and a straight line in a depth direction of the corresponding camera device. By removing shadow points and unstable points, and then fusing the point cloud data, the fusion speed is fast, and the precision is high. When applied to a radiotherapy scene, the radiotherapy work flow and throughput can be accelerated, and the effectiveness and safety of radiotherapy can be improved.

Description

Technical Field

[0001] This application relates to the field of point cloud fusion processing technology, and in particular to a point cloud data processing method, apparatus, computer equipment, and radiotherapy system. Background Technology

[0002] In optically guided radiotherapy systems, a camera is needed to provide simulated positioning information and develop a radiotherapy plan. The calculated required radiation dose is then transmitted to the tumor center in the form of a radiation beam.

[0003] To ensure that the radiation dose reaches the tumor accurately, it is necessary to maintain consistency in the patient's body posture during the imaging and treatment processes, accurately position the patient before treatment, and monitor the patient's physiological and non-physiological movements in real time during treatment.

[0004] To improve system stability, multiple cameras are often used to collect point cloud information from the patient's body surface, avoiding the impact of occlusion during radiotherapy. The fusion of point cloud information from multiple cameras plays a crucial role; however, current point cloud fusion methods mostly involve simply overlaying the point clouds collected from multiple cameras or converting the point clouds into surfaces for overlay. This process is time-consuming and significantly impacts real-time tracking in radiotherapy systems. Summary of the Invention

[0005] Therefore, it is necessary to provide a point cloud data processing method, device, computer equipment, and radiotherapy system that can effectively improve the speed of point cloud fusion computing in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a point cloud data processing method, which includes:

[0007] Acquire the first point cloud data captured by each camera device; there are at least two camera devices, and they are set up in different locations;

[0008] Remove the first target point cloud data from the first point cloud data to obtain the second point cloud data for each camera device;

[0009] By fusing the second point cloud data from each camera device, a point cloud fused image is obtained;

[0010] Wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0011] In one embodiment, the second point cloud data from each camera device are fused to obtain a fused point cloud image, including:

[0012] Identify matching point pairs in the second point cloud data of every two camera devices;

[0013] A point cloud fusion image is obtained by fusing the second point cloud data of the two corresponding camera devices based on matching point pairs.

[0014] In one embodiment, a point cloud fused image is obtained by fusing second point cloud data from two corresponding camera devices based on matching point pairs, including:

[0015] Remove the target matching point pairs from the matching point pairs to obtain the filtered matching point pairs;

[0016] A point cloud fusion image is obtained by fusing the second point cloud data of the two corresponding camera devices based on the filtered matching point pairs.

[0017] Wherein, the target matching point pair is a matching point pair where the second included angle is greater than the second preset included angle, or a matching point pair where the absolute value of the cosine of the second included angle is less than the second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair.

[0018] In one embodiment, a point cloud fused image is obtained by fusing second point cloud data from two corresponding camera devices based on matching point pairs, including:

[0019] The third point cloud data is obtained by deduplicating the second point cloud data from different camera devices;

[0020] A point cloud fusion image is obtained by fusing the third point cloud data based on matching point pairs.

[0021] In one embodiment, deduplication is performed between the second point cloud data from different camera devices to obtain third point cloud data, including:

[0022] Calculate the average distance between neighboring points of the second point cloud data for each camera device;

[0023] A distance threshold is determined based on the average distance of each camera device;

[0024] The second target point cloud data is obtained by removing the second target point cloud data from the second point cloud data; the second target point cloud data is the point cloud data of each camera device whose distance from the second point cloud data of other camera devices is less than a distance threshold.

[0025] In one embodiment, a point cloud fused image is obtained by fusing second point cloud data from two corresponding camera devices based on matching point pairs, including:

[0026] Remove the first matching point from the matching point pair and keep the second matching point;

[0027] By fusing the second matching point and the non-matching points in each of the second point cloud data, a point cloud fusion image is obtained;

[0028] Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point.

[0029] In one embodiment, the first point cloud data is point cloud data obtained by the camera device from images of the object to be treated. After the step of fusing the second point cloud data from each camera device to obtain a fused point cloud image, the method further includes:

[0030] Send the point cloud fusion image to the workstation, which is connected to the radiotherapy equipment.

[0031] Secondly, a point cloud data processing device is provided, the device comprising:

[0032] The first point cloud data acquisition module is used to acquire the first point cloud data captured by each camera device; there are at least two camera devices, which are set in different locations;

[0033] The high-quality point cloud data filtering module is used to remove the first target point cloud data from the first point cloud data to obtain the second point cloud data of each camera device.

[0034] The point cloud data fusion module is used to fuse the second point cloud data from each camera device to obtain a fused point cloud image.

[0035] Wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0036] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0037] Fourthly, a radiation therapy system is provided, comprising:

[0038] At least two camera devices are installed in different locations in the treatment room. The camera devices are used to photograph the patients to be treated in the treatment room to obtain the first point cloud data.

[0039] The aforementioned computer equipment is connected to each camera device;

[0040] A workstation, connected to a computer device, is used to determine the target area of ​​the object to be treated based on point cloud fusion images;

[0041] A radiation therapy device, connected to a workstation, used to perform targeted radiation therapy on a target area under the control of the workstation.

[0042] The above-mentioned point cloud processing method, apparatus, computer equipment, and radiotherapy system have at least the following beneficial effects:

[0043] The process involves acquiring first point cloud data from each of at least two cameras positioned at different locations. Point cloud data captured by each camera is discarded if the first angle (the angle between the normal to the first point cloud data and the line representing the depth direction of the corresponding camera, ranging from 0-90°) is greater than a preset angle. These discarded points are taken by the cameras at angles far from the depth direction, resulting in larger errors; these points are considered shadows or unstable. After discarding these points, higher-quality second point cloud data is obtained. The second point cloud data is smaller and more stable than the first, allowing for faster and more accurate point cloud fusion. In radiotherapy, this can accelerate workflow and throughput, improving the effectiveness and safety of radiotherapy. Attached Figure Description

[0044] Figure 1 This is an application environment diagram of a point cloud data processing method in one embodiment;

[0045] Figure 2 This is a flowchart illustrating a point cloud data processing method in one embodiment;

[0046] Figure 3a This is a schematic diagram of the first angle between the normal and the line containing the depth direction in one embodiment;

[0047] Figure 3b This is a schematic diagram of the first angle between the normal and the line containing the depth direction in another embodiment;

[0048] Figure 4 This is a schematic diagram of the steps in one embodiment to fuse the second point cloud data of each camera device to obtain a point cloud fused image.

[0049] Figure 5 This is a schematic diagram of the steps in one embodiment to obtain a point cloud fused image by fusing the second point cloud data of two corresponding camera devices based on matching point pairs.

[0050] Figure 6 This is a flowchart illustrating the steps of fusing second point cloud data from two corresponding camera devices based on matching point pairs to obtain a point cloud fused image in another embodiment.

[0051] Figure 7 This is a flowchart illustrating the steps of deduplicating second point cloud data from different camera devices to obtain third point cloud data in one embodiment.

[0052] Figure 8 This is a flowchart illustrating the steps of fusing second point cloud data from two corresponding camera devices based on matching point pairs to obtain a point cloud fused image in another embodiment.

[0053] Figure 9 This is a flowchart illustrating a point cloud data processing method in one embodiment;

[0054] Figure 10a This is a schematic diagram illustrating the effect of fusing point cloud data before deduplication in one embodiment.

[0055] Figure 10b This is a schematic diagram of the point cloud fusion image obtained by performing point cloud data fusion after point cloud deduplication in one embodiment.

[0056] Figure 11a and 11b This is a schematic diagram of the first point cloud data collected by the first two cameras without processing, as shown in one embodiment.

[0057] Figure 11c In one embodiment, the fusion is achieved by performing a point cloud data processing method. Figure 11a and Figure 11b A schematic diagram of a point cloud fusion image obtained from point cloud data;

[0058] Figures 12a-12c These are schematic diagrams of the first point cloud data collected by three camera devices in one embodiment;

[0059] Figure 12d In one embodiment, the fusion is achieved by performing a point cloud data processing method. Figures 12a-12c A schematic diagram of a point cloud fusion image obtained from point cloud data;

[0060] Figure 13 This is a schematic diagram of the point cloud data processing device in one embodiment;

[0061] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The point cloud data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, camera device 102 captures images of the patient in the treatment room to obtain first point cloud data, which is then transmitted to computer device 104 via a network. Computer device 104 processes the first point cloud data to obtain a point cloud fusion image, which is then sent to workstation 105. Based on the point cloud fusion image, workstation 105 uses a point cloud registration algorithm to position the patient and track motion during radiotherapy. Based on this, it generates control commands and sends them to radiotherapy device 106 to guide the pose adjustment of radiotherapy device 106. This enables real-time motion tracking of the radiation projected by radiotherapy device 106 onto the target area of ​​the patient, with high real-time performance and accuracy, thus improving the treatment effect during radiotherapy. For example, when radiotherapy device 106 is a six-axis robot, the control commands can include poses for six degrees of freedom to guide the movement of each axis of radiotherapy device 106. Computer device 104 can be, but is not limited to, various personal computers, laptops, and tablets.

[0064] In one embodiment, such as Figure 2 As shown, a point cloud data processing method is provided, which can be applied to... Figure 1 Taking computer device 104 as an example, the following steps are included:

[0065] S202, acquire the first point cloud data captured by each camera device; there are at least two camera devices, positioned in different locations. The camera devices can be depth cameras, etc. Camera devices positioned in different locations can capture images of the patient from different angles, and the shooting ranges of at least two cameras overlap to avoid inaccurate target area localization due to occlusion during treatment. After acquiring point cloud data of the patient's body surface using multiple camera devices, point cloud fusion is required. This process must consider both the increased computation time due to data redundancy and the need to retain and restore as much detail as possible to improve the accuracy of patient motion tracking. Therefore, the point cloud data processing method also includes:

[0066] S204, the first target point cloud data is removed from the first point cloud data to obtain the second point cloud data for each camera device; wherein, the first target point cloud data is point cloud data where the first included angle is greater than the first preset angle, or point cloud data where the cosine value of the first included angle is less than the first preset value. Additionally, as... Figure 3a , 3bAs shown, the first included angle α is the angle between the normal Z2 of the first point cloud data D1 and the straight line Z1 containing the depth direction S of the corresponding camera device. The angle between the two lines ranges from 0 to 90°. The depth direction of the camera device refers to the main shooting direction S of the camera device. A single point cloud data and its adjacent N (N greater than 1) point cloud data can determine a surface DM. The straight line containing the normal vector F of the tangent plane Q at each point cloud on surface DM can be understood as the normal Z2 of these point clouds. According to Figure 3a and Figure 3b It can be seen that for the first point cloud data that has a convex or concave surface, the first point cloud data in such a way... Figure 3a On the convex surface shown, the angle between its normal vector and the depth direction is obtuse, and the larger this obtuse angle, the better the quality of the first point cloud data; the first point cloud data in... Figure 3b On the concave surface shown, the angle between its normal vector and the depth direction is acute, and the larger the angle, the worse the quality of the first point cloud data. However, in both cases, the angle between the line Z2 containing the normal vector and the line Z1 containing the depth direction is within the range of 0-90°. For example, based on the surface determined by the 16 point cloud data around each point cloud data, the normal vector F of the tangent plane Q of each point cloud data can be calculated, and the line containing the normal vector F is the normal line Z2. The second point cloud data can be understood as the first point cloud data after the above-mentioned culling process. For example, the first point cloud data with an angle greater than 15° is culled to obtain the second point cloud data. The amount of data in the second point cloud data is less than that in the first point cloud data. It should be understood that the point cloud data culling and deduplication operations in this application can be understood as culling and deduplicating the set of points that meet the culling and deduplication conditions.

[0067] The removal of the first target point cloud data can be achieved by comparing the cosine of the first included angle with a first preset value. Specifically, two first point cloud data points along the depth direction of the camera device can be selected, and the normal vector in the depth direction can be determined based on the coordinates in the first point cloud data. Similarly, the normal vector of the tangent plane Q can be determined based on the coordinates in the first point cloud data at the tangent point. More conveniently, the unit normal vector of the tangent plane Q can be determined. At the same time, the magnitudes (diagonal lengths of the parallelogram) of the normal vector of the tangent plane Q and the normal vector in the depth direction can be calculated based on the coordinates of the first point cloud data. According to the magnitude formula a*b=|a|*|b|*cosθ, where a is the normal vector of the tangent plane Q, |a| is the magnitude of a, b is the normal vector in the depth direction, and |b| is the magnitude of b, a*b, |a|, and |b| can be calculated from the coordinates in the first point cloud data. Based on this, cosθ can be directly determined, and the absolute value of cosθ is the cosine of the first included angle.

[0068] The cosine value corresponding to the first included angle ranges from 1 to 0. As the included angle increases, the cosine value decreases. A smaller cosine value indicates that the first point cloud data was captured at an angle further away from the depth direction by the camera device, resulting in poorer point cloud quality. Therefore, a first preset value can be set to filter the first point cloud data. Point cloud data with values ​​smaller than this preset value are discarded, yielding high-quality second point cloud data for subsequent point cloud data fusion. By reasonably setting the first preset value, the computational load during point cloud fusion can be reduced, while preserving as much detail as possible.

[0069] In practical implementation, if the absolute value of cosθ corresponding to the first point cloud data is less than a first preset value, it can be determined that the first point cloud data was obtained by shooting off-center from the depth direction and has poor quality. This data can then be used as the first target point cloud data and discarded. The implementation scheme of discarding the first point cloud data based on the absolute value of cosθ can be directly implemented using the coordinates in the point cloud data. This calculation is simple and beneficial for improving the real-time motion tracking of the target area in radiotherapy scenarios.

[0070] The first preset value can be determined based on the user's desired shooting angle. For example, if the first point cloud data captured within a 15° range based on the depth direction is considered to be of good quality, then cos15° can be determined as the first preset value.

[0071] The removal of the first target point cloud data can also be achieved by comparing the first included angle with a first preset included angle. In this process, the aforementioned modulus formula can be used to determine cosθ, and then an inverse cosine operation can be performed to obtain the angle θ between the normal vector of the tangent plane Q and the normal vector of the depth direction. The first included angle can then be determined based on θ. Specifically, when θ is less than or equal to 90°, the first included angle is equal to θ; when θ is greater than 90°, the first included angle is equal to 180° - θ.

[0072] S206, fuse the second point cloud data from each camera device to obtain a point cloud fused image.

[0073] The setting of the first preset angle can be based on the requirements of computation time and accuracy in the application scenario. For example, in a radiotherapy scenario, the first preset angle can be configured according to the real-time tracking of the target area and the allowable error distance during radiotherapy. The larger the first preset angle, the more details are preserved and restored, and the smaller the error; while the smaller the first preset angle, the more redundant information is eliminated in the process of obtaining the second point cloud data from the first point cloud data, and the less computation time is required to obtain the point cloud fusion image, and the higher the real-time performance. A balance point can be selected between the two based on actual needs, for example, 15° can be selected as the first preset angle.

[0074] Specifically, the process involves acquiring first point cloud data from each camera device; at least two cameras are used, positioned at different locations. Point cloud data captured by each camera device with a first angle (the angle between the normal of the first point cloud data and the line containing the depth direction of the corresponding camera device, ranging from 0-90°) greater than a first preset angle is discarded. These first point cloud data are captured by the cameras far from the depth direction within the shooting angle range, resulting in large errors; these points are shadow points and unstable points. After discarding them, higher-quality second point cloud data is obtained. The second point cloud data has less data volume than the first point cloud data and higher stability. Therefore, the point cloud fusion image obtained by fusing the second point cloud data from each camera device is fast and accurate. In radiotherapy scenarios, this can accelerate the radiotherapy workflow and throughput, improving the effectiveness and safety of radiotherapy.

[0075] In one embodiment, if the second point cloud data is obtained by calculating the cosine of the angle between the normal vector and the depth direction of each first point cloud data, since the angle between the vectors may be obtuse, point cloud data in which the absolute value of the cosine of the angle between the normal vector and the depth direction is less than a second preset value can be removed from the first point cloud data.

[0076] In one embodiment, such as Figure 4 As shown, the second point cloud data from each camera device are fused to obtain a point cloud fused image, including:

[0077] S402, determine the matching point pairs in the second point cloud data of every two camera devices. A matching point pair refers to a pair of point cloud data that match in two point cloud images. A matching pair of point cloud data can be point cloud data that are at the same coordinate position when the point cloud data of multiple camera devices are transformed to the same coordinate system. The point cloud matching process can be implemented using algorithms such as the point-to-surface ICP algorithm. Other point cloud matching algorithms can also be used. When using the point-to-surface ICP algorithm, its convergence speed is faster than that of the point-to-point ICP algorithm.

[0078] S404, based on the matching point pairs, fuse the second point cloud data of the two corresponding camera devices to obtain a point cloud fused image.

[0079] The second point cloud data, simultaneously acquired by each camera device, is paired and registered using methods such as point-to-surface ICP algorithm. Based on the paired point data, point cloud data fusion is performed to generate a fused overall point cloud, i.e., a point cloud fusion image. When the first point cloud data is the point cloud data of the body surface of the subject to be treated, this point cloud fusion image can be used for pre-radiotherapy positioning, facilitating the consistency between the images acquired during radiotherapy and the patient's body posture. This allows for real-time and accurate monitoring of the patient's physiological and non-physiological movements, enabling dynamic and precise tracking of the target area and accurate delivery of radiation.

[0080] The process of determining matching point pairs in the second point cloud data of every two camera devices can be illustrated using three camera devices as an example. First, the second point cloud data of two camera devices can be matched to obtain matching point pairs between these two point clouds. Then, the fused point cloud is used as the new point cloud and matched with the second point cloud data of the third camera device. Alternatively, the second point cloud data of the first and second camera devices can be matched first, and then the second point cloud data of the third camera device can be matched with the second point cloud data of the first camera device. For instance, if matching point pairs between the first and second camera devices are determined, and matching point pairs between the third and first camera devices are also determined, the matching point pairs between the first and third camera devices can be determined based on the common matching point pairs between the second, first, and third camera devices. This eliminates the need to separately calculate the matching point pairs between the first and third camera devices, allowing for point cloud fusion.

[0081] In one embodiment, a point cloud fused image is obtained by fusing the second point cloud data corresponding to the two camera devices based on matching point pairs, such as... Figure 5 As shown, it includes:

[0082] S502, remove the target matching point pair from the matching point pair to obtain the filtered matching point pair; the target matching point pair is the matching point pair whose second included angle is greater than the second preset included angle, or the matching point pair whose absolute value of the cosine of the second included angle is less than the second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair.

[0083] The second included angle between each matching point pair can be obtained; the angle between the two normal vectors ranges from 0 to 180°. The larger the included angle between the two normal vectors of a matching point pair, the greater the probability that the two second point cloud data points of the matching point pair are mismatched. Therefore, matching point pairs with a second included angle greater than a second preset angle can be removed from the matching point pairs, i.e., mismatched point pairs can be removed, resulting in filtered matching point pairs. By deleting useless point cloud data, the quality of point cloud data used for point cloud data fusion and the efficiency of point cloud data fusion can be effectively improved.

[0084] The specific implementation of removing mismatched point pairs can be achieved using the magnitude of the cosine value. By calculating the cosine value of the angle between the two normal vectors of the matching point pair (the cosine value of the second angle), matching point pairs whose absolute value of the cosine value is less than a second preset value are removed. For example, matching point pairs whose absolute value of the cosine value is less than cos15° are removed, thus achieving the function of removing mismatched points. This can improve the speed of point cloud data fusion operations. In addition, it can also avoid mistakenly deleting useful second point cloud data when performing other processing such as overlapping point deletion based on matching point pairs, thus preserving more details of the patient's body surface. The calculation of the cosine value of the angle between the normal vectors can be referred to the description of the calculation of the cosine value of the first angle in the above embodiment. It can be calculated based on the coordinates in the point cloud data and the modulus formula, and will not be elaborated here. If the number of mismatched points is too large, for example, if the proportion of mismatched point pairs exceeds a preset ratio, such as when 90% of the matching points are mismatched points, the point cloud data of the current frame can be discarded to avoid affecting the point cloud matching accuracy in subsequent motion tracking and other processing processes.

[0085] S504, based on the filtered matching point pairs, fuse the second point cloud data of the two corresponding camera devices to obtain a point cloud fused image.

[0086] Specifically, after finding matching points in every two point cloud images from the camera devices, matching point pairs are generated, and the normal vector of the cross-section of each point cloud data is calculated. Then, by calculating the angle or cosine of the angle between the two normal vectors of the matching point pair and comparing it with corresponding preset values, mismatched points are eliminated. On the one hand, this avoids accidentally deleting useful point cloud data during subsequent processing such as deleting overlapping points, thus preserving more details of the patient's body surface. On the other hand, it can improve the processing speed by reducing the amount of data used for point cloud data fusion.

[0087] In one embodiment, the normal vector of the tangent plane of each point cloud data can be calculated using 16 points surrounding each point cloud data.

[0088] In one embodiment, a point cloud fused image is obtained by fusing the second point cloud data corresponding to the two camera devices based on matching point pairs, such as... Figure 6 As shown, it includes:

[0089] S602, deduplication is performed between the second point cloud data from different camera devices to obtain third point cloud data. Optionally, deduplication can be performed after filtering matching point pairs to avoid accidentally deleting useful second point cloud data. The third point cloud data can be understood as the deduplicated second point cloud data. Deduplication is performed between point cloud data from different camera devices. Deduplication can be omitted between point cloud data from the same camera device to retain more details.

[0090] S604, based on the fusion of matching point pairs with third point cloud data, obtains a point cloud fusion image. Due to deduplication, the data volume is further optimized, and the processing speed is improved, thereby increasing the processing efficiency of the point cloud fusion image. During radiotherapy, this facilitates real-time and accurate monitoring of the physiological and non-physiological movements of the treated subject, enabling dynamic tracking and positioning of the target area, and facilitating targeted therapy.

[0091] In one embodiment, deduplication is performed between the second point cloud data from different camera devices to obtain the third point cloud data, such as... Figure 7 As shown, it includes:

[0092] S702, calculate the average distance between neighboring points of the second point cloud data for each camera device. To improve the deduplication effect, fully consider the point cloud data acquisition characteristics of a single camera device, calculate the average distance between neighboring points of all the second point cloud data of that camera device, and use this average distance as a metric to guide the deduplication process. Optionally, the second point cloud data here can be the second point cloud data after matching point filtering to reduce the amount of calculation required for deduplication.

[0093] S704, determine the distance threshold based on the average distance of each camera device. There are many ways to determine this threshold. For example, weighted processing can be used. Weighted processing is illustrated using two camera devices as an example, transforming the point cloud data of the first and second camera devices to the same coordinate system. The point cloud data volume of the first camera device is SJ1, and the average distance of the point cloud data of the first camera device in this coordinate system is JL1; the point cloud data volume collected by the second camera device is SJ2, and the average distance of the point cloud data of the second camera device is JL2. Then, the distance threshold JY = (SJ1*JL1 + SJ2*JL2) / (SJ1 + SJ2). Those skilled in the art should understand that for n camera devices, the distance threshold JY = (SJ1*JL1 + SJ2*JL2 + ... + SJ2*JL2) / (SJ1 + SJ2). n *JL n ) / (SJ1+SJ2+…+SJ n ), where n is greater than or equal to 2. SJ n JL represents the point cloud data volume of the nth camera device. n The average distance of the point cloud data of the nth camera device.

[0094] S706, the second target point cloud data is removed from the second point cloud data to obtain the third point cloud data; the second target point cloud data consists of point cloud data from each camera device whose distance to the second point cloud data of other camera devices is less than a distance threshold. By deduplicating the second point cloud data whose distance to each pair of camera devices is less than the average distance, only one overlapping second point cloud is retained, optimizing the overlap of point clouds between different camera devices. This also reduces the amount of data in the third point cloud data, avoiding increased computational load that could affect the accuracy of positioning and motion tracking during radiotherapy. Compared to the traditional method of deleting overlapping points by directly specifying a preset distance threshold, the method provided in this application, which uses the average distance between neighboring points in each point cloud as the distance threshold, fully considers the data density of a single point cloud to guide the deduplication of point clouds between different camera devices, resulting in better deduplication performance.

[0095] In one embodiment, a point cloud fused image is obtained by fusing the second point cloud data corresponding to the two camera devices based on matching point pairs, such as... Figure 8 As shown, it includes:

[0096] S802, remove the first matching point from the matching point pair and keep the second matching point. In addition to deduplication within the same point cloud, it also considers that there may be duplicate points when fusing different point clouds. Based on this, for a matching point pair, only the higher-quality point cloud data is retained.

[0097] S804, fuse the second matching point and the non-matching points in each of the second point cloud data to obtain a point cloud fusion image; wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or, the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point.

[0098] To avoid overlap after point cloud fusion from multiple cameras, the weaker point cloud data in a matched point pair can be removed before fusion. As described in the previous embodiments, the smaller the first included angle in a matched point pair, the closer it is to the depth direction of the camera, and the higher its quality. Therefore, the second matched point with the smaller first included angle is retained, while the first matched point with the larger first included angle is removed. Similarly, as described in the previous embodiments, the larger the first included angle, the smaller its corresponding cosine value. The cosine values ​​of the normal vector of the cross-section and the normal vector of the depth direction can be directly calculated based on the coordinates of the point cloud data. The matched point with the smaller absolute value of this cosine value is then removed as the first matched point. After deduplication between matched point pairs, point cloud data fusion is performed, fusing the second matched point pair with the non-matched point pairs in the second point cloud data to obtain a fused point cloud image from multiple cameras. Non-matched points refer to those point cloud data that were not matched when matching two point clouds.

[0099] In one embodiment, a first matching point can be removed from the matching point pairs while retaining a second matching point, and second target point cloud data can be removed from the second point cloud data of each camera device to obtain third point cloud data. The matching and non-matching points in the third point cloud data can then be fused to obtain a fused point cloud image. This further improves the processing speed of the fused point cloud image while preserving more detail.

[0100] In one embodiment, the first point cloud data is the point cloud data obtained by the camera device from the object to be treated. After the step of fusing the second point cloud data from each camera device to obtain a point cloud fused image, as follows... Figure 9 As shown, it also includes:

[0101] S902 sends a point cloud fusion image to the workstation, which is connected to the radiotherapy equipment. The radiotherapy equipment is an instrument that delivers radiation to the surface of the patient's body for treatment. By sending the point cloud fusion image to the workstation, the workstation's computing power can track the target area markers on the patient's body surface in real time. The point cloud fusion image is kept consistent with the patient's current posture. Then, control commands are generated and sent to the radiotherapy equipment to guide its posture adjustment, ensuring that the radiation projected by the radiotherapy equipment is always projected onto the target area, guaranteeing precise radiotherapy during the treatment process.

[0102] To better assist those skilled in the art in understanding the execution process of the point cloud data processing method provided in the embodiments of this application, an example of its application in a radiotherapy scenario is used for illustration, but this does not limit the actual scope of protection of this application.

[0103] Before and during radiotherapy, multiple camera devices were used to acquire first-point cloud data of the patient's body surface. The normal vectors of all first-point cloud data were calculated and compared with the unit normal vector along the depth direction of the camera devices to remove shadowed and unstable points. Based on an understanding of the camera device hardware and the imaging principle of the depth camera, point cloud data generated along the depth direction of the camera devices was found to be of the highest quality. Therefore, the normal vectors of all points in the first-point cloud data were calculated, and the cosine of the angle between the normal vector of each point and the unit normal vector along the depth direction was also calculated. First-point cloud data with an absolute cosine value greater than cos20° were retained. These point cloud data points were of high quality, thus removing the remaining shadowed and unstable points.

[0104] At this point, if point cloud data fusion is performed, the following can be obtained: Figure 10aThe point cloud fusion image shown indicates some overlapping points. Then, the second point cloud data acquired simultaneously by each camera device is registered using a point-to-surface ICP algorithm to search for matching points pairwise. To improve registration accuracy, after finding matching points, a normal vector elimination method is used to remove mismatched points. Specifically, after finding matching points, matching point pairs are generated, and the normal vector of the cross-section of each point cloud data is calculated using the 16 point cloud data surrounding each point cloud data. Then, by calculating the cosine of the angle between the normal vectors of the two matching points in the matching point pair, matching point pairs with an absolute value of cos15° are eliminated, thus removing mismatched points. This process avoids accidentally deleting useful point cloud data during subsequent deletion of overlapping points, preserving more details of the patient's body surface.

[0105] Furthermore, the average distance between neighboring points in each point cloud image is calculated. A distance threshold is then calculated based on this average distance, and overlapping points between different camera devices are removed using this threshold to reduce redundant information. Simultaneously, the angle between the normal vector of each matching point and the depth direction of the camera device is calculated. The second matching point with the smaller absolute value of the angle is retained, while the other first matching point is deleted. These two methods effectively remove overlapping points, resulting in the image shown below. Figure 10b The point cloud fusion image shown avoids overlapping parts after point cloud fusion from multiple cameras, and at the same time avoids increasing the amount of computation and affecting the accuracy of subsequent positioning and motion tracking.

[0106] The generated fused point cloud, or point cloud fusion image, can be applied to pre-radiotherapy positioning. During radiotherapy, the fused image maintains consistency with the patient's body posture, facilitating real-time and accurate monitoring of the patient's physiological and non-physiological movements. It can achieve real-time motion tracking of the patient's target area during treatment, improving the accuracy of radiotherapy and reducing the requirements for the patient's posture during radiotherapy, thus improving the user experience.

[0107] In one embodiment, during the process of obtaining the second point cloud data, noise reduction processing such as outlier filtering and Gaussian filtering can be performed on the first point cloud data line, and then the second point cloud data can be obtained based on the size relationship between the first included angle and the first preset included angle.

[0108] In one example, such as Figure 11a The image shown is a point cloud image captured by a camera device, including 23,451 first point cloud data points, as shown below. Figure 11b Point cloud images acquired by another camera device, including 17,854 first point cloud data points, were obtained by performing the steps of the above method, as shown below. Figure 11c The point cloud fusion image shown includes 35,184 points, which preserves more details and avoids the problem of slow processing speed caused by point cloud overlap.

[0109] In another example, three camera devices were used to acquire the first point cloud data of the body surface of the subject to be treated. The acquired point cloud images are shown below. Figures 12a-12c As shown, by performing the above method steps, the following result is obtained: Figure 12d The point cloud fusion image shown demonstrates that point cloud fusion from multiple cameras avoids incomplete capture caused by occlusion. Furthermore, the fused point cloud image retains more details and has no obvious duplicate points, resulting in high validity of the point cloud data and high computational efficiency.

[0110] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0111] Based on the same inventive concept, this application also provides a point cloud data processing apparatus for implementing the point cloud data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more point cloud data processing apparatus embodiments provided below can be found in the limitations of the point cloud data processing method described above, and will not be repeated here.

[0112] In one embodiment, such as Figure 13 As shown, a point cloud data processing device is provided, comprising:

[0113] The first point cloud data acquisition module 1302 is used to acquire the first point cloud data captured by each camera device; there are at least two camera devices, which are set in different locations;

[0114] The high-quality point cloud data filtering module 1304 is used to remove the first target point cloud data from the first point cloud data to obtain the second point cloud data of each camera device.

[0115] The point cloud data fusion module 1306 is used to fuse the second point cloud data from each camera device to obtain a point cloud fused image.

[0116] Wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0117] The definitions of the first point cloud data and the second point cloud data can be found in the descriptions of the above method embodiments, and will not be repeated here. Specifically, the first point cloud data acquired by the first point cloud data acquisition module 1302 is acquired by each camera device and sent to the high-quality point cloud data filtering module 1304. There are at least two camera devices, located in different positions. The high-quality point cloud data filtering module 1304 removes point cloud data from the first point cloud data where the first included angle is greater than a first preset angle or the cosine value of the first included angle is less than a first preset value, obtaining the second point cloud data for each camera device; wherein the first included angle is the angle between the normal of the first point cloud data and the line containing the depth direction of the corresponding camera device. Then, the point cloud data fusion module 1306 fuses the second point cloud data from each camera device to obtain a fused point cloud image.

[0118] In one embodiment, the point cloud data fusion module 1306 includes:

[0119] A matching point pair determination unit is used to determine matching point pairs in the second point cloud data of every two camera devices;

[0120] The second point cloud fusion unit is used to fuse the second point cloud data of the two corresponding camera devices based on the matching point pairs to obtain a point cloud fused image.

[0121] In one embodiment, the second point cloud fusion unit includes:

[0122] The matching point pair filtering unit is used to remove the target matching point pair from the matching point pairs to obtain the filtered matching point pairs;

[0123] The second point cloud fusion unit after filtering is used to fuse the second point cloud data of the two corresponding camera devices based on the filtered matching point pairs to obtain a point cloud fusion image; wherein, the target matching point pair is a matching point pair with a second included angle greater than a second preset included angle, or a matching point pair with a second included angle whose absolute value of the cosine of the second included angle is less than a second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair.

[0124] In one embodiment, the second point cloud fusion unit further includes:

[0125] The point cloud deduplication unit is used to deduplicatize the second point cloud data from different camera devices to obtain the third point cloud data.

[0126] The third point cloud fusion unit is used to fuse third point cloud data based on matching point pairs to obtain a point cloud fused image.

[0127] In one embodiment, the point cloud deduplication unit includes:

[0128] The distance calculation unit is used to calculate the average distance between neighboring points of the second point cloud data of each camera device;

[0129] The distance threshold determination unit is used to determine the distance threshold based on the average distance of each camera device;

[0130] The third point cloud data acquisition unit is used to remove the second target point cloud data from the second point cloud data to obtain the third point cloud data; the second target point cloud data is the point cloud data in the second point cloud data of each camera device whose distance from the second point cloud data of other camera devices is less than a distance threshold.

[0131] In one embodiment, the second point cloud fusion unit further includes:

[0132] The matching point pair deduplication unit is used to remove the first matching point from the matching point pair and retain the second matching point;

[0133] The fusion execution unit is used to fuse the second matching point and the non-matching points in each of the second point cloud data to obtain a point cloud fusion image;

[0134] Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point.

[0135] In one embodiment, the first point cloud data is point cloud data obtained by the camera device from photographing the object to be treated, and the device further includes:

[0136] A point cloud fusion image sending unit is used to send point cloud fusion images to a workstation, which is connected to a radiotherapy device.

[0137] Each module in the aforementioned point cloud data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0138] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 14As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a point cloud data processing method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0139] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0141] Acquire the first point cloud data captured by each camera device; there are at least two camera devices, and they are set up in different locations;

[0142] Remove the first target point cloud data from the first point cloud data to obtain the second point cloud data for each camera device;

[0143] By fusing the second point cloud data from each camera device, a point cloud fused image is obtained;

[0144] Wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device.

[0145] In one embodiment, when the processor executes the computer program, it also implements other method steps in the point cloud data processing method described above, and achieves the corresponding beneficial effects.

[0146] In one embodiment, this application provides a radiotherapy system, including: at least two camera devices, the aforementioned computer equipment, and radiotherapy equipment.

[0147] The camera devices are installed at different locations in the treatment room to photograph the patient and obtain first point cloud data. The meanings of "first point cloud data" and similar terms can be found in the descriptions of the above embodiments and will not be repeated here.

[0148] The computer equipment is connected to each camera device to acquire the first point cloud data captured by each camera device. Then, the steps of the point cloud data processing method described above are executed to obtain a point cloud fusion image, which is sent to the workstation. The workstation determines the target area of ​​the object to be treated based on the point cloud fusion image, and controls the radiotherapy equipment to perform targeted radiotherapy on the target area based on the determined target area.

[0149] Using multiple cameras can avoid the effects of obstruction during radiotherapy. Furthermore, by performing the aforementioned point cloud data processing on the first point cloud data, higher quality point clouds can be selected, which not only preserves and restores more details as much as possible, but also reduces the amount of data that needs to be processed during point cloud data fusion, thereby improving the accuracy and real-time performance of motion tracking of the treated object during radiotherapy.

[0150] Performing the steps of any of the above point cloud processing method embodiments by means of a computer device, including the radiotherapy system of the computer device, has corresponding beneficial effects, which will not be elaborated here.

[0151] Among them, computer equipment and radiation therapy equipment can be integrated into one device or separate devices, and their form is not restricted.

[0152] In one embodiment, the computer equipment can be located in the control room, and the radiation therapy equipment and camera device can be installed in the treatment room. Operators can configure and modify reference data such as the first preset angle and the second preset angle, as well as the first preset value and the second preset value, on the computer equipment from the control room.

[0153] In one embodiment, the computer equipment, workbench, and radiation therapy equipment can be wirelessly connected; of course, wired communication is also possible. With wired communication, the cabling can be routed through the ceiling-mounted installation space to minimize its impact on the usable space of the treatment room and operating room.

[0154] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements any step of the point cloud data processing method described above and achieves the corresponding beneficial effects. See the descriptions in the above embodiments for further details.

[0155] In one embodiment, a computer program product is provided, comprising a computer program that, when executed by a processor, implements any step of the point cloud data processing method described above and achieves the corresponding beneficial effects. See the descriptions in the above embodiments for further details.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A point cloud data processing method, characterized in that, Applied to a radiotherapy system, the method includes: Acquire the first point cloud data of the subject to be treated captured by each camera device; there are at least two camera devices, which are set in different locations in the treatment room; the first point cloud data is the surface point cloud data of the subject to be treated. The first target point cloud data is removed from the first point cloud data to obtain the second point cloud data for each of the camera devices; By fusing the second point cloud data from each of the aforementioned camera devices, a point cloud fused image is obtained; Wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device; the first preset included angle and the first preset value are both used to characterize the degree of deviation between the first point cloud data and the depth direction of the camera device.

2. The method according to claim 1, characterized in that, The process of fusing the second point cloud data from each of the camera devices to obtain a point cloud fused image includes: Determine matching point pairs in the second point cloud data of every two of the camera devices; The point cloud fused image is obtained by fusing the second point cloud data of the two corresponding camera devices based on the matching point pairs.

3. The method according to claim 2, characterized in that, The process of fusing the second point cloud data corresponding to the two camera devices based on the matching point pairs to obtain the point cloud fused image includes: Remove the target matching point pairs from the matching point pairs to obtain the filtered matching point pairs; Based on the filtered matching point pairs, the second point cloud data of the two corresponding camera devices are fused to obtain the point cloud fused image; Wherein, the target matching point pair is the matching point pair where the second included angle is greater than the second preset included angle, or the matching point pair where the absolute value of the cosine of the second included angle is less than the second preset value; the second included angle is the angle between the normal vectors corresponding to the two matching points in the matching point pair.

4. The method according to claim 2, characterized in that, The process of fusing the second point cloud data corresponding to the two camera devices based on the matching point pairs to obtain the point cloud fused image includes: The second point cloud data from the different camera devices are deduplicated to obtain the third point cloud data; The point cloud fusion image is obtained by fusing the third point cloud data based on the matching point pairs.

5. The method according to claim 4, characterized in that, The process of deduplicating the second point cloud data from different camera devices to obtain the third point cloud data includes: Calculate the average distance between neighboring points of the second point cloud data for each of the camera devices; A distance threshold is determined based on the average distance of each of the aforementioned camera devices; The second target point cloud data is removed from the second point cloud data to obtain the third point cloud data; the second target point cloud data is the point cloud data in the second point cloud data of each of the camera devices whose distance from the second point cloud data of other camera devices is less than the distance threshold.

6. The method according to claim 2, characterized in that, The process of fusing the second point cloud data corresponding to the two camera devices based on the matching point pairs to obtain the point cloud fused image includes: Remove the first matching point from the matching point pair and retain the second matching point; The point cloud fusion image is obtained by fusing the second matching point and the non-matching points in each of the second point cloud data. Wherein, the first included angle corresponding to the first matching point is greater than the first included angle corresponding to the second matching point, or the cosine value of the first included angle corresponding to the first matching point is less than the cosine value of the first included angle corresponding to the second matching point.

7. The method according to any one of claims 1-6, characterized in that, The first point cloud data is the point cloud data obtained by the camera device from the object to be treated. After the step of fusing the second point cloud data from each of the camera devices to obtain a point cloud fused image, the method further includes: The point cloud fusion image is sent to a workstation connected to a radiotherapy device.

8. A point cloud data processing device, characterized in that, The device is used in a radiotherapy system and includes: The first point cloud data acquisition module is used to acquire the first point cloud data of the object to be treated captured by each camera device; there are at least two camera devices, which are set in different locations in the treatment room; the first point cloud data is the point cloud data of the body surface of the object to be treated. A high-quality point cloud data filtering module is used to remove the first target point cloud data from the first point cloud data to obtain the second point cloud data of each of the camera devices; The point cloud data fusion module is used to fuse the second point cloud data of each of the camera devices to obtain a point cloud fused image. Wherein, the first target point cloud data is point cloud data with a first included angle greater than a first preset included angle, or point cloud data with a cosine value of the first included angle less than a first preset value; the first included angle is the angle between the normal of the first point cloud data and the straight line containing the depth direction of the corresponding camera device; the first preset included angle and the first preset value are both used to characterize the degree of deviation between the first point cloud data and the depth direction of the camera device.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A radiotherapy system, characterized in that, include: At least two camera devices are installed in different locations in the treatment room. The camera devices are used to photograph the object to be treated in the treatment room to obtain the first point cloud data. The computer device as described in claim 9 is connected to each of the aforementioned camera devices; A workstation, connected to the computer device, is used to determine the target area of ​​the object to be treated based on the point cloud fusion image; A radiation therapy device, connected to the workstation, is used to perform targeted radiation therapy on the target area under the control of the workstation.

Images