A method for ROI region association and labeling
By using pose correlation calculation and transformation matrix processing, the problem of the inability to monitor the diseased area in radiotherapy equipment in real time was solved, realizing the automatic identification and tracking monitoring of the ROI area and improving the accuracy of radiotherapy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KLARITY MEDICAL & EQUIP GZ
- Filing Date
- 2022-04-25
- Publication Date
- 2026-04-17
AI Technical Summary
Existing radiotherapy equipment cannot monitor minute displacements in the affected area of a patient in real time during tumor radiotherapy, making it impossible to achieve real-time tracking, identification, and treatment.
By acquiring the real-time surface contour point set and reference surface contour point set of the detected object, pose correlation calculation is performed to generate the correlation transformation matrix, thereby realizing automatic annotation and tracking monitoring of the ROI region.
It enables automatic identification and labeling of ROI regions on a real-time surface contour point set during radiotherapy, allowing for real-time tracking, monitoring, and treatment.
Smart Images
Figure CN116802686B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of imaging technology, and more specifically, to a method for ROI region association and annotation. Background Technology
[0002] In recent years, various radiotherapy equipment has been widely used in the diagnosis and treatment of a variety of complex and variable diseases. During radiotherapy, accurate monitoring of the patient's target area has a significant impact on the precision treatment of tumors.
[0003] For example, in current tumor radiotherapy methods, doctors need to manually delineate and mark the tumor area or other key regions on the surface contour point set obtained from the scan during each treatment session. The radiotherapy equipment then delivers radiation to the marked tumor area or other key regions. However, existing methods can only delineate a single frame of the scan. In reality, the real-time surface contour point set is constantly updated as the camera scans. Therefore, during radiotherapy, the body may experience slight displacement due to breathing and other factors, which current methods do not account for. Furthermore, current methods cannot achieve real-time tracking and identification of the tumor area or other key regions, resulting in a failure to monitor the affected area in real time.
[0004] Therefore, how to quickly and automatically determine the diseased areas on the body surface is a technical problem that urgently needs to be solved. Summary of the Invention
[0005] In view of this, this application provides a method for ROI region association and annotation, which can realize the association recognition and automatic annotation of ROI regions on a real-time surface contour point set.
[0006] To achieve the above objectives, the following solution is proposed:
[0007] A method for ROI region association and annotation includes:
[0008] Obtain the real-time surface contour point set of the currently scanned detection object;
[0009] Obtain a reference surface contour point set of the object to be tested, wherein the reference surface contour point set is the surface contour point set obtained by scanning the object to be tested with a positioning standard;
[0010] Pose correlation calculation is performed on the reference surface contour point set and the real-time surface contour point set to obtain the correlation transformation matrix;
[0011] The real-time surface contour point set is transformed according to the correlation transformation matrix to generate a transformed contour point set;
[0012] A first ROI region is determined from the reference surface contour point set, and the first ROI region is transferred and transformed onto the transformed contour point set to generate a second ROI region. The first ROI region is the ROI region marked on the reference surface contour point set.
[0013] Based on the associated transformation matrix, the transformed contour point set containing the second ROI region is inversely transformed to generate a target contour point set, which contains a third ROI region obtained by inversely transforming the second ROI region.
[0014] Preferably, determining a first ROI region from the reference surface contour point set and transforming the first ROI region onto the transformed contour point set to generate a second ROI region includes:
[0015] Obtain the coordinate positions of feature points in the first ROI region of the reference surface contour point set, where the first ROI region is the ROI region marked on the reference surface contour point set;
[0016] Identify feature points whose coordinate positions in the transformed contour point set are the same as those of feature points in the first ROI region;
[0017] The second ROI region is determined based on the feature points whose coordinate positions are the same as those of the feature points in the first ROI region.
[0018] Preferably, determining the first ROI region of the reference surface contour point set includes:
[0019] In response to human operation, the point set parameters of the outlined graphic area and the reference surface contour point set are obtained;
[0020] Determine the view space transformation matrix corresponding to the point set parameters;
[0021] According to the view space transformation matrix, a view space transformation is performed between the graphic region and the reference surface contour point set, and a first ROI region corresponding to the graphic region is generated on the reference surface contour point set based on the transformation result.
[0022] Preferably, pose correlation calculation is performed on the reference surface contour point set and the real-time surface contour point set to obtain the correlation transformation matrix, including:
[0023] A first pose correlation calculation is performed on the reference surface contour point set and the real-time surface contour point set to generate a correlation transformation matrix;
[0024] Based on the previous pose association calculation, a second pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set, and the association transformation matrix is updated.
[0025] If the current pose association calculation result is detected to be outside the allowable difference range, then return to the process of performing a second pose association calculation on the reference surface contour point set and the real-time surface contour point set based on the previous pose association calculation, until the current pose association calculation result is within the allowable difference range.
[0026] Preferably, a first pose correlation calculation is performed on the reference surface contour point set and the real-time surface contour point set to generate a correlation transformation matrix, including:
[0027] The reference surface contour point set and the real-time surface contour point set are respectively filtered and denoised;
[0028] Determine each feature point that constitutes the reference surface contour point set and the normal vector of each feature point that constitutes the real-time surface contour point set;
[0029] Based on the normal vector, calculate and generate the feature point geometric description set of the reference surface contour point set and the real-time surface contour point set;
[0030] Based on the geometric description set of the feature points, determine the correlation transformation matrix of the real-time surface contour point set to be calculated in association with the pose of the reference surface contour point set.
[0031] Preferably, according to the view space transformation matrix, a view space transformation is performed between the graphic region and the reference surface contour point set, and a first ROI region corresponding to the graphic region is generated on the reference surface contour point set based on the transformation result, including:
[0032] Determine the 2D coordinates of each feature point within the graphic region;
[0033] Based on the view space transformation matrix, determine the 2D coordinates corresponding to each feature point of the reference surface contour point set;
[0034] The feature points whose 2D coordinates are the same as those of the feature points in the graphic region are identified as target feature points.
[0035] The region where the target feature point is located on the reference surface contour point set is determined as the first ROI region.
[0036] Preferably, according to the view space transformation matrix, a view space transformation is performed between the graphic region and the reference surface contour point set, and a first ROI region corresponding to the graphic region is generated on the reference surface contour point set based on the transformation result, including:
[0037] Determine the 2D coordinates of each feature point within the graphic region;
[0038] Based on the inverse of the view space transformation matrix, determine the 3D coordinates corresponding to the 2D coordinates of each feature point within the graphic region;
[0039] The feature points whose coordinates on the reference surface contour point set are the same as the 3D coordinates of each feature point in the graphic region are determined as target feature points;
[0040] The region where the target feature point is located on the reference surface contour point set is determined as the first ROI region.
[0041] Preferably, before transforming the real-time surface contour point set according to the correlation transformation matrix to generate the transformed contour point set, the method further includes:
[0042] Check whether the pose correlation calculation results are within the allowable difference range;
[0043] If so, then the process of transforming the real-time surface contour point set according to the associated transformation matrix to generate a transformed contour point set is executed.
[0044] If not, a prompt message will be displayed to remind the object being tested to adjust its position.
[0045] Preferably, after transforming the real-time surface contour point set according to the correlation transformation matrix to generate the transformed contour point set, the method further includes:
[0046] A unique correspondence is determined between each feature point constituting the transformed contour point set and each feature point constituting the reference surface contour point set, and the same index number is assigned to each pair of corresponding feature points.
[0047] As can be seen from the above technical solution, the ROI region association and annotation method provided in this application obtains a real-time surface contour point set of the currently scanned detection object and a reference surface contour point set of the detection object. The reference surface contour point set is the surface contour point set obtained by scanning the detection object with a positioning standard. Pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set to obtain an association transformation matrix. The real-time surface contour point set is then transformed according to the association transformation matrix to generate a transformed contour point set. A first ROI region is determined from the reference surface contour point set, and the first ROI region is migrated and transformed onto the transformed contour point set to generate a second ROI region. The first ROI region is the ROI region annotated on the reference surface contour point set. Finally, according to the association transformation matrix, an inverse transformation is performed on the transformed contour point set containing the second ROI region to generate a target contour point set. The target contour point set contains a third ROI region obtained by the inverse transformation of the second ROI region.
[0048] This application uses pose correlation calculation to detect a reference surface contour point set and a real-time surface contour point set. It then transforms the first Region of Interest (ROI) marked on the reference surface contour point set to the transformed contour point set obtained after pose correlation calculation, resulting in a transformed contour point set containing a second ROI. An inverse transformation is then performed on this transformed contour point set to obtain a target contour point set containing a third ROI. Since the transformation process uses an associated transformation matrix, and the inverse transformation process uses the inverse of that matrix, the target contour point set is identical to the real-time surface contour point set, except for the marked third ROI. This application enables automatic ROI identification and marking on the same areas of the scanned real-time surface contour point set based on the reference surface contour point set containing the first ROI, thereby allowing for tracking, monitoring, and treatment of ROI areas on the real-time surface contour point set during radiotherapy. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0050] Figure 1 This is a flowchart of a ROI region association and annotation method disclosed in this application;
[0051] Figure 2 This is a schematic diagram of a reference surface profile point set and a target profile point set disclosed in an embodiment of this application;
[0052] Figure 3 This is a schematic diagram illustrating the delineation of a Region of Interest (ROI) as disclosed in an embodiment of this application. Detailed Implementation
[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0054] The following section introduces the proposed solution; see [link / reference] Figure 1 , Figure 1 This is a flowchart of a ROI region association and annotation method disclosed in an embodiment of this application.
[0055] like Figure 1 As shown, the method may include:
[0056] Step S1: Obtain the real-time surface contour point set of the currently scanned detection object.
[0057] Specifically, the real-time surface contour point set consists of several feature points. After the object to be detected enters the scanning device, it can be scanned by various depth scanning devices, and the real-time surface contour point set can be constructed using modeling software such as 3ds Max and Maya. This application does not limit the specific scanning equipment; any device capable of scanning the object to be detected and generating the corresponding real-time surface contour point set is considered a scanning device usable under this application.
[0058] Step S2: Obtain the reference surface contour point set of the object to be detected.
[0059] Specifically, the reference surface contour point set is the surface contour point set obtained by scanning the positioning standard of the test object, wherein the reference surface contour point set is composed of several feature points. At this time, the posture, position, etc. of the test object meet the requirements of detection and scanning. For example, the reference surface contour point set can be the surface contour point set obtained by scanning the positioning standard of the human body.
[0060] In practical applications, after the instrument scans the test object of the positioning standard, it can obtain and store the reference surface contour point set of the test object. In the subsequent ROI region determination process, the corresponding reference surface contour point set of the test object can be retrieved directly according to the identity information, ID information, code, etc. of the test object, and the subsequent ROI region determination can be carried out.
[0061] Step S3: Perform pose correlation calculation on the reference surface contour point set and the real-time surface contour point set to obtain the correlation transformation matrix.
[0062] Specifically, a global pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set. This pose association calculation can be achieved through a combination of a first pose association calculation and a second pose association calculation. After the pose association calculation is completed, an association transformation matrix is generated. This association transformation matrix can be used to transform the real-time surface contour point set.
[0063] Step S4: Transform the real-time surface contour point set according to the associated transformation matrix to generate a transformed contour point set.
[0064] Specifically, the real-time surface contour point set is subjected to matrix operations based on the correlation transformation matrix obtained by the pose correlation calculation. After transformations such as rotation, displacement, and stretching, the transformed contour point set is obtained. The transformed contour point set is basically consistent with the reference surface contour point set in terms of position, orientation, etc. It can be considered that the transformed contour point set obtained by the transformation is basically close to the reference surface contour point set.
[0065] Step S5: Determine the first ROI region of the reference surface contour point set, and migrate and transform the first ROI region onto the transformed contour point set to generate the second ROI region.
[0066] Specifically, the first ROI region is the ROI region marked on the reference surface contour point set. The first ROI region can be a pre-marked ROI region on the reference surface contour point set, or it can be an ROI region drawn and marked in real-time. After determining the first ROI region on the reference surface contour point set, since the transformed contour point set is essentially overlapping with the reference surface contour point set, the first ROI region can be migrated and transformed onto the transformed contour point set to generate the second ROI region. The method of migrating and transforming the first ROI region to generate the second ROI region can include mapping the first ROI region onto the transformed contour point set to determine the second ROI region on the transformed contour point set. The position and size of the second ROI region on the transformed contour point set are the same as the position and size of the first ROI region on the reference surface contour point set.
[0067] Step S6: Based on the associated transformation matrix, perform an inverse transformation on the transformed contour point set containing the second ROI region to generate the target contour point set.
[0068] Specifically, the target contour point set includes a third ROI region obtained by inverse transformation of the second ROI region. The inverse matrix of the correlation transformation matrix is calculated, and the transformed contour point set containing the second ROI region is inversely transformed using the inverse matrix of the correlation transformation matrix. This restores the real-time surface contour point set that has undergone rotation, displacement, and stretching in step S4 to its original shape before rotation, displacement, and stretching, thereby generating a target contour point set containing the third ROI region obtained by inverse transformation of the second ROI region.
[0069] In practical applications, after obtaining the reference surface contour point set and the real-time surface contour point set labeled with the first ROI region, the target contour point set containing the corresponding third ROI region can be obtained according to this method. For example... Figure 2 As shown, the position and size of the second ROI region on the transformed contour point set are the same as the position and size of the first ROI region on the reference surface contour point set. The third ROI region is obtained by transforming the second ROI region. The region represented by the third ROI region on the target surface contour point set is the same region represented by the first ROI region on the reference surface contour point set, which is the diseased area that needs to be radiotherapy.
[0070] As can be seen from the above technical solution, the ROI region association and annotation method provided in this application obtains a real-time surface contour point set of the currently scanned detection object and a reference surface contour point set of the detection object. The reference surface contour point set is the surface contour point set obtained by scanning the detection object with a positioning standard. Pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set to obtain an association transformation matrix. The real-time surface contour point set is then transformed according to the association transformation matrix to generate a transformed contour point set. A first ROI region is determined from the reference surface contour point set, and the first ROI region is migrated and transformed onto the transformed contour point set to generate a second ROI region. The first ROI region is the ROI region annotated on the reference surface contour point set. Finally, according to the association transformation matrix, an inverse transformation is performed on the transformed contour point set containing the second ROI region to generate a target contour point set. The target contour point set contains a third ROI region obtained by the inverse transformation of the second ROI region.
[0071] This application uses pose correlation calculation to detect a reference surface contour point set and a real-time surface contour point set. It then transforms the first Region of Interest (ROI) marked on the reference surface contour point set to the transformed contour point set obtained after pose correlation calculation, resulting in a transformed contour point set containing a second ROI. An inverse transformation is then performed on this transformed contour point set to obtain a target contour point set containing a third ROI. Since the transformation process uses an associated transformation matrix, and the inverse transformation process uses the inverse of that matrix, the target contour point set is identical to the real-time surface contour point set, except for the marked third ROI. This application enables automatic ROI identification and marking on the same areas of the scanned real-time surface contour point set based on the reference surface contour point set containing the first ROI, thereby allowing for tracking, monitoring, and treatment of ROI areas on the real-time surface contour point set during radiotherapy.
[0072] Optionally, after step S4, transforming the real-time surface contour point set according to the associated transformation matrix to generate the transformed contour point set, the process may further include:
[0073] Step S7: Determine the unique correspondence between each feature point constituting the transformed contour point set and each feature point constituting the reference surface contour point set, and set the same index number for each group of corresponding feature points.
[0074] Specifically, a unique correspondence is established between each feature point constituting the transformed contour point set and each feature point constituting the reference surface contour point set. For each feature point constituting the transformed contour point set, there is a unique corresponding feature point constituting the reference surface contour point set. To avoid repetitive pose association calculations, each pair of corresponding feature points can be assigned the same index number, ensuring that no two index numbers are repeated. Based on this index number, the corresponding feature points in that pair can be directly located.
[0075] In practical applications, if it is necessary to add more ROI regions, it is only necessary to determine each feature point in the newly added ROI region on the reference surface contour point set. Then, each feature point on the transformed contour point set with the same index number can be found, and the corresponding second ROI region can be quickly determined without repeating the previous pose association calculation and transformation process.
[0076] Optionally, considering that the human body may shift in practical applications, in order to avoid damage caused by ineffective irradiation due to human body shift, it is also possible to further determine whether the placement of the object to be detected needs to be adjusted based on the correlation transformation matrix.
[0077] Before step S4, which transforms the real-time surface contour point set according to the associated transformation matrix to generate the transformed contour point set, the following may also be included:
[0078] Step S8: Check whether the pose association calculation result is within the allowable difference range;
[0079] If so, then execute step S4, which involves transforming the real-time surface contour point set according to the associated transformation matrix to generate a transformed contour point set.
[0080] If not, a prompt message will be displayed to remind the object being tested to adjust its position.
[0081] Specifically, the system determines whether to administer radiation therapy based on the detected situation. If the pose correlation calculation result is within the allowable difference range, the subsequent ROI region determination and labeling work continues. If, during real-time monitoring, the position of the detected object changes significantly and the pose correlation calculation result is outside the allowable difference range, a prompt message is displayed to remind the detected object to adjust its position, and radiation is not administered. If the position of the detected object is restored to the set threshold range in the next monitoring, i.e., the pose correlation calculation result is within the allowable difference range, the system will automatically resume the ROI region determination and labeling work, as well as administer radiation therapy to the ROI region.
[0082] In practical applications, the first ROI region on the reference surface profile point set can be a pre-annotated ROI region or a ROI region annotated at the current moment. Considering these two different approaches, two different implementation methods are provided.
[0083] In some embodiments of this application, the process of step S5, determining the first ROI region of the reference surface contour point set, and migrating and transforming the first ROI region onto the transformed contour point set to generate the second ROI region, is described, and may specifically include:
[0084] The second type is where the first ROI region is already marked on the reference unit model.
[0085] Step S51: Obtain the coordinate positions of feature points in the first ROI region of the reference surface contour point set, where the first ROI region is the ROI region marked on the reference surface contour point set.
[0086] Specifically, the reference surface contour point set contains a first ROI region that has already been labeled. At this time, the coordinate positions of each feature point in the first ROI region on the reference surface contour point set are obtained.
[0087] Step S52: Determine the feature points whose coordinate positions in the transformed contour point set are the same as the coordinate positions of the feature points in the first ROI region.
[0088] Specifically, each feature point in the transformed contour point set that closely coincides with the reference surface contour point set is identified as having the same coordinate position as each feature point in the transformed contour point set. Alternatively, a mapping method can be used to determine the feature points in the first ROI region that are mapped onto the transformed contour point set and have the same coordinate position as feature points in the first ROI region.
[0089] Step S53: Determine the second ROI region based on the feature points whose coordinate positions are the same as those of the feature points in the first ROI region.
[0090] Specifically, after determining the feature points whose coordinate positions are the same as those of the feature points in the first ROI region, the area enclosed by all the feature points is the second ROI region.
[0091] The second method involves labeling the first ROI region on the reference unit model in real time.
[0092] The process of determining the first ROI region of the reference surface contour point set in step S5 is described below, which may specifically include:
[0093] Step S54: In response to human operation, obtain the point set parameters of the outlined graphic area and the reference surface contour point set.
[0094] Specifically, such as Figure 3 As shown, the first ROI region can be labeled on the reference surface contour point set at the current moment. For example, the user can determine the ROI region to be generated by rotating the reference surface contour point set and drawing with the mouse. The closed-shape graphic region formed by manual drawing is obtained, and the point set parameters of the graphic region and the reference surface contour point set are determined.
[0095] Step S55: Determine the view space transformation matrix corresponding to the point set parameters.
[0096] Specifically, the point set parameters can be used to obtain a view space transformation matrix that matches the model. The view space transformation matrix is used for view space transformation, enabling switching between three-dimensional space and a two-dimensional plane. The view space transformation process sequentially includes model matrix transformation, view matrix transformation, and projection matrix transformation, including:
[0097] 1) Model matrix transformation: Calculate the position of each point of the model in world space so that the model can be positioned in world space;
[0098] 2) View matrix transformation: Calculate the relative position of the object with respect to the camera;
[0099] 3) Projection matrix transformation: This is a perspective projection transformation that converts a three-dimensional object into a two-dimensional image that can be displayed on the screen.
[0100] Step S56: Perform a view space transformation between the graphic region and the reference surface contour point set according to the view space transformation matrix, and generate a first ROI region corresponding to the graphic region on the reference surface contour point set according to the transformation result.
[0101] Specifically, the view space transformation matrix can realize the view space transformation between the graphic region and the reference surface contour point set. That is, the feature points on the reference surface contour point set can be determined by the feature points of the graphic region, and the first ROI region on the reference surface contour point set corresponding to the graphic region can be determined accordingly. Alternatively, the feature points in the reference surface contour point set can be transformed, and the feature points that fall within the graphic region after transformation can be determined as the first ROI region corresponding to the graphic region in the area formed by the reference surface contour point set.
[0102] In some embodiments of this application, two optional methods are provided for determining the first ROI region based on the view space transformation matrix, namely, two different implementations of step S56, which involves performing a view space transformation between the graphic region and the reference surface contour point set based on the view space transformation matrix, and generating a first ROI region corresponding to the graphic region on the reference surface contour point set based on the transformation result. These two methods are described below, and may specifically include:
[0103] The first type
[0104] ① Determine the 2D coordinates of each feature point within the graphic region.
[0105] ② Determine the 2D coordinates corresponding to each feature point of the reference surface contour point set based on the view space transformation matrix.
[0106] ③ The feature points whose 2D coordinates are the same as those of the feature points in the graphic region are identified as target feature points.
[0107] ④ The region where the target feature point is located on the reference surface contour point set is determined as the first ROI region.
[0108] Specifically, based on the point set parameters, a view space transformation matrix matching the point set parameters can be determined. After determining the 3D coordinates of each feature point of the reference surface contour point set, the 2D coordinates corresponding to each feature point of the reference surface contour point set can be determined based on the view space transformation matrix. The 2D coordinates of each feature point within the graphic region are determined, and the 2D coordinates corresponding to each feature point of the reference surface contour point set are compared with the 2D coordinates of each feature point within the graphic region. Feature points whose 2D coordinates in the reference surface contour point set are the same as the 2D coordinates of the feature points within the graphic region are determined as target feature points, i.e., feature points of the reference surface contour point set that fall within the graphic region after transformation are determined as target feature points. Finally, the region or the region enclosed by the target feature points on the reference surface contour point set is determined as the first ROI region.
[0109] The second type
[0110] ① Determine the 2D coordinates of each feature point within the graphic region.
[0111] ② Determine the 3D coordinates corresponding to the 2D coordinates of each feature point within the graphic region based on the inverse of the visual space transformation matrix.
[0112] ③ The feature points whose coordinates on the reference surface contour point set are the same as the 3D coordinates of each feature point in the graphic region are determined as target feature points.
[0113] ④ The region where the target feature point is located on the reference surface contour point set is determined as the first ROI region.
[0114] Specifically, based on the point set parameters, a view space transformation matrix matching the point set parameters can be determined, and its inverse can be obtained by transformation. After determining the 2D coordinates of each feature point within the graphic region, the 3D coordinates corresponding to the 2D coordinates of each feature point within the graphic region can be determined based on the inverse of the view space transformation matrix. The 3D coordinates corresponding to each feature point in the reference surface contour point set are determined, and the 3D coordinates corresponding to each feature point within the graphic region are compared with those corresponding to each feature point in the reference surface contour point set. Feature points on the reference surface contour point set whose coordinates are the same as the 3D coordinates corresponding to each feature point within the graphic region are determined as target feature points. That is, each feature point within the graphic region is mapped to a corresponding feature point on the reference surface contour point set, and is thus determined as a target feature point. Finally, the region or the region enclosed by the target feature points on the reference surface contour point set is determined as the first ROI region.
[0115] In some embodiments of this application, in order to speed up the pose association calculation as much as possible, improve the pose association calculation efficiency, and at the same time not affect the pose association calculation accuracy, this embodiment provides a pose association calculation method that combines the first pose association calculation and the second pose association calculation.
[0116] The process of step S3, which involves performing pose correlation calculations on the reference surface contour point set and the real-time surface contour point set to obtain the correlation transformation matrix, is described below. Specifically, it may include:
[0117] Step S31: Perform first pose correlation calculation on the reference surface contour point set and the real-time surface contour point set to generate a correlation transformation matrix.
[0118] Step S32: Based on the previous pose association calculation, perform a second pose association calculation on the reference surface contour point set and the real-time surface contour point set, and update the association transformation matrix.
[0119] Step S33: If the current pose association calculation result is detected to be outside the allowable difference range, then return to the process of performing a second pose association calculation on the reference surface contour point set and the real-time surface contour point set based on the previous pose association calculation, until the current pose association calculation result is within the allowable difference range.
[0120] Specifically, firstly, a first pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set, and an association transformation matrix is generated. Based on the completion of the first pose association calculation, a second pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set, and the association transformation matrix is updated. It is then checked whether the current pose association calculation result obtained after this second pose association calculation is within the allowable difference range. If it is, the pose association calculation is considered complete; otherwise, based on this second pose association calculation, a second pose association calculation is performed again on the reference surface contour point set and the real-time surface contour point set, and the association transformation matrix is updated again, until the current pose association calculation result is within the allowable difference range.
[0121] The following provides an optional implementation process for first pose correlation calculation, describing step S31, which involves performing first pose correlation calculation on the reference surface contour point set and the real-time surface contour point set to generate the correlation transformation matrix. Specifically, it may include:
[0122] ① Filter and denoise the real-time surface contour point set and the reference surface contour point set respectively.
[0123] Specifically, the real-time surface contour point set and the reference surface contour point set are first filtered and denoised to avoid the influence of impurities on the pose association calculation process.
[0124] ② Determine the normal vectors of each first feature point in the reference surface contour point set and each second feature point in the real-time surface contour point set.
[0125] ③ Based on the normal vector, calculate and generate the feature point geometric description set of the real-time surface contour point set and the reference surface contour point set.
[0126] Specifically, the feature point geometric description set is generated by parametrically querying the spatial differences between a point and its neighboring points, and forming a multi-dimensional histogram to describe the geometric attributes within the k-neighborhood of a point. The steps for generating the feature point geometric description set include:
[0127] 1) Pre-set a 3D point set P{p1,p2,...p k} is a point p in the point set P. n Calculate the normal vector for n∈[1,k].
[0128] 2) with p i Using the center point, determine a k-neighborhood with a radius r, and calculate p. i The feature triples (α, φ, θ) between each point and its k-neighborhood are then used to statistically obtain the feature descriptor SP(p) of a point. i ).in:
[0129] α=v·n j
[0130]
[0131] θ = arctan(w·n) j ,u·n j )
[0132] p i ,p j Let n be two three-dimensional coordinate points, and let n be their respective normal vectors. i ,n j
[0133]
[0134] 3) For each point in the k-neighborhood, determine the k-neighborhood and calculate the feature descriptor for each point in the same way as in the previous step.
[0135] 4) Perform weighted statistics on each descriptor in the neighborhood, using the following formula;
[0136]
[0137] Among them, w j For point pair p i ,p j Weights are determined by p i ,p j It is measured by distance in space.
[0138] ④ Based on the geometric description set of the feature points, determine the association transformation matrix when the real-time surface contour point set is transformed to maintain pose association with the reference surface contour point set.
[0139] Furthermore, the second pose association calculation method used in step S32 of this application may include the following steps:
[0140] 1) Determine the first feature point in the reference surface contour point set that is closest to each second feature point in the real-time surface contour point set, and use the closest point as the matching point of the second feature point.
[0141] 2) Calculate the minimum matching error between matching points according to the following formula to obtain the pose transformation matrix.
[0142]
[0143] t * =p-Rp'
[0144] Where p is the reference surface profile point set, p' is the real-time surface profile point set, R is the rotation matrix, t is the translation matrix, and R and t together form the updated correlation transformation matrix.
[0145] 3) Transform the real-time surface contour point set according to the correlation transformation matrix obtained in the previous step to obtain the transformed real-time surface contour point set.
[0146] 4) Repeat the above process of calculating the pose transformation matrix and transforming the real-time surface contour point set according to the pose transformation matrix until the matching error between the matching points is minimized to less than the set threshold. At this point, the update of the associated transformation matrix can be determined based on the finally generated pose transformation matrix.
[0147] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0149] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for associating and labeling ROI regions, characterized in that, include: Obtain the real-time surface contour point set of the currently scanned detection object; Obtain a reference surface contour point set of the object to be tested, wherein the reference surface contour point set is the surface contour point set obtained by scanning the object to be tested with a positioning standard; Pose correlation calculation is performed on the reference surface contour point set and the real-time surface contour point set to obtain the correlation transformation matrix; The real-time surface contour point set is transformed according to the correlation transformation matrix to generate a transformed contour point set; A first ROI region is determined from the reference surface contour point set, and the first ROI region is transferred and transformed onto the transformed contour point set to generate a second ROI region. The first ROI region is the ROI region marked on the reference surface contour point set. According to the correlation transformation matrix, the transformed contour point set containing the second ROI region is inversely transformed to generate a target contour point set, which contains a third ROI region obtained by inversely transforming the second ROI region. Pose correlation calculations are performed on the reference surface contour point set and the real-time surface contour point set to obtain the correlation transformation matrix, including: A first pose correlation calculation is performed on the reference surface contour point set and the real-time surface contour point set to generate a correlation transformation matrix; Based on the previous pose association calculation, a second pose association calculation is performed on the reference surface contour point set and the real-time surface contour point set, and the association transformation matrix is updated. If the current pose association calculation result is detected to be outside the allowable difference range, then return to the process of performing a second pose association calculation on the reference surface contour point set and the real-time surface contour point set based on the previous pose association calculation, until the current pose association calculation result is within the allowable difference range.
2. The method according to claim 1, characterized in that, Determine a first ROI region from the reference surface contour point set, and transform the first ROI region onto the transformed contour point set to generate a second ROI region, including: Obtain the coordinate positions of feature points in the first ROI region of the reference surface contour point set, where the first ROI region is the ROI region marked on the reference surface contour point set; Identify feature points whose coordinate positions in the transformed contour point set are the same as those of feature points in the first ROI region; The second ROI region is determined based on the feature points whose coordinate positions are the same as those of the feature points in the first ROI region.
3. The method according to claim 1, characterized in that, Determining the first ROI region of the reference surface profile point set includes: In response to human operation, the point set parameters of the outlined graphic area and the reference surface contour point set are obtained; Determine the view space transformation matrix corresponding to the point set parameters; According to the view space transformation matrix, a view space transformation is performed between the graphic region and the reference surface contour point set, and a first ROI region corresponding to the graphic region is generated on the reference surface contour point set based on the transformation result.
4. The method according to claim 1, characterized in that, Perform a first pose correlation calculation on the reference surface contour point set and the real-time surface contour point set to generate a correlation transformation matrix, including: The reference surface contour point set and the real-time surface contour point set are respectively filtered and denoised; Determine each feature point that constitutes the reference surface contour point set and the normal vector of each feature point that constitutes the real-time surface contour point set; Based on the normal vector, calculate and generate the feature point geometric description set of the reference surface contour point set and the real-time surface contour point set; Based on the geometric description set of the feature points, determine the correlation transformation matrix of the real-time surface contour point set to be calculated in association with the pose of the reference surface contour point set.
5. The method according to claim 3, characterized in that, Based on the view space transformation matrix, a view space transformation is performed between the graphic region and the reference surface contour point set, and a first ROI region corresponding to the graphic region is generated on the reference surface contour point set according to the transformation result, including: Determine the 2D coordinates of each feature point within the graphic region; Based on the view space transformation matrix, determine the 2D coordinates corresponding to each feature point of the reference surface contour point set; The feature points whose 2D coordinates are the same as those of the feature points in the graphic region are identified as target feature points. The region where the target feature point is located on the reference surface contour point set is determined as the first ROI region.
6. The method according to claim 3, characterized in that, Based on the view space transformation matrix, a view space transformation is performed between the graphic region and the reference surface contour point set, and a first ROI region corresponding to the graphic region is generated on the reference surface contour point set according to the transformation result, including: Determine the 2D coordinates of each feature point within the graphic region; Based on the inverse of the view space transformation matrix, determine the 3D coordinates corresponding to the 2D coordinates of each feature point within the graphic region; The feature points whose coordinates on the reference surface contour point set are the same as the 3D coordinates of each feature point in the graphic region are determined as target feature points; The region where the target feature point is located on the reference surface contour point set is determined as the first ROI region.
7. The method according to claim 1, characterized in that, Before transforming the real-time surface contour point set according to the correlation transformation matrix to generate the transformed contour point set, the method further includes: Check whether the pose correlation calculation results are within the allowable difference range; If so, then the process of transforming the real-time surface contour point set according to the associated transformation matrix to generate a transformed contour point set is executed. If not, a prompt message will be displayed to remind the object being tested to adjust its position.
8. The method according to any one of claims 1-7, characterized in that, After transforming the real-time surface contour point set according to the correlation transformation matrix to generate the transformed contour point set, the method further includes: A unique correspondence is determined between each feature point constituting the transformed contour point set and each feature point constituting the reference surface contour point set, and the same index number is assigned to each pair of corresponding feature points.
Citation Information
Patent Citations
Point cloud registration method based on shape constraint
CN110276790A