A method and system for extracting target area from three-dimensional point cloud of body surface

Abstract

The present invention relates to the field of computer vision technology, and proposes a method and system for extracting a target area of ​​a body surface three-dimensional point cloud, wherein the method comprises the following steps: performing three-dimensional scanning and photographing on a target patient's body surface equipped with a windowed diaphragm to obtain three-dimensional point cloud data and a two-dimensional image of the target patient's body surface; constructing a mapping relationship based on the three-dimensional point cloud data and the two-dimensional image; inputting the two-dimensional image into an image segmentation model to obtain an ROI recognition result of the outer contour of the diaphragm windowed area in the two-dimensional image; traversing the three-dimensional points in the three-dimensional point cloud data, mapping the three-dimensional points to two-dimensional points of a one-dimensional image based on the mapping relationship, and judging whether they are located within the ROI recognition result of the viewing angle of the two-dimensional image, and if so, adding the three-dimensional point to a target area set, otherwise judging the next three-dimensional point; after completing the traversal, outputting the target area set as a target area extraction result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a method and system for extracting a target area from a three-dimensional point cloud of a body surface. Background Art

[0002] Tumor radiotherapy is a local treatment method that uses radiation to treat tumors. During radiotherapy, the accuracy of the radiation irradiation position must be ensured to avoid damaging other tissues and organs. During radiotherapy, a low-temperature thermoplastic film is often used to fix the patient's position, and the low-temperature thermoplastic film is opened according to clinical needs. In order to improve the accuracy of patient positioning, it is also necessary to use a body surface optical image guidance system to perform a three-dimensional body surface scan of the patient, obtain a three-dimensional point cloud of the patient's body surface, and perform body surface registration on the area where the film window is opened to confirm the accuracy of positioning and improve the safety of treatment.

[0003] At present, it is proposed to obtain the position of the object in the two-dimensional image and generate a two-dimensional bounding box, predict the three-dimensional bounding box in the three-dimensional point cloud data, and identify the physical center of the three-dimensional point cloud data. Finally, the virtual three-dimensional model is detected and identified based on the physical center and the three-dimensional bounding box to realize the identification of the diaphragm window area in the three-dimensional point cloud of the patient's body surface. However, due to the lack of texture information in the acquired three-dimensional point cloud of the patient's body surface, it is impossible to identify the corresponding area of ​​the diaphragm window in the three-dimensional point cloud of the patient's body surface according to the color of the diaphragm. At the same time, due to the high degree of fit between the diaphragm and the patient's body surface, it is impossible to identify the corresponding area of ​​the diaphragm window in the three-dimensional point cloud of the patient's body surface according to the shape of the window. In addition, during radiotherapy, as the treatment frame rotates, there is also the problem of occlusion of the window area scan, so it is difficult to accurately identify the target area of ​​the diaphragm window in the three-dimensional point cloud of the patient's body surface. Summary of the invention

[0004] In order to overcome the defect of the above-mentioned prior art that it is difficult to accurately identify the target area of ​​the diaphragm window in the three-dimensional point cloud of the patient's body surface, the present invention provides a method and system for extracting the target area of ​​the three-dimensional point cloud of the body surface.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A method for extracting a target area from a body surface three-dimensional point cloud comprises the following steps:

[0007] Performing three-dimensional scanning and photographing of the body surface of the target patient equipped with the window membrane to obtain three-dimensional point cloud data and two-dimensional images of the body surface of the target patient;

[0008] Constructing a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0009] Inputting the two-dimensional image into an image segmentation model to obtain a ROI recognition result about the outer contour of the diaphragm window area in the two-dimensional image;

[0010] Traverse the three-dimensional points in the three-dimensional point cloud data, and transform the three-dimensional points into 2D points mapped to 1D images and judge the two-dimensional point Is it located within the ROI recognition result in the 2D image? If so, the 3D point Add to the target area set, otherwise judge the next 3D point;

[0011] After the traversal is completed, the target area set is output as a target area extraction result.

[0012] Furthermore, the present invention also proposes a body surface three-dimensional point cloud target region extraction system, which applies the body surface three-dimensional point cloud target region extraction method proposed in the present invention. The system comprises:

[0013] An acquisition module is used to scan and photograph the body surface of a target patient equipped with a window membrane to obtain three-dimensional point cloud data and a two-dimensional image of the body surface of the target patient;

[0014] A mapping module, used to construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0015] An ROI recognition module, on which an image segmentation model is mounted, is used to input the two-dimensional image into the image segmentation model and output an ROI recognition result about the outer contour of the diaphragm window area in the two-dimensional image;

[0016] A target area extraction module is used to traverse the three-dimensional points in the three-dimensional point cloud data and convert the three-dimensional points into 2D points mapped to 1D images and judge the two-dimensional point Is it located within the ROI recognition result of the viewing angle of the two-dimensional image? If so, the three-dimensional point Add the target area set, otherwise judge the next three-dimensional point; after completing the traversal, output the target area set as the target area extraction result.

[0017] Furthermore, the present invention also proposes a device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the method for extracting target areas from a three-dimensional point cloud of a body surface as proposed in the present invention.

[0018] Furthermore, the present invention also proposes a storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the method for extracting a target area of ​​a body surface three-dimensional point cloud as proposed in the present invention.

[0019] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0020] The present invention performs three-dimensional scanning and two-dimensional image shooting of the body surface of the target patient equipped with a fenestrated diaphragm, uses a trained image segmentation model to accurately extract the ROI recognition result of the outer contour of the diaphragm fenestration area, and finally maps the three-dimensional point cloud into the two-dimensional image, screens the three-dimensional point cloud located in the ROI target area, and effectively and accurately completes the extraction of the target area of ​​the body surface three-dimensional point cloud;

[0021] The present invention utilizes an image segmentation model to perform ROI recognition on the outer contour of the diaphragm window area, which can adapt to two-dimensional image ROI recognition under different patients, different treatment sites, different diaphragm window sizes and positions, and different ambient light influences, and can accurately extract the ROI area of ​​the diaphragm window, thereby effectively improving the accuracy of extracting the target area of ​​the three-dimensional point cloud on the patient's body surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flowchart of a method for extracting a target area from a three-dimensional point cloud of a body surface according to an embodiment of the present invention.

[0023] Figure 2 The figure is a flow chart of target area recognition and determination according to an embodiment of the present invention.

[0024] Figure 3 The figure is a schematic diagram of a ROI recognition result of a two-dimensional image windowing region according to an embodiment of the present invention.

[0025] Figure 4 The figure is a diagram showing the effect of extracting a target area according to an embodiment of the present invention.

[0026] Figure 5 The figure is a schematic diagram of ROI recognition results of a windowed area of ​​a two-dimensional image with occlusion according to an embodiment of the present invention.

[0027] Figure 6 The figure is a diagram showing the effect of extracting a target area according to an embodiment of the present invention.

[0028] Figure 7 The figure is an architecture diagram of a body surface three-dimensional point cloud target region extraction system according to one embodiment of the present invention. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0030] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Example 1

[0034] This embodiment proposes a method for extracting a target area of ​​a body surface three-dimensional point cloud. Figure 1 , which is a flow chart of the method for extracting target areas from body surface three-dimensional point clouds in this embodiment.

[0035] The method for extracting a target area from a three-dimensional point cloud of a body surface proposed in this embodiment includes the following steps:

[0036] S1. Performing three-dimensional scanning and photographing of the body surface of the target patient equipped with the fenestration membrane to obtain three-dimensional point cloud data and a two-dimensional image of the body surface of the target patient;

[0037] S2. constructing a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0038] S3, inputting the two-dimensional image into an image segmentation model to obtain a ROI recognition result of the outer contour of the diaphragm window area in the two-dimensional image;

[0039] S4, traversing the three-dimensional points in the three-dimensional point cloud data, and mapping the three-dimensional points based on the mapping relationship. 2D points mapped to 1D images and judge the two-dimensional point Is it located within the ROI recognition result of the viewing angle of the two-dimensional image? If so, the three-dimensional point Add the target area set, otherwise judge the next three-dimensional point; after completing the traversal, output the target area set as the target area extraction result.

[0040] In this embodiment, the target patient's body surface equipped with a windowed diaphragm is three-dimensionally scanned and photographed, and the ROI recognition result of the outer contour of the diaphragm window area is accurately extracted through a trained image segmentation model. Finally, it is determined whether the three-dimensional points in the three-dimensional point cloud data are located in the ROI area of ​​any two-dimensional image. Based on the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the target area in the three-dimensional point cloud is determined, that is, the diaphragm window area of ​​the three-dimensional point cloud of the patient's body surface.

[0041] Exemplarily, in step S1, before the target patient is three-dimensionally scanned and photographed, a fixed window diaphragm is made for the target patient in a model making room, wherein a window is opened in an area where the target area motion correlation is relatively large, and the diaphragm is used to fix the target patient in the treatment room.

[0042] In an optional embodiment, in step S1, three-dimensional scanning and photographing of the target patient's body surface equipped with a fenestrated membrane comprises the following steps:

[0043] A scanning device is used to scan the body surface of the target patient to obtain three-dimensional point cloud data of the body surface of the target patient and two-dimensional images at different viewing angles; wherein, there are at least two scanning devices, and they are uniformly calibrated; the scanning devices are arranged at the same height, and adjacent scanning devices are arranged at equal intervals.

[0044] Further, in an optional embodiment, the scanning device includes a three-dimensional scanning device and a two-dimensional scanning device, or a binocular structured light camera.

[0045] When the scanning device adopts a binocular structured light camera, the binocular structured light camera is used to acquire a two-dimensional image to perform three-dimensional measurement and reconstruction, thereby obtaining three-dimensional point cloud data of the target patient's body surface.

[0046] When the scanning device uses a three-dimensional scanning device and a two-dimensional scanning device, the three-dimensional point cloud data of the target patient's body surface is collected by the three-dimensional scanning device, and the two-dimensional images of the target patient's body surface at different perspectives are collected by the two-dimensional scanning device. The two are associated through calibration.

[0047] Specifically, the number and / or location of the scanning devices are set according to the space in the treatment room and / or the actual needs of the position, shape, size, etc. of the diaphragm window configured on the target patient, and the present invention does not impose any limitation on this.

[0048] Exemplarily, one three-dimensional scanning device and three two-dimensional photographing devices are configured in the treatment room.

[0049] Exemplarily, when the number of installed scanning devices is greater than or equal to 2, each scanning device is set at an equal distance from the window membrane of the target patient, and adjacent scanning devices are set at equal intervals.

[0050] Exemplarily, the scanning device is installed at a first angle less than or equal to 30° looking downward.

[0051] Exemplarily, the scanning device is installed to look down at 30°.

[0052] Specifically, by uniformly calibrating the 3D scanning equipment, the internal parameters of the equipment and the coordinate system conversion parameters can be obtained, which can be further used to construct the mapping relationship between the 3D point cloud data and the 2D image. By setting the installation positions and the top-view angles of multiple 2D shooting devices and uniformly calibrating the 2D shooting devices, the mapping accuracy between 3D points and 2D pixel points can be improved.

[0053] For example, during the calibration process of the 3D scanning device, the calibration plate is optionally placed in the scanning area and ensured to be parallel to the lens of the 3D scanning device, and then the camera distance and angle are adjusted so that the calibration plate is completely covered within the camera field of view, and then the calibration plate is scanned using the 3D scanning device, and optionally, images from multiple perspectives are acquired by moving, rotating or tilting the 3D scanning device. When all necessary images are acquired and saved, the punctuation points on the calibration plate in the image are extracted by image processing software and the camera parameters are calculated to obtain and save the external parameter matrix, the internal parameter matrix and the distortion coefficient.

[0054] Exemplarily, in the calibration process of a two-dimensional shooting device, Zhang's calibration method can be optionally used. Specifically, a series of parallel calibration plates are placed on a plane, and the least squares method is used to solve and save the intrinsic parameter matrix and extrinsic parameter matrix of the camera according to the image coordinates and physical coordinates of the calibration plates in different positions and postures.

[0055] In step S1, a three-dimensional scan is performed on the body surface of the target patient equipped with a fenestrated membrane to obtain three-dimensional point cloud data of the body surface of the target patient, which may optionally include a spatial normal vector of the three-dimensional point.

[0056] In an optional example, in step S2, building a mapping relationship based on the three-dimensional point cloud data and the two-dimensional image includes the following steps:

[0057] The three-dimensional point cloud coordinates of the world coordinate system Convert to camera coordinate system coordinates , the expression of the world coordinate system-camera coordinate system conversion is:

[0058]

[0059] in, is the external parameter matrix, are the external parameters obtained by calibrating the scanning device, representing the three-dimensional rotation matrix and the three-dimensional translation matrix respectively;

[0060] The 3D point cloud coordinates of the camera coordinate system Convert to pixel coordinates , the camera coordinate system-pixel coordinate conversion expression is:

[0061]

[0062] in, is the internal parameter matrix, and The camera is x and y The focal length in the coordinate direction represents the equivalent value of the focal length from the camera optical center to the image plane in the pixel coordinate system, and its unit is pixel; and are the intersection points of the camera optical axis and the image plane in the image coordinate system. x and y Coordinates represent the projection point of the camera optical axis on the image, usually close to the geometric center of the image, and its unit is pixel;

[0063] Construct a three-dimensional point cloud according to the world coordinate system-camera coordinate system conversion formula and the camera coordinate system-pixel coordinate conversion formula With two-dimensional point The mapping relationship of .

[0064] Specifically, based on the mapping relationship, a three-dimensional point is determined In the process of whether the 3D point is located within the ROI recognition result in any 2D image, the 3D point is calculated by the mapping relationship. The two-dimensional pixel points after mapping , and then determine the two-dimensional pixel point Whether it is within the ROI area.

[0065] Further optionally, the two-dimensional pixel point is determined based on whether the product of the normal vector of the three-dimensional point and the unit vector is less than 0. Whether it is located at the viewing angle of the corresponding two-dimensional image.

[0066] In an optional embodiment, in step S3, the image segmentation model used for identifying the outer contour ROI of the diaphragm window area is pre-trained, and includes the following steps:

[0067] Collect two-dimensional images of different patients undergoing treatment at different parts of the body, and annotate the outer contour of the diaphragm window area in the two-dimensional image to obtain training data;

[0068] Performing data enhancement on the training data to obtain a training data set;

[0069] The training data set is input into the image segmentation model for training to obtain a trained image segmentation model, which is used to extract the ROI recognition result about the outer contour of the diaphragm window area from the input two-dimensional image.

[0070] Exemplarily, the acquired two-dimensional images include two-dimensional images of different patients, different treatment sites, different diaphragm window sizes and positions, and under different ambient lights.

[0071] Exemplarily, the labelme annotation tool is used to annotate the outer contour of the diaphragm window area in the two-dimensional image.

[0072] Exemplarily, operations for data enhancement on training data include rotation, cropping, deformation, and the like.

[0073] Exemplarily, in the process of training the image segmentation model, an adaptive learning rate adjustment strategy can be optionally adopted to dynamically adjust the learning rate according to the loss value during the training process, which helps to maintain the stability and convergence of the model during the training process.

[0074] Further optionally, the image segmentation model uses a YOLOv5 network.

[0075] Exemplarily, the image segmentation model includes a backbone network, a neck network, a head network, a post-processing network and a loss function. The backbone network is used to extract useful features from the input image; the neck network is used to integrate feature maps of different levels; the head network is used to predict the category and bounding box position of the target; the post-processing network is used to post-process the detection results, remove redundant detection boxes, and obtain the final detection results; the loss function is used to optimize the parameters of the model during the training process, and guide the update direction of the model by calculating the difference between the predicted value and the true value.

[0076] Exemplarily, in the process of training the image segmentation model, the CIoU (Complete Intersection over Union) loss function is optionally used to measure the matching degree of the target box, which can more accurately reflect the matching status of the target box, thereby improving the detection accuracy.

[0077] In this embodiment, after marking the outer contour of the window area of ​​the diaphragm in the training data set, the data set is input into the image segmentation model for learning. For situations in which the position and shape of the diaphragm are distributed differently, the ambient light is too bright or too dark, occlusion and other noise influences in the two-dimensional image, the image segmentation network can still effectively extract the outer contour detection frame of the window area and output it as the ROI recognition result.

[0078] Exemplarily, the image segmentation model in the present invention may also optionally use neural networks such as U-Net, R-FCN, Mask R-CNN, etc., which can effectively extract ROI, and the present invention is not limited thereto.

[0079] Compared with other neural networks, the YOLOv5 network is selected as the image segmentation model. Its ROI extraction has advantages in terms of efficiency, accuracy, ease of use, real-time performance and versatility.

[0080] In an optional embodiment, in step S4, determining whether the three-dimensional point is located within the ROI recognition result in any two-dimensional image includes the following steps:

[0081] (1) For i 3D points , based on the mapping relationship, the three-dimensional point Map to m 2D image A m Two-dimensional points in , and determine the two-dimensional point Is it in the m 2D image A m In the ROI recognition result:

[0082] If yes, go to step (2);

[0083] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged;

[0084] (2) The three-dimensional point Convert to camera coordinate system , calculate the 3D point in the camera coordinate system The normal vector With unit vector Is the product of less than 0:

[0085] If so, then the three-dimensional point Add the target area set and add the next 3D point Execute step (1) until all three-dimensional points have been judged;

[0086] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged;

[0087] in, i =1,2,..., N , N is the total number of three-dimensional points; m =1,2,..., M , M is the total number of 2D images.

[0088] Each two-dimensional image in this embodiment represents an image captured by a scanning device at a viewing angle. By determining the two-dimensional images one by one, it is possible to determine whether the current three-dimensional point falls within the ROI of the corresponding device viewing angle from different viewing angles. When it is determined that the current three-dimensional point falls within the ROI of the current two-dimensional image, the current three-dimensional point is not only determined by the ROI of the next image, but also by the ROI of the next image, to avoid repeated determination.

[0089] Among them, the three-dimensional point in the camera coordinate system The normal vector according to get.

[0090] like Figure 2 As shown, it is a target area recognition and judgment flow chart.

[0091] In this embodiment, the judgment condition in step (1) is that the three-dimensional point is located within the ROI recognition result of the two-dimensional image, specifically, the two-dimensional point mapped from the three-dimensional point to the two-dimensional image is inside the closed outer contour ROI.

[0092] For example, when determining a two-dimensional point Is it in the m 2D image A m When the ROI recognition result is within the ROI recognition result, the following steps are included:

[0093] Step 1: Traverse the adjacent two-dimensional points in the ROI recognition result to form line segments, and finally form a set of line segments with multiple ordered index IDs;

[0094] Step 2: Based on the two-dimensional point coordinates obtained through mapping , calculate the straight line in the two-dimensional image Whether there is an intersection point with the line segments in the line segment set; wherein, when there is an endpoint and The line segment satisfies and ,or and When , it means a straight line There is an intersection point with the line segments in the line segment set, and the horizontal axis coordinate of the intersection point is calculated at this time , get the coordinates of the intersection point;

[0095] Step 3: For each two-dimensional point obtained through mapping, calculate the intersection point set composed of the intersection points, sort the horizontal coordinates of the intersection point set from small to large, and record the sorted index sequence number ID;

[0096] Step 4: In the sorted set of intersection points, find the first one that is not less than The index number id of the intersection point and judge the two-dimensional point Whether it is inside the closed outer contour ROI; let the index difference of the first intersection point after distance sorting be id− 0, when satisfied id % 2=1, then the two-dimensional point is judged Located inside the closed outer contour ROI.

[0097] According to the above judgment rules, the two-dimensional point Is it in the m 2D image A m The ROI recognition result is within .

[0098] In this embodiment, the judgment condition in step (2) is the three-dimensional point Normal Vector vs Unit Vector The product of is less than 0, that is, the three-dimensional point The angle between the normal vector and the optical axis unit vector of the 3D scanning device is greater than 90°, indicating that the 3D point Towards the scanning end of the 3D scanning device, or the 3D point Located within the scanning range of the 3D scanning device.

[0099] Specifically, the three-dimensional model obtained by scanning the scanning device should be under the viewing angle of the scanning device, and the angle between the unit vector of the optical axis of the three-dimensional scanning device and the normal vector of the three-dimensional point cloud should be greater than 90°, that is, the product of the normal vector of the three-dimensional point cloud and the unit vector of the optical axis is less than 0. Only then can the point be the point scanned by the device. Otherwise, it means that the three-dimensional point belongs to the back of the viewing angle of the scanning device and does not match the current viewing angle.

[0100] This embodiment performs body surface three-dimensional point cloud target area extraction through double determination, which can effectively improve the accuracy of three-dimensional point cloud target area extraction.

[0101] Specifically, after the traversal is completed, the three-dimensional point cloud data is marked according to the target area set, and the marked three-dimensional point cloud data is output, and the marked area is the target area extraction result.

[0102] Specifically, this embodiment also verifies the body surface three-dimensional point cloud target area extraction method of the present invention through experiments.

[0103] like Figure 3 FIG. 1 is a schematic diagram of a ROI recognition result of a two-dimensional image windowing region according to an embodiment of the present invention, wherein Figure 3 Parts (a), (b), and (c) are two-dimensional images of ROI recognition results with windowed areas at different angles.

[0104] like Figure 4 FIG. 1 is a diagram showing the effect of extracting a target area according to an embodiment of the present invention, wherein Figure 4 Parts (a) and (b) are comparison images before and after target area extraction, respectively.

[0105] As well as Figure 5 FIG. 1 is a schematic diagram of a ROI recognition result of a windowed two-dimensional image with occlusion according to an embodiment of the present invention, wherein Figure 5 Parts (a), (b), and (c) are two-dimensional images of ROI recognition results with windowed areas at different angles, and part (c) contains occlusion representation.

[0106] Figure 6 The figure is a diagram showing the effect of extracting a target area according to an embodiment of the present invention.

[0107] The target area extraction result shown in the figure shows that the target area extraction method of the three-dimensional point cloud of the body surface proposed in the present invention can still accurately identify the target area of ​​the diaphragm window in the three-dimensional point cloud of the patient's body surface in the presence of occlusion. Specifically, by mapping and matching the ROI area of ​​the two-dimensional image with the three-dimensional point cloud data, it is determined whether the three-dimensional point is located within the ROI recognition result in any two-dimensional image and belongs to the scanning range of the corresponding three-dimensional scanning device. The three-dimensional point is then used as a three-dimensional point in the target area corresponding to the diaphragm window. Finally, the target area set is output as the target area extraction result to complete the target area extraction of the three-dimensional point cloud of the body surface.

[0108] Example 2

[0109] This embodiment proposes a body surface three-dimensional point cloud target region extraction system, and applies the body surface three-dimensional point cloud target region extraction method proposed in Embodiment 1. Figure 7 , which is an architecture diagram of the body surface three-dimensional point cloud target area extraction system of this embodiment.

[0110] The body surface three-dimensional point cloud target area extraction system proposed in this embodiment includes:

[0111] An acquisition module is used to scan and photograph the body surface of a target patient equipped with a window membrane to obtain three-dimensional point cloud data and a two-dimensional image of the body surface of the target patient;

[0112] A mapping module, used to construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0113] An ROI recognition module, on which an image segmentation model is mounted, is used to input the two-dimensional image into the image segmentation model and output an ROI recognition result about the outer contour of the diaphragm window area in the two-dimensional image;

[0114] The target area extraction module is used to traverse the three-dimensional points in the three-dimensional point cloud data, and based on the mapping relationship, determine whether they are located in the ROI recognition result of the perspective of the two-dimensional image. If so, the three-dimensional point is added to the target area set, otherwise the next three-dimensional point is judged; after completing the traversal, the target area set is output as the target area extraction result.

[0115] In an optional embodiment, the acquisition module is used to scan the body surface of the target patient to obtain three-dimensional point cloud data of the body surface of the target patient and two-dimensional images at different viewing angles.

[0116] Wherein, the acquisition module optionally includes at least two scanning devices, and the scanning devices are uniformly calibrated.

[0117] Further optionally, the scanning devices are arranged at the same height, and adjacent scanning devices are arranged at equal intervals.

[0118] Further, in an optional embodiment, the scanning device includes a three-dimensional scanning device and a two-dimensional scanning device, or a binocular structured light camera.

[0119] Further, in an optional embodiment, when the mapping module constructs a mapping relationship based on the three-dimensional point cloud data and the two-dimensional image, the mapping module performs the following steps:

[0120] The three-dimensional point cloud coordinates of the world coordinate system Convert to camera coordinate system coordinates , the expression of the world coordinate system-camera coordinate system conversion is:

[0121]

[0122] in, is the external parameter matrix, is the external parameter obtained by calibrating the scanning equipment;

[0123] The 3D point cloud coordinates of the camera coordinate system Convert to pixel coordinates , the camera coordinate system-pixel coordinate conversion expression is:

[0124]

[0125] in, is the internal parameter matrix, and The camera is x and y The focal length in the coordinate direction, and are the intersection points of the camera optical axis and the image plane in the image coordinate system. x and y coordinate;

[0126] Construct a three-dimensional point cloud according to the world coordinate system-camera coordinate system conversion formula and the camera coordinate system-pixel coordinate conversion formula With two-dimensional point The mapping relationship of .

[0127] Furthermore, in an optional embodiment, the acquisition module is also used to acquire two-dimensional images of different patients during treatment of different parts of the body, and to mark the outer contour of the diaphragm window area in the two-dimensional image to obtain training data.

[0128] Furthermore, the training data is used to pre-train the image segmentation model carried by the ROI recognition module. The training process includes:

[0129] Performing data enhancement on the training data to obtain a training data set;

[0130] The training data set is input into the image segmentation model for training to obtain a trained image segmentation model, which is used to extract the ROI recognition result about the outer contour of the diaphragm window area from the input two-dimensional image.

[0131] Furthermore, in an optional embodiment, the image segmentation model carried by the ROI recognition module includes a YOLOv5 network model.

[0132] Further, in an optional embodiment, when determining whether the three-dimensional point is located within the ROI recognition result in any two-dimensional image, the target region extraction module performs the following steps:

[0133] (1) For i 3D points , based on the mapping relationship, determine whether it is in the first m 2D image Am In the ROI recognition result:

[0134] If yes, go to step (2);

[0135] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1).

[0136] (2) i Convert 3D points to camera coordinate system , calculate the 3D point The normal vector With unit vector Is the product of less than 0:

[0137] If so, then the three-dimensional point Add the target area set and add the next 3D point Execute step (1) until all three-dimensional points have been judged;

[0138] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged;

[0139] in, i =1,2,..., N , N is the total number of three-dimensional points; m =1,2,..., M , M is the total number of 2D images.

[0140] It can be understood that the system of this embodiment corresponds to the method of the above-mentioned embodiment 1, and other optional items in the above-mentioned embodiment 1 are also applicable to this embodiment, so they are not described repeatedly here.

[0141] Example 3

[0142] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the method for extracting a target area from a three-dimensional point cloud of a body surface as proposed in Embodiment 1.

[0143] Example 4

[0144] This embodiment proposes a storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the method for extracting a target area from a body surface three-dimensional point cloud proposed in Embodiment 1.

[0145] Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0146] By way of example, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.

[0147] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of the method for extracting the target area of ​​the body surface three-dimensional point cloud described in Example 1.

[0148] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely exemplary, in which the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same or one or more software and / or hardware when implementing the scheme of the present invention. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the scheme of this embodiment.

[0149] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for extracting a target area from a body surface three-dimensional point cloud, characterized in that: The following steps are involved: Performing three-dimensional scanning and photographing of the body surface of the target patient equipped with the window membrane to obtain three-dimensional point cloud data and two-dimensional images of the body surface of the target patient; Constructing a mapping relationship between the three-dimensional point cloud data and the two-dimensional image; Inputting the two-dimensional image into an image segmentation model to obtain a ROI recognition result about the outer contour of the diaphragm window area in the two-dimensional image; Traverse the three-dimensional points in the three-dimensional point cloud data, and transform the three-dimensional points into 2D points mapped to 1D images and judge the two-dimensional point Is it located within the ROI recognition result of the viewing angle of the two-dimensional image? If so, the three-dimensional point Add to the target area set, otherwise judge the next three-dimensional point; wherein, in judging the two-dimensional point Whether it is located within the ROI recognition result of the viewing angle of the two-dimensional image, the following steps are included: (1) For i 3D points , based on the mapping relationship, the three-dimensional point Map to m 2D image A m Two-dimensional points in , and determine the two-dimensional point Is it in the m 2D image A m In the ROI recognition result: If yes, go to step (2); Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged; (2) The three-dimensional point Convert to camera coordinate system , calculate the three-dimensional point The normal vector With unit vector Is the product of less than 0: If so, then the three-dimensional point Add the target area set and add the next 3D point Execute step (1) until all three-dimensional points have been judged; Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged; in, i =1,2,..., N , N is the total number of three-dimensional points; m =1,2,..., M , M is the total number of 2D images; After the traversal is completed, the target area set is output as a target area extraction result.

2. The method for extracting target areas from a body surface three-dimensional point cloud according to claim 1, characterized in that: The method further comprises the following steps: Collect two-dimensional images of different patients undergoing treatment at different parts of the body, and annotate the outer contour of the diaphragm window area in the two-dimensional image to obtain training data; Performing data enhancement on the training data to obtain a training data set; The training data set is input into the image segmentation model for training to obtain a trained image segmentation model, which is used to extract the ROI recognition result about the outer contour of the diaphragm window area from the input two-dimensional image.

3. The method for extracting target areas from a body surface three-dimensional point cloud according to claim 2, characterized in that: The image segmentation model includes a YOLOv5 network model.

4. The method for extracting target areas from body surface three-dimensional point clouds according to claim 1, characterized in that: The step of constructing a mapping relationship based on the three-dimensional point cloud data and the two-dimensional image comprises the following steps: The three-dimensional point cloud coordinates of the world coordinate system Convert to camera coordinate system coordinates , the expression of the world coordinate system-camera coordinate system conversion is: in, is the external parameter matrix, is the external parameter obtained by calibrating the scanning equipment; The 3D point cloud coordinates of the camera coordinate system Convert to pixel coordinates , the camera coordinate system-pixel coordinate conversion expression is: in, is the internal parameter matrix, and The camera is x and y The focal length in the coordinate direction, and are the intersection points of the camera optical axis and the image plane in the image coordinate system. x and y coordinate; Construct a three-dimensional point cloud according to the world coordinate system-camera coordinate system conversion formula and the camera coordinate system-pixel coordinate conversion formula With two-dimensional point The mapping relationship of .

5. The method for extracting a target area from a three-dimensional point cloud of a body surface according to any one of claims 1 to 4, characterized in that: The three-dimensional scanning and photographing of the target patient's body surface equipped with the fenestrated membrane comprises the following steps: Scanning the body surface of the target patient using a scanning device to obtain three-dimensional point cloud data of the body surface of the target patient and two-dimensional images at different viewing angles; There are at least two scanning devices, which are uniformly calibrated; the scanning devices are arranged at the same height, and adjacent scanning devices are arranged at equal intervals.

6. The method for extracting target areas from body surface three-dimensional point clouds according to claim 5, characterized in that: The scanning device includes a three-dimensional scanning device and a two-dimensional scanning device, or includes a binocular structured light camera.

7. A body surface three-dimensional point cloud target area extraction system, characterized in that: An acquisition module is used to scan and photograph the body surface of a target patient equipped with a window membrane to obtain three-dimensional point cloud data and a two-dimensional image of the body surface of the target patient; A mapping module, used to construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image; An ROI recognition module, on which an image segmentation model is mounted, is used to input the two-dimensional image into the image segmentation model and output an ROI recognition result about the outer contour of the diaphragm window area in the two-dimensional image; A target area extraction module is used to traverse the three-dimensional points in the three-dimensional point cloud data and convert the three-dimensional points into 2D points mapped to 1D images and judge the two-dimensional point Is it located within the ROI recognition result of the viewing angle of the two-dimensional image? If so, the three-dimensional point Add to the target area set, otherwise judge the next 3D point; After the traversal is completed, the target area set is output as a target area extraction result; The target area extraction module determines the two-dimensional point If the image is located within the ROI recognition result of the viewing angle of the two-dimensional image, the following steps are performed: (1) For i 3D points , based on the mapping relationship, the three-dimensional point Map to m 2D image A m Two-dimensional points in , and determine the two-dimensional point Is it in the m 2D image A m In the ROI recognition result: If yes, go to step (2); Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged; (2) The three-dimensional point Convert to camera coordinate system , calculate the 3D point The normal vector With unit vector Is the product of less than 0: If so, then the three-dimensional point Add the target area set and add the next 3D point Execute step (1) until all three-dimensional points have been judged; Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all 2D images have been judged, then the next 3D point Execute step (1) until all three-dimensional points have been judged; in, i =1,2,..., N , N is the total number of three-dimensional points; m =1,2,..., M , M is the total number of 2D images.

8. The body surface three-dimensional point cloud target area extraction system according to claim 7, characterized in that: The acquisition module is also used to acquire two-dimensional images of different patients undergoing treatment of different parts of the body, and to mark the outer contour of the diaphragm window area in the two-dimensional image to obtain training data; The training data is used to train the image segmentation model; The training process includes: Performing data enhancement on the training data to obtain a training data set; The training data set is input into the image segmentation model for training to obtain a trained image segmentation model, which is used to extract the ROI recognition result about the outer contour of the diaphragm window area from the input two-dimensional image.

9. The body surface three-dimensional point cloud target area extraction system according to claim 8, characterized in that: The image segmentation model includes a YOLOv5 network model.

10. The body surface three-dimensional point cloud target area extraction system according to claim 7, characterized in that: When the mapping module constructs a mapping relationship based on the three-dimensional point cloud data and the two-dimensional image, the mapping module performs the following steps: The three-dimensional point cloud coordinates of the world coordinate system Convert to camera coordinate system coordinates , the expression of the world coordinate system-camera coordinate system conversion is: in, is the external parameter matrix, is the external parameter obtained by calibrating the scanning equipment; The 3D point cloud coordinates of the camera coordinate system Convert to pixel coordinates , the camera coordinate system-pixel coordinate conversion expression is: in, is the internal parameter matrix, and The camera is x and y The focal length in the coordinate direction, and are the intersection points of the camera optical axis and the image plane in the image coordinate system. x and y coordinate; Construct a three-dimensional point cloud according to the world coordinate system-camera coordinate system conversion formula and the camera coordinate system-pixel coordinate conversion formula With two-dimensional point The mapping relationship of .

11. The body surface three-dimensional point cloud target area extraction system according to any one of claims 7 to 10, characterized in that: When scanning and photographing the body surface of a target patient equipped with a windowed membrane, the acquisition module performs the following steps: Scanning the body surface of the target patient to obtain three-dimensional point cloud data of the body surface of the target patient and two-dimensional images at different viewing angles; Wherein, the acquisition module includes at least two scanning devices, and the scanning devices are uniformly calibrated; the scanning devices are arranged at the same height, and adjacent scanning devices are arranged at the same intervals.

12. The body surface three-dimensional point cloud target area extraction system according to claim 11, characterized in that: The scanning device includes a three-dimensional scanning device and a two-dimensional scanning device, or includes a binocular structured light camera.

13. A device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor executes the steps of the method for extracting a target area from a three-dimensional point cloud of a body surface as described in any one of claims 1 to 6.

14. A storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the method for extracting a target area from a three-dimensional point cloud of a body surface as described in any one of claims 1 to 6 are implemented.

Images