Method and apparatus for registration of two-dimensional images and three-dimensional models

Abstract

The application is suitable for the field of image processing technology, and provides a registration method and device for a two-dimensional image and a three-dimensional model. The method first extracts a two-dimensional image contour of a preset projection image, wherein the preset projection image is formed by projecting a target surface of a three-dimensional model of a target object to a preset plane, the target surface of the three-dimensional model is a surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction. Then, ray casting is performed on the two-dimensional image contour to extract a three-dimensional feature line of the three-dimensional model. Finally, based on the three-dimensional feature line, the two-dimensional image containing the target object and the three-dimensional model are registered. Thus, by the method of the two-dimensional image contour and the ray casting, the three-dimensional feature line on the three-dimensional model of the target object is automatically extracted and used for subsequent registration. Compared with the manual drawing method, the accuracy of the three-dimensional feature line extraction can be significantly improved, and thus the accurate registration of the two-dimensional image and the three-dimensional model is realized.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a method and apparatus for registering two-dimensional images with three-dimensional models. Background Technology

[0002] Registration of two-dimensional (2D) images with three-dimensional (3D) models refers to the process of aligning and matching a two-dimensional image containing a target object with a three-dimensional model of the target object. This process is widely used in fields such as medical imaging, computer vision, and computer graphics.

[0003] In related technologies, before registration, it is necessary to manually mark the feature lines of the target object on the two-dimensional image and the three-dimensional model, and then achieve registration by matching the two-dimensional feature lines and the three-dimensional feature lines.

[0004] However, certain feature lines in the 3D model of the target object, such as the upper or lower boundary lines, need to be determined by the pose of the model and the pose of the rendering camera. They are difficult to draw manually, which leads to the inability to achieve accurate registration between the 2D image and the 3D model. Summary of the Invention

[0005] This application provides a registration method and apparatus for two-dimensional images and three-dimensional models, which can solve the problem that some feature lines in a three-dimensional model are difficult to draw manually, thus making it impossible to achieve accurate registration between two-dimensional images and three-dimensional models.

[0006] The first aspect of this application provides a registration method for a two-dimensional image and a three-dimensional model, comprising: extracting a two-dimensional image contour of a preset projection image, wherein the preset projection image is formed by projecting the target surface of a three-dimensional model of a target object onto a preset plane, the target surface of the three-dimensional model being a surface of the three-dimensional model facing a preset direction, and the preset plane being perpendicular to the preset direction; projecting light onto the two-dimensional image contour to extract three-dimensional feature lines of the three-dimensional model; and registering the two-dimensional image containing the target object and the three-dimensional model based on the three-dimensional feature lines.

[0007] In one possible implementation of the first aspect, the above-mentioned registration of a two-dimensional image containing a target object with a three-dimensional model based on three-dimensional feature lines includes:

[0008] Based on the three-dimensional feature lines and the two-dimensional feature lines in the two-dimensional image corresponding to the three-dimensional feature lines, the two-dimensional image and the three-dimensional model are registered.

[0009] Optionally, in another possible implementation of the first aspect, the registration result of the two-dimensional image and the three-dimensional model is a target registration matrix, which represents the mapping relationship between the two-dimensional image and the three-dimensional model. The above-mentioned registration of the two-dimensional image and the three-dimensional model based on the three-dimensional feature lines and the two-dimensional feature lines in the two-dimensional image corresponding to the three-dimensional feature lines includes:

[0010] Based on 3D and 2D feature lines, the 2D image and 3D model are iteratively registered to generate a reference registration matrix. The reference registration matrix is ​​determined as the target registration matrix when it meets the preset convergence condition.

[0011] Optionally, in another possible implementation of the first aspect, the above-mentioned iterative registration of the two-dimensional image and the three-dimensional model based on three-dimensional feature lines and two-dimensional feature lines to generate a reference registration matrix, until the reference registration matrix satisfies a preset convergence condition, then the reference registration matrix is ​​determined as the target registration matrix, including:

[0012] Determine the two-dimensional projection lines corresponding to the three-dimensional feature lines on the preset projection image;

[0013] Between the two-dimensional projection lines and the two-dimensional feature lines, determine multiple pairs of two-dimensional matching points;

[0014] Based on multiple sets of two-dimensional matching point pairs, determine multiple sets of two-dimensional matching point pairs between three-dimensional feature lines and two-dimensional feature lines;

[0015] Registration calculations are performed on multiple sets of 2D-3D matching point pairs to generate a reference registration matrix;

[0016] When the reference registration matrix does not meet the preset convergence condition, the pose of the 3D model is changed using the reference registration matrix, the preset projection image is re-determined, and the above iterative registration steps are repeated to iteratively register the 2D image and the 3D model.

[0017] When the reference registration matrix satisfies the preset convergence condition, the reference registration matrix is ​​determined as the target registration matrix.

[0018] Optionally, in another possible implementation of the first aspect, determining multiple sets of two-dimensional matching point pairs between the two-dimensional projection lines and the two-dimensional feature lines includes:

[0019] Based on multiple first starting points on the two-dimensional projection line, multiple first matching points corresponding to the two-dimensional feature line are determined, wherein the first starting point is any point on the two-dimensional projection line, and the first starting point is matched with the first matching point one by one.

[0020] Based on multiple second starting points on the two-dimensional feature line, multiple second matching points corresponding to the two-dimensional projection line are determined, wherein the second starting point is any point on the two-dimensional feature line, and the second starting point is matched with the second matching point one by one.

[0021] Based on multiple first starting points, multiple first matching points, multiple second starting points, and multiple second matching points, multiple sets of two-dimensional matching point pairs are determined.

[0022] Optionally, in another possible implementation of the first aspect, the above-mentioned registration calculation for multiple sets of two-dimensional-three-dimensional matching point pairs to generate a reference registration matrix includes:

[0023] The Random Sample Consensus-Perspective-n-Point (RANSAC-PnP) algorithm is used to perform registration calculations on multiple sets of 2D-3D matching point pairs to generate a reference registration matrix.

[0024] Optionally, in another possible implementation of the first aspect, the above-mentioned ray projection on the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model includes:

[0025] By projecting light onto the contour of a two-dimensional image, a three-dimensional surface contour of the target object can be obtained.

[0026] Based on the surface contour of the three-dimensional target, extract the three-dimensional feature lines of the three-dimensional model.

[0027] Optionally, in another possible implementation of the first aspect, the aforementioned preset projection image is formed by projecting the surface of the three-dimensional model facing a preset direction onto a preset plane using a virtual camera. The aforementioned projection of light onto the two-dimensional image contour to obtain the three-dimensional target surface contour of the target object includes:

[0028] Based on the optical center of the virtual camera and each point included in the two-dimensional image contour, multiple light rays are generated;

[0029] Collision detection is performed on each ray and the surface patch of the 3D model to identify multiple target surface patches;

[0030] Multiple target surface patches are used to form a three-dimensional target surface contour.

[0031] Optionally, in another possible implementation of the first aspect, the above-mentioned three-dimensional feature lines include three-dimensional target contour lines and three-dimensional visible ridge lines, or the three-dimensional feature lines include three-dimensional target contour lines, three-dimensional visible ridge lines and three-dimensional fixed feature lines. The three-dimensional target contour lines change with the pose of the three-dimensional model, while the three-dimensional visible ridge lines and three-dimensional fixed feature lines do not change with the pose of the three-dimensional model. The three-dimensional target contour lines and the three-dimensional visible ridge lines together constitute the boundary lines of the three-dimensional target surface contour.

[0032] When the three-dimensional feature lines include the three-dimensional target contour lines and the three-dimensional visible ridge lines, the two-dimensional feature lines corresponding to the three-dimensional feature lines in the two-dimensional image include the two-dimensional target contour lines and the two-dimensional visible ridge lines. When the three-dimensional feature lines include the three-dimensional target contour lines, the three-dimensional visible ridge lines and the three-dimensional fixed feature lines, the two-dimensional feature lines include the two-dimensional target contour lines, the two-dimensional visible ridge lines and the two-dimensional fixed feature lines. The two-dimensional target contour lines and the two-dimensional visible ridge lines together constitute the boundary lines of the target objects in the two-dimensional image.

[0033] The 3D visible ridge is the intersection of the 3D target surface contour of the target object and the complete ridge of the 3D model. The 3D target contour is the 3D target surface contour without the 3D visible ridge.

[0034] Optionally, in another possible implementation of the first aspect, the above registration method further includes:

[0035] The system detects the first editing command input by the user through the visual editing interface and obtains the first line segment drawing information carried in the first editing command.

[0036] In response to the first line segment drawing information in the first editing instruction, a three-dimensional fixed feature line is generated on the three-dimensional model.

[0037] Optionally, in another possible implementation of the first aspect, the registration method further includes:

[0038] Obtain a 3D reference model, which includes 3D reference feature lines;

[0039] By utilizing the mapping relationship between the 3D model and the 3D reference model, the 3D reference feature lines are mapped onto the 3D model to obtain 3D fixed feature lines.

[0040] Optionally, in another possible implementation of the first aspect, the registration method further includes:

[0041] The system detects a second editing command input by the user through the visual editing interface, which carries information about drawing a second line segment.

[0042] In response to the second line segment drawing information in the second editing instruction, at least one of the following is generated on the two-dimensional image: a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line.

[0043] Optionally, in another possible implementation of the first aspect, the above registration method further includes:

[0044] Extract multiple candidate feature lines from a 2D image;

[0045] The system detected a third editing command input by the user through the visual editing interface, which carried information about the start and end points of the line segment.

[0046] In response to the start and end point information of the line segment in the third editing instruction, at least one of the following is generated: a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line.

[0047] Optionally, in another possible implementation of the first aspect, the registration method further includes:

[0048] A pre-trained deep learning model is invoked to perform line segment extraction processing on a two-dimensional image to generate at least one of a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line.

[0049] Optionally, in another possible implementation of the first aspect, before extracting the two-dimensional image contour of the preset projection image, the registration method further includes:

[0050] Based on the adjacency matrix of the surface of the 3D model, and using the region growing method, the surface of the 3D model facing the preset direction is determined.

[0051] Optionally, in another possible implementation of the first aspect, determining the surface of the 3D model facing a preset direction based on the adjacency matrix of the surface patches of the 3D model and using the region growing method includes:

[0052] Intersect the preset local coordinate axis corresponding to the 3D model with the 3D model to determine the seed point of the surface of the 3D model facing the preset direction, where the positive direction of the preset local coordinate axis faces the preset direction;

[0053] Based on the adjacency matrix, the model expands outward from the seed point until it reaches the complete ridge of the 3D model, in order to determine the surface of the 3D model facing the preset direction.

[0054] Optionally, in another possible implementation of the first aspect, before the registration method, based on the adjacency matrix and expanding outward from the seed point until a complete ridge of the 3D model is encountered to determine the surface of the 3D model facing a preset direction, further includes:

[0055] The complete ridge line of the 3D model is determined based on the normal angles of adjacent faces on the 3D model.

[0056] Optionally, in another possible implementation of the first aspect, the aforementioned preset projection image is formed by projecting the surface of the 3D model facing a preset direction onto a preset plane using a virtual camera. Before extracting the 2D image contour of the preset projection image, the registration method further includes:

[0057] Using a virtual camera, the surface of a 3D model facing a preset direction is rendered to obtain a preset projection image.

[0058] Optionally, in another possible implementation of the first aspect, the extraction of the two-dimensional image contour of the preset projection image includes:

[0059] The preset projection image is binarized to determine the target region containing the target object in the preset projection image.

[0060] By narrowing down the target area, a two-dimensional image outline is obtained.

[0061] Optionally, in another possible implementation of the first aspect, the above-mentioned reduction of the target region to obtain a two-dimensional image contour includes:

[0062] Morphological erosion is performed on the target region to reduce its size and obtain a two-dimensional image contour.

[0063] A second aspect of this application provides a registration apparatus for a two-dimensional image and a three-dimensional model, comprising:

[0064] The contour extraction module is used to extract the two-dimensional image contour of the preset projection image. The preset projection image is formed by projecting the target surface of the three-dimensional model of the target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction.

[0065] The ray casting module is used to cast rays onto the contours of a two-dimensional image in order to extract the three-dimensional feature lines of the three-dimensional model;

[0066] The registration module is used to register a 2D image containing a target object with a 3D model based on 3D feature lines.

[0067] A third aspect of this application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the registration method of the two-dimensional image and the three-dimensional model described in the first aspect.

[0068] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the registration method for two-dimensional images and three-dimensional models described in the first aspect.

[0069] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the registration method of the two-dimensional image and three-dimensional model described in the first aspect.

[0070] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment discloses a method and apparatus for registering a two-dimensional image and a three-dimensional model. The method first extracts the two-dimensional image contour of a preset projection image, which is formed by projecting the target surface of a three-dimensional model of a target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction. Then, light is projected onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model. Finally, based on the three-dimensional feature lines, the two-dimensional image containing the target object and the three-dimensional model are registered. Therefore, by using the two-dimensional image contour and light projection method, the three-dimensional feature lines on the three-dimensional model of the target object are automatically extracted and used for subsequent registration. Compared with manual drawing, this significantly improves the accuracy of three-dimensional feature line extraction, thereby achieving accurate registration of the two-dimensional image and the three-dimensional model. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart illustrating a registration method between a two-dimensional image and a three-dimensional model provided in Embodiment 1 of this application;

[0073] Figure 2 This is a schematic diagram of the adjacency relationship of the upper slices of the three-dimensional liver model provided in the embodiments of this application;

[0074] Figure 3 This is a complete ridge line diagram of the three-dimensional liver model provided in the embodiments of this application;

[0075] Figure 4 This is a schematic diagram of the front surface of a three-dimensional liver model provided in an embodiment of this application;

[0076] Figure 5This is a schematic diagram illustrating an application scenario of projecting light onto a two-dimensional liver image contour, as provided in this application embodiment.

[0077] Figure 6 This is a schematic diagram of the anterior surface contour of the liver provided in an embodiment of this application;

[0078] Figure 7 This is a flowchart illustrating a registration method between a two-dimensional image and a three-dimensional model provided in Embodiment 2 of this application;

[0079] Figure 8 This is a schematic diagram of the three-dimensional feature lines in the three-dimensional liver model provided in the embodiments of this application;

[0080] Figure 9 This is a schematic diagram of two-dimensional feature lines of an endoscopic video frame provided in an embodiment of this application;

[0081] Figure 10 This is a flowchart illustrating a registration method between a two-dimensional image and a three-dimensional model provided in Embodiment 3 of this application;

[0082] Figure 11 This is a schematic diagram of the structure of a registration device for a two-dimensional image and a three-dimensional model provided in Embodiment 4 of this application;

[0083] Figure 12 This is a schematic diagram of the structure of a terminal device provided in Embodiment 5 of this application. Detailed Implementation

[0084] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0085] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0086] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0087] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0088] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0089] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0090] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0091] In related technologies, before registering a 2D image with a 3D model, it is necessary to manually mark the feature lines of the target object on both the 2D image and the 3D model, and then achieve registration by matching the 2D and 3D feature lines. However, some feature lines in the 3D model of the target object, such as the upper or lower boundary lines, need to be determined by the pose of the model and the pose of the rendering camera, which is difficult to draw manually, thus making it impossible to achieve accurate registration between the 2D image and the 3D model.

[0092] In view of this, embodiments of this application provide a method and apparatus for registering a two-dimensional image with a three-dimensional model. First, a two-dimensional image contour of a preset projection image is extracted. This preset projection image is formed by projecting the target surface of a three-dimensional model of a target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction. Then, light is projected onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model. Finally, based on the three-dimensional feature lines, the two-dimensional image containing the target object and the three-dimensional model are registered. Thus, by using the two-dimensional image contour and light projection method, the three-dimensional feature lines on the three-dimensional model of the target object are automatically extracted and used for subsequent registration. Compared to manual drawing, this significantly improves the accuracy of three-dimensional feature line extraction, thereby achieving precise registration between the two-dimensional image and the three-dimensional model.

[0093] The following examples illustrate the application scenarios of the registration method for two-dimensional images and three-dimensional models provided in this application. This application can be applied to endoscopic minimally invasive surgery, a mainstream method for treating soft tissue organs, such as laparoscopic liver tumor resection. However, in minimally invasive surgery, the limited endoscopic field of view restricts the perception of the surgical environment, making it difficult for surgeons to locate key internal structures such as tumors. The registration method provided in this application allows for the registration of the preoperatively constructed three-dimensional model to the endoscopic field of view, achieving registration between the preoperative model and the intraoperative image, providing surgeons with precise guidance during the operation.

[0094] To illustrate the technical solution of this application, specific embodiments are described below.

[0095] Reference Figure 1 This diagram illustrates a flowchart of a registration method for a two-dimensional image and a three-dimensional model according to Embodiment 1 of this application. This method can be executed by a registration device for a two-dimensional image and a three-dimensional model. This registration device can be implemented in software and / or hardware, and can be implemented as a terminal or a server. Figure 1 As shown, the registration method between the two-dimensional image and the three-dimensional model may include the following steps:

[0096] Step 101: Extract the two-dimensional image contour of the preset projection image.

[0097] The preset projection image is formed by projecting the target surface of the three-dimensional model of the target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction.

[0098] It should be noted that when constructing a 3D model, a default viewing direction is usually set. The entire 3D model can be observed by changing the viewing direction or rotating the model. For ease of explanation, in this embodiment, the visible portion of the 3D model in the viewing direction is defined as the front part of the 3D model, and the invisible portion in the viewing direction is defined as the rear part of the 3D model. From a given viewpoint, only the front part of the 3D model of the target object will be seen. Therefore, the currently visible portion of the 3D model can be defined as the target surface, that is, the surface of the 3D model facing the preset direction.

[0099] As an example, a registration device can be used to invoke a virtual camera to project a 3D model onto a preset plane, forming a preset projected image. The virtual camera simulates a real-world camera, possessing a viewpoint (position) and a field of view (visible range). The virtual camera's position and orientation determine the viewing angle of the scene, while the field of view determines the range of the scene the camera can capture. The virtual camera can render a 3D model from a specific viewpoint, thereby rendering the scene into one or more 2D images.

[0100] In one possible implementation of this application embodiment, before step 101 above, the method provided in this application embodiment may further include: the terminal device acquiring a preset projection image. For example, the terminal device can acquire the preset projection image in the following way: the terminal device uses a virtual camera to render the surface of the three-dimensional model facing a preset direction to obtain the preset projection image.

[0101] As an example, the surface of a 3D model facing a predetermined direction can be obtained from the adjacency matrix of the surface patches of the 3D model using the region growing method.

[0102] For example, a 3D model is typically composed of many faces, which can be triangles, quadrilaterals, or other polygonal shapes. These faces are connected together to create a complete 3D shape. For instance, a triangular face consists of three vertices and three sides, while a quadrilateral face consists of four vertices and four sides. An adjacency matrix is ​​a matrix that represents the structure of a graph, where each element indicates whether there is a connection between two nodes. In other words, the adjacency relationship of faces in a 3D model can be obtained through the adjacency matrix.

[0103] For example, region growing refers to the process of developing a group of pixels or regions into a larger region. It can start from a seed point and gradually expand the region using the adjacency relationships of the surface patches on the 3D model until a surface of the 3D model facing a preset direction is obtained. Here, a seed point is an initial point or position used to start a certain calculation or operation. Seed points are usually used to guide algorithms, segmentation, simulation, or other calculation processes involving spatial data.

[0104] The adjacency matrix of the surface patches in a 3D model can be used to expand outward from the seed point until a boundary is encountered. This boundary is the complete ridge of the 3D model. A ridge is a curve formed by the highest or lowest point along a certain direction on a surface. Specifically, in 3D space, surfaces can have various morphological features, including undulations, valleys, and slopes. These features represent changes in the height or position of points on the surface. A ridge is a curve on the surface along which the surface reaches its highest or lowest point. In other words, a ridge is a special path on the surface, and following this path, one can traverse the peaks or valleys of the surface.

[0105] As a possible implementation of this application, the process of determining the surface of the 3D model facing the preset direction based on the adjacency matrix of the surface of the 3D model and using the region growing method may include: intersecting the preset local coordinate axis corresponding to the 3D model with the 3D model to determine the seed point of the surface of the 3D model facing the preset direction, wherein the positive direction of the preset local coordinate axis faces the preset direction; based on the adjacency matrix, expanding outward from the seed point until the expansion stops at the complete ridge line of the 3D model to determine the surface of the 3D model facing the preset direction.

[0106] For example, suppose the preset local coordinate axis is the z-axis, and the positive direction of the z-axis points towards the aforementioned preset direction. Based on this, the intersection of the positive z-axis of the 3D model's local coordinate axis with the 3D model itself can determine the seed point of the surface of the 3D model facing the preset direction. Then, starting from the seed point, by checking the adjacent faces of the seed point, faces adjacent to the seed point are selected for expansion. This process requires defining a growth condition; for example, expansion is only performed when the angle between the normal of the adjacent face and the normal of the seed point is less than a certain threshold. The selected adjacent faces are added to a growth queue and marked as visited. Iteratively, the next face is taken from the growth queue, its adjacent faces are checked, and adjacent faces that meet the growth condition are added to the growth queue. This process is repeated until a complete ridge of the 3D model is encountered. The area formed at this point is the target surface, that is, the surface of the 3D model facing the preset direction.

[0107] Since the complete ridge of a 3D model exhibits relative continuity where the normal changes significantly, this characteristic can be used to determine the complete ridge of the 3D model. In one possible implementation of this application, the registration method, which expands outward from a seed point based on an adjacency matrix until it encounters the complete ridge of the 3D model, before determining the surface of the 3D model facing a preset direction, may further include: determining the complete ridge of the 3D model based on the normal angles of adjacent faces on the 3D model.

[0108] Specifically, the process begins by calculating the normal vector for each facet of the 3D model. The normal vector is perpendicular to the facet and describes its orientation. Then, for each pair of adjacent faces, the difference in normal angles between them is calculated, for example, using methods like dot product or cross product to calculate the angle between vectors. When the difference in normals between a facet and its surrounding faces is too large, that facet may be part of a ridge. Therefore, a normal angle threshold can be set, which can be adjusted based on model characteristics and continuity requirements. Finally, based on the difference in normal angles and the normal angle threshold, the portion where the difference in normal angles exceeds the threshold is identified as a complete ridge.

[0109] Optionally, to accurately extract the 2D image contour of the preset projection image, the preset projection image can first be binarized to determine the target region containing the target object. Binarization converts the preset projection image into an image containing only two pixel values, such as white and black, or red and black; the region containing one of these colors is the target region containing the target object. This target region can then be reduced in size to ensure that each ray intersects with the 3D model during ray projection in subsequent step 102. After reducing the target region, its contour is extracted to obtain the 2D image contour of the target object.

[0110] As an example, morphological erosion can be performed on the target region to shrink it and obtain a two-dimensional image contour. Morphological erosion is mainly used to remove certain parts of an image; for example, it can be used to shrink the extracted target region inward by a few pixels, thus achieving the goal of reducing the target region size.

[0111] To make it easier to understand, the following example of a liver surgery scenario will be used to illustrate the complete process of step 101 above.

[0112] It should be noted that a three-dimensional model of the liver can be constructed preoperatively using a CT (Computed Tomography) sequence, which is an image sequence composed of multiple CT scan slices.

[0113] Reference Figure 2 This diagram illustrates the adjacency relationship of the upper slices of the three-dimensional liver model provided in this embodiment. Figure 2 As shown, the adjacency relationships between faces can be obtained by calculating the adjacency matrix of the faces in the 3D liver model. Assume... Figure 2 There is a reference patch 21 on the three-dimensional model of the liver. The 11 triangular patches formed by the white lines around the reference patch 21 are the patches adjacent to the reference patch 21.

[0114] Reference Figure 3 This diagram illustrates the complete ridge line of a three-dimensional liver model provided in this embodiment. While the complete ridge line 31 of the liver is a closed ring, only the front portion is actually visible on the three-dimensional liver model; the rear portion is obscured by the liver parenchyma. Potential ridge lines can be identified by the difference in normal angles between adjacent facets. Outlier filtering can then remove erroneous facets, resulting in the complete ridge line 31 of the three-dimensional liver model. Outlier filtering is a data processing technique used to identify and remove outliers or abnormal points in a dataset. Since the detected potential ridge lines may contain erroneous facets, such as noise or local anomalies, outlier filtering can be used to identify and remove these errors, thus preserving a more realistic ridge structure.

[0115] Next, the front surface of the liver 3D model (the target surface of the 3D model) can be obtained using the region growing method. It should be noted that the purpose of region growing is to obtain one of two sub-regions of the liver surface segmented by the complete ridge line. The specific surface chosen is determined by the seed point obtained. After the sub-region is determined, the front surface of the liver 3D model can be obtained by combining it with the current viewing perspective of the 3D model. (Reference) Figure 4 This is a schematic diagram of the front surface of a three-dimensional liver model provided in this application embodiment. The front surface 41 of the three-dimensional liver model is shown below. Figure 4 As shown. Additionally... Figure 4 The front surface 41 of the three-dimensional liver model in the viewpoint is specifically the visceral surface of the liver.

[0116] The front surface of a 3D liver model is rendered using a virtual camera to obtain a preset projected image. Additionally, the background can be changed to a solid RGB (0,0,0) color during rendering, which is beneficial for subsequent extraction of the 2D image contour.

[0117] Finally, the preset projection image is sequentially binarized and subjected to morphological erosion operations to extract the two-dimensional image contour of the liver preset projection image.

[0118] Step 102: Perform ray projection on the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model.

[0119] In this embodiment, ray projection onto a two-dimensional image contour can map the two-dimensional image contour onto a three-dimensional model of the target object, thereby obtaining the three-dimensional surface contour of the target object. Compared with the traditional manual drawing method, the three-dimensional feature lines automatically extracted based on the three-dimensional target surface contour are more accurate. That is, step 102 above can include: ray projection onto the two-dimensional image contour to obtain the three-dimensional target surface contour of the target object; and extracting the three-dimensional feature lines of the three-dimensional model based on the three-dimensional target surface contour.

[0120] Specifically, starting from the optical center of the virtual camera, a ray can be generated through each point on the two-dimensional image contour. Each ray will collide with a patch on the surface of the three-dimensional model. The colliding patches are recorded, and these patches can be used to form the three-dimensional surface contour of the target object.

[0121] As one possible implementation of this application, the above-described method of projecting light onto a two-dimensional image contour to obtain a three-dimensional target surface contour of the target object may include: generating multiple light rays based on the optical center corresponding to the virtual camera and each point included in the two-dimensional image contour; performing collision detection on each light ray and a patch on the surface of the three-dimensional model to determine multiple target surface patches; and using the multiple target surface patches to construct a three-dimensional target surface contour. The target surface patches are those that have collided.

[0122] It should be noted that the edges or contours of each target surface patch can be extracted, and then the three-dimensional target surface contour can be formed by connecting the edges of adjacent target surface patches.

[0123] For example, taking a liver surgery scenario, refer to... Figure 5 This diagram illustrates an application scenario for ray projection onto the contours of a two-dimensional liver image. Figure 5 The lower right corner area is the preset projection image 51. The final extracted anterior surface contour of the liver 61 (three-dimensional target surface contour) is as follows. Figure 6 As shown.

[0124] Step 103: Based on the three-dimensional feature lines, register the two-dimensional image containing the target object with the three-dimensional model.

[0125] In this embodiment, three-dimensional feature lines on a three-dimensional model can be accurately extracted based on the surface contour of a three-dimensional target. Then, two-dimensional feature lines corresponding to the three-dimensional feature lines are also extracted from the two-dimensional image. Thus, two-dimensional-three-dimensional line matching can be performed using the three-dimensional feature lines on the three-dimensional model and the two-dimensional feature lines on the two-dimensional image to find the correspondence between the two-dimensional image and the three-dimensional model, thereby achieving registration between the two-dimensional image and the three-dimensional model.

[0126] The registration method for two-dimensional images and three-dimensional models disclosed in the above embodiments of this application first extracts the two-dimensional image contour of a preset projection image. The preset projection image is formed by projecting the target surface of a three-dimensional model of a target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction. Then, light is projected onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model. Finally, based on the three-dimensional feature lines, the two-dimensional image containing the target object and the three-dimensional model are registered. Thus, by using the two-dimensional image contour and light projection method, the three-dimensional feature lines on the three-dimensional model of the target object are automatically extracted and used for subsequent registration. Compared with manual drawing, this significantly improves the accuracy of three-dimensional feature line extraction, thereby achieving accurate registration of the two-dimensional image and the three-dimensional model.

[0127] The following specific example illustrates the process of registering a two-dimensional image with a three-dimensional model using three-dimensional feature lines on a three-dimensional model and two-dimensional feature lines on a two-dimensional image.

[0128] See Figure 7 The diagram illustrates a flowchart of a registration method for a two-dimensional image and a three-dimensional model provided in Embodiment 2 of this application. Figure 7 As shown, the registration method between the two-dimensional image and the three-dimensional model may include the following steps:

[0129] Step 701: Extract the two-dimensional image contour of the preset projection image.

[0130] The specific implementation process and principle of step 701 above can be found in the detailed description of the above embodiments, and will not be repeated here.

[0131] Step 702: Perform ray projection on the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model.

[0132] Specifically, to further improve the accuracy of registration between 2D images and 3D models, a combination of contour feature lines and visible ridge lines can be used to achieve registration between 2D images and 3D models. Specifically, 3D feature lines can include 3D target contour lines and 3D visible ridge lines. The 3D visible ridge line is the intersection of the 3D target surface contour and the complete ridge line of the 3D model. The 3D target contour line is the portion of the 3D target surface contour minus the 3D visible ridge line. The 3D target contour line changes with the pose of the 3D model, while the 3D visible ridge line does not change with the pose of the 3D model. The 3D target contour line and the 3D visible ridge line together constitute the boundary line of the 3D target surface contour. It should be understood that since a complete ridge line is a closed loop, only a portion is visible on the 3D model—the 3D visible ridge line—so the 3D visible ridge line can be directly used in the subsequent registration process.

[0133] For example, taking a liver surgery scenario as an example, refer to Figure 8 This shows a schematic diagram of the three-dimensional feature lines in a three-dimensional model of the liver. (Example) Figure 8 As shown, the three-dimensional feature lines include the three-dimensional upper contour line 81 and the three-dimensional visible ridge line 82. The three-dimensional upper contour line 81 changes with the pose of the three-dimensional model, while the three-dimensional visible ridge line 82 does not change with the pose of the three-dimensional model. The three-dimensional target contour line and the three-dimensional visible ridge line together constitute the boundary line of the three-dimensional target surface contour.

[0134] Optionally, the three-dimensional feature lines may also include fixed feature lines that are fully visible on the surface of the three-dimensional model facing a preset direction, and these fixed three-dimensional feature lines do not change with the pose of the three-dimensional model. Matching the above three feature lines simultaneously can further improve the registration accuracy. For example, in a liver surgery scenario, the fixed three-dimensional feature line could be the falciform ligament line 83 on the liver.

[0135] The extraction method for three-dimensional feature lines is explained in detail below.

[0136] The extraction method of the complete ridge line can be understood through the aforementioned embodiments. Therefore, the 3D visible ridge line can be obtained by intersecting the complete ridge line with the 3D target surface contour. Furthermore, since the 3D target contour line and the 3D visible ridge line together constitute the boundary line of the 3D target surface contour, the 3D target contour line can be obtained by removing the portion of the 3D visible ridge line from the 3D target surface contour. That is, as a possible implementation of this application embodiment, step 702 above may include: determining the 3D visible ridge line as the intersection of the 3D target surface contour and the complete ridge line of the 3D model; and determining the 3D target contour line as the portion of the 3D target surface contour after removing the 3D visible ridge line. This application embodiment, considering that the 3D visible ridge line does not change with the pose of the 3D model and has some similarity to the 3D target surface contour, extracts the 3D visible ridge line of the 3D model and registers it with the 3D target surface contour, thus improving the registration accuracy.

[0137] In one possible extraction method for three-dimensional fixed feature lines, the position of the three-dimensional fixed feature lines on the three-dimensional model does not change with the liver pose. Therefore, users can directly draw them manually on the three-dimensional model. That is, by detecting the first editing command input by the user through the visual editing interface, the first line segment drawing information carried in the first editing command is obtained; in response to the first line segment drawing information in the first editing command, the three-dimensional fixed feature lines on the three-dimensional model are generated.

[0138] Among them, a visual editing interface refers to an interface that allows users to edit, modify, or manipulate data, images, text, graphics, 3D models, or other visual content through a graphical user interface or other visual tools. This enables users to interact with data or content in an intuitive and interactive way without needing to delve into programming or complex command-line operations.

[0139] Another possible method for extracting 3D fixed feature lines is to automatically extract them using statistical atlases. This involves establishing a general 3D reference model based on statistical data, then establishing a mapping relationship between the 3D reference model and the 3D model, and finally mapping the 3D fixed feature lines of the 3D reference model onto the 3D model. Specifically, this can be achieved by detecting the first editing command input by the user through the visual editing interface, obtaining the first line segment drawing information carried in the first editing command, and generating 3D fixed feature lines on the 3D model in response to the first line segment drawing information in the first editing command.

[0140] In another possible extraction method for three-dimensional fixed feature lines, in the context of liver surgery, the three-dimensional falciform ligament line (three-dimensional fixed feature line) can be extracted from the CT sequence corresponding to the liver using a pre-trained deep learning model. After constructing the three-dimensional model, the falciform ligament line on the three-dimensional model can be obtained directly.

[0141] Step 703: Based on the three-dimensional feature lines and the two-dimensional feature lines in the two-dimensional image corresponding to the three-dimensional feature lines, register the two-dimensional image and the three-dimensional model.

[0142] Among them, the three-dimensional target contour line corresponds to the two-dimensional target contour line in the two-dimensional image, the three-dimensional visible ridge line corresponds to the two-dimensional visible ridge line in the two-dimensional image, and the three-dimensional fixed feature line corresponds to the two-dimensional fixed feature line in the two-dimensional image.

[0143] For example, taking a liver surgery scenario, refer to... Figure 9 This illustrates a two-dimensional feature line diagram of an endoscopic video frame. (See diagram for example.) Figure 9 As shown, the 3D upper contour line (3D target contour line) corresponds to the 2D upper contour line (2D target contour line) 91, the 3D visible ridge line corresponds to the 2D visible ridge line 92, and the 3D falciform ligament line (3D fixed feature line) corresponds to the 2D falciform ligament line (3D fixed feature line) 93. It is important to note that the 2D feature line corresponding to the 3D feature line must be detectable under endoscopic visualization; otherwise, it cannot be used for subsequent registration.

[0144] It should be noted that, corresponding to the three-dimensional feature lines, before registering the two-dimensional image and the three-dimensional model, it is necessary to extract the two-dimensional feature lines of the target object in the two-dimensional image.

[0145] The following section elaborates on the extraction methods for two-dimensional feature lines (two-dimensional target contour lines, two-dimensional visible ridge lines, and two-dimensional fixed feature lines).

[0146] In one possible method for extracting two-dimensional feature lines, the user can manually draw each two-dimensional feature line on the two-dimensional image. Specifically, by detecting a second editing command input by the user through a visual editing interface, which carries second line segment drawing information, at least one of a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line is generated on the two-dimensional image in response to the second line segment drawing information in the second editing command. This process is the same as the manual extraction of three-dimensional fixed feature lines in the aforementioned embodiments, and will not be repeated here.

[0147] In another possible method for extracting two-dimensional feature lines, candidate contours and candidate anatomical feature lines can be extracted, and then the user can manually click on the start and end points of the line segments to determine the target line segment. That is, multiple candidate feature lines can be extracted from the two-dimensional image; a third editing command input by the user through a visual editing interface can be detected, carrying the start and end point information of the line segments; in response to the start and end point information of the line segments in the third editing command, at least one of a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line can be generated.

[0148] Another possible method for extracting two-dimensional feature lines is through deep learning. Specifically, this involves using a pre-trained deep learning model, such as the CaseNet model or the UNet (U-shaped network) model, to extract two-dimensional feature lines. In other words, by calling a pre-trained deep learning model, line segment extraction can be performed on a two-dimensional image to generate at least one of two-dimensional target contour lines, two-dimensional visible ridge lines, or two-dimensional fixed feature lines. The CaseNet model is an end-to-end trainable convolutional neural network architecture used to solve category-aware semantic edge detection problems. The CaseNet model extends traditional edge detection tasks, enabling it to handle more complex scenes, such as those with multiple object categories. The UNet model is a neural network architecture for image segmentation, suitable for image segmentation tasks, and has achieved significant success in the field of medical image processing.

[0149] The registration method for two-dimensional images and three-dimensional models disclosed in the above embodiments of this application first extracts the two-dimensional image contour of a preset projection image; then, it projects light onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model; finally, it registers the two-dimensional image and the three-dimensional model based on the three-dimensional feature lines and the two-dimensional feature lines in the two-dimensional image corresponding to the three-dimensional feature lines. Thus, by using the three-dimensional feature lines on the three-dimensional model and the two-dimensional feature lines to perform two-dimensional-three-dimensional line matching, accurate registration of the two-dimensional image and the three-dimensional model is achieved.

[0150] In one possible implementation of this application, the registration result between the 2D image and the 3D model is a target registration matrix, which represents the mapping relationship between the 2D image and the 3D model. The target registration matrix can be constructed by iteratively registering the 2D image and the 3D model. Based on the target registration matrix, fast, real-time, and accurate registration between the 2D image and the 3D model can be achieved. The iterative registration process is described in detail below.

[0151] See Figure 10 The diagram illustrates a flowchart of a registration method for a two-dimensional image and a three-dimensional model provided in Embodiment 3 of this application. Figure 10 As shown, the registration method between the two-dimensional image and the three-dimensional model may include the following steps:

[0152] Step 1001: Extract the two-dimensional image contour of the preset projection image.

[0153] Step 1002: Perform ray projection on the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model.

[0154] The specific implementation process and principle of steps 1001-1002 above can be found in the detailed description of the above embodiments, and will not be repeated here.

[0155] Step 1003: Based on the three-dimensional feature lines and the two-dimensional feature lines, iteratively register the two-dimensional image and the three-dimensional model to generate a reference registration matrix until the reference registration matrix meets the preset convergence condition, then the reference registration matrix is ​​determined as the target registration matrix.

[0156] In this embodiment of the application, the preset convergence condition is specifically described in the following formula:

[0157] |T n+1 -T n |2 <threshold;

[0158] Among them, T n+1 T represents the reference registration matrix for the (n+1)th iteration. n represents the reference registration matrix for the nth iteration, and threshold represents the threshold for convergence assessment.

[0159] It should be understood that when the reference registration matrix satisfies the above-mentioned preset convergence condition, the reference registration matrix is ​​determined as the target registration matrix. The convergence judgment threshold can be set according to the actual application scenario, and this embodiment does not impose any restrictions on it.

[0160] In one possible implementation of this application, step 1003 may include: determining a two-dimensional projection line corresponding to a three-dimensional feature line on a preset projection image; determining multiple sets of two-dimensional-two-dimensional matching point pairs between the two-dimensional projection line and the two-dimensional feature line; determining multiple sets of two-dimensional-three-dimensional matching point pairs between the three-dimensional feature line and the two-dimensional feature line based on the multiple sets of two-dimensional-two-dimensional matching point pairs; performing registration calculations on the multiple sets of two-dimensional-three-dimensional matching point pairs to generate a reference registration matrix; when the reference registration matrix does not meet the preset convergence condition, changing the pose of the three-dimensional model using the reference registration matrix, re-determining the preset projection image, and repeating the above iterative registration steps to iteratively register the two-dimensional image and the three-dimensional model; when the reference registration matrix meets the preset convergence condition, determining the reference registration matrix as the target registration matrix.

[0161] Specifically, a bidirectional nearest-point matching method can be used to find two-dimensional matching point pairs. This involves performing two corresponding point matches: starting from the two-dimensional projection line, finding the nearest point on the two-dimensional feature line for each point on that line as its matching point; then, starting from the two-dimensional feature line, finding the nearest matching point on the two-dimensional projection line. This mutual matching ensures that a sufficient number of matching point pairs participate in subsequent registration, avoiding registration failure due to insufficient point pairs. As one possible implementation of this application, determining multiple sets of two-dimensional matching point pairs between the two-dimensional projection line and the two-dimensional feature line includes: determining first matching points on the two-dimensional feature line corresponding to each first starting point based on multiple first starting points on the two-dimensional projection line, wherein each first starting point and each first matching point is matched one-to-one; determining multiple second matching points on the two-dimensional projection line based on multiple second starting points on the two-dimensional feature line, wherein each second starting point and each second matching point is matched one-to-one; and determining multiple sets of two-dimensional matching point pairs based on multiple first starting points, multiple first matching points, multiple second starting points, and multiple second matching points. Here, the first starting point is any point on the two-dimensional projection line, and the second starting point is any point on the two-dimensional feature line. For any first starting point, the point closest to it on the two-dimensional feature line is found and used as the first matching point corresponding to the first starting point. Correspondingly, for any second starting point, the point closest to it on the two-dimensional projection line is found and used as the second matching point corresponding to the second starting point.

[0162] It is important to note that each corresponding feature line must be matched independently.

[0163] In one possible implementation of this application, the RANSAC-PnP algorithm can be used to perform registration calculations on multiple sets of 2D-3D matching point pairs to generate a reference registration matrix. The RANSAC-PnP algorithm is a computer vision algorithm used to solve camera pose estimation problems. It is typically used to estimate the camera's position and orientation from a set of 2D image points and their corresponding points in a 3D image.

[0164] The registration method for two-dimensional images and three-dimensional models disclosed in the above embodiments of this application first extracts the two-dimensional image contour of a preset projection image; then, it projects light onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model; finally, based on the three-dimensional feature lines and the two-dimensional feature lines, iteratively registers the two-dimensional image and the three-dimensional model to generate a reference registration matrix until the reference registration matrix meets a preset convergence condition, at which point the reference registration matrix is ​​determined as the target registration matrix. Thus, by iteratively registering the two-dimensional image and the three-dimensional model, a target registration matrix is ​​constructed, and based on the target registration matrix, fast, real-time, and accurate registration of the two-dimensional image and the three-dimensional model can be achieved.

[0165] See Figure 11 The diagram shows a structural schematic of a registration device for a two-dimensional image and a three-dimensional model provided in Embodiment 4 of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0166] The registration device for two-dimensional images and three-dimensional models may specifically include the following modules:

[0167] The contour extraction module 1101 is used to extract the two-dimensional image contour of the preset projection image. The preset projection image is formed by projecting the target surface of the three-dimensional model of the target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction.

[0168] The ray projection module 1102 is used to project rays onto the contour of a two-dimensional image in order to extract the three-dimensional feature lines of a three-dimensional model.

[0169] The registration module 1103 is used to register a two-dimensional image containing a target object with a three-dimensional model based on three-dimensional feature lines.

[0170] The registration apparatus for two-dimensional images and three-dimensional models disclosed in the above embodiments of this application first extracts the two-dimensional image contour of a preset projection image. The preset projection image is formed by projecting the target surface of a three-dimensional model of a target object onto a preset plane. The target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction. Then, light is projected onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model. Finally, based on the three-dimensional feature lines, the two-dimensional image containing the target object and the three-dimensional model are registered. Thus, by using the two-dimensional image contour and light projection method, the three-dimensional feature lines on the three-dimensional model of the target object are automatically extracted and used for subsequent registration. Compared with manual drawing, this significantly improves the accuracy of three-dimensional feature line extraction, thereby achieving accurate registration of the two-dimensional image and the three-dimensional model.

[0171] In one possible implementation of Embodiment 4 of this application, the registration module 1103 may specifically include the following sub-modules:

[0172] The first execution submodule is used to register the two-dimensional image and the three-dimensional model based on the three-dimensional feature lines and the two-dimensional feature lines in the two-dimensional image that correspond to the three-dimensional feature lines.

[0173] Furthermore, in another possible implementation of Embodiment 4 of this application, the registration result of the two-dimensional image and the three-dimensional model is a target registration matrix, which is used to represent the mapping relationship between the two-dimensional image and the three-dimensional model. The first execution submodule may specifically include the following units:

[0174] The first execution unit is used to iteratively register a two-dimensional image and a three-dimensional model based on three-dimensional feature lines and two-dimensional feature lines to generate a reference registration matrix until the reference registration matrix meets the preset convergence condition, at which point the reference registration matrix is ​​determined as the target registration matrix.

[0175] Furthermore, in yet another possible implementation of Embodiment 4 of this application, the first execution unit described above may also be used for:

[0176] On the preset projection image, determine the two-dimensional projection lines corresponding to the three-dimensional feature lines; between the two-dimensional projection lines and the two-dimensional feature lines, determine multiple sets of two-dimensional-two-dimensional matching point pairs; based on the multiple sets of two-dimensional-two-dimensional matching point pairs, determine multiple sets of two-dimensional-three-dimensional matching point pairs between the three-dimensional feature lines and the two-dimensional feature lines; perform registration calculations on the multiple sets of two-dimensional-three-dimensional matching point pairs to generate a reference registration matrix; if the reference registration matrix does not meet the preset convergence conditions, use the reference registration matrix to change the pose of the three-dimensional model, redetermine the preset projection image, and repeat the above iterative registration steps to iteratively register the two-dimensional image and the three-dimensional model; when the reference registration matrix meets the preset convergence conditions, determine the reference registration matrix as the target registration matrix.

[0177] Furthermore, in another possible implementation of Embodiment 4 of this application, the first execution unit described above may also be used for:

[0178] Based on multiple first starting points on the two-dimensional projection line, multiple first matching points corresponding to the two-dimensional feature line are determined, wherein the first starting point is any point on the two-dimensional projection line, and the first starting point is matched with the first matching point one by one; based on multiple second starting points on the two-dimensional feature line, multiple second matching points corresponding to the two-dimensional projection line are determined, wherein the second starting point is any point on the two-dimensional feature line, and the second starting point is matched with the second matching point one by one; based on multiple first starting points, multiple first matching points, multiple second starting points, and multiple second matching points, multiple sets of two-dimensional-two-dimensional matching point pairs are determined.

[0179] Furthermore, in another possible implementation of Embodiment 4 of this application, the first execution unit described above can also be used to: use the Random Sampling Consensus-Perspective Point Problem (RANSAC-PnP) algorithm to perform registration calculations on multiple sets of two-dimensional-three-dimensional matching point pairs to generate a reference registration matrix.

[0180] Furthermore, in yet another possible implementation of Embodiment 4 of this application, the light projection module 1102 may further include the following sub-modules:

[0181] The second execution submodule is used to project light onto the two-dimensional image contour to obtain the three-dimensional target surface contour of the target object.

[0182] The third execution submodule is used to extract the three-dimensional feature lines of the three-dimensional model based on the three-dimensional target surface contour.

[0183] Furthermore, in another possible implementation of Embodiment 4 of this application, the aforementioned preset projection image is formed by projecting the surface of the three-dimensional model facing a preset direction onto a preset plane using a virtual camera. The aforementioned second execution submodule may specifically include the following units:

[0184] The first generation unit is used to generate multiple rays based on the optical center corresponding to the virtual camera and each point included in the two-dimensional image contour.

[0185] The second execution unit is used to perform collision detection on each ray and the surface patch of the 3D model to identify multiple target surface patches.

[0186] The third execution unit is used to construct a three-dimensional target surface contour using multiple target surface patches.

[0187] Furthermore, in another possible implementation of Embodiment 4 of this application, the above-mentioned three-dimensional feature lines include three-dimensional target contour lines and three-dimensional visible ridge lines, or the three-dimensional feature lines include three-dimensional target contour lines, three-dimensional visible ridge lines and three-dimensional fixed feature lines. The three-dimensional target contour lines change with the pose of the three-dimensional model, while the three-dimensional visible ridge lines and three-dimensional fixed feature lines do not change with the pose of the three-dimensional model. The three-dimensional target contour lines and the three-dimensional visible ridge lines together constitute the boundary lines of the three-dimensional target surface contour.

[0188] When the three-dimensional feature lines include the three-dimensional target contour lines and the three-dimensional visible ridge lines, the two-dimensional feature lines corresponding to the three-dimensional feature lines in the two-dimensional image include the two-dimensional target contour lines and the two-dimensional visible ridge lines. When the three-dimensional feature lines include the three-dimensional target contour lines, the three-dimensional visible ridge lines and the three-dimensional fixed feature lines, the two-dimensional feature lines include the two-dimensional target contour lines, the two-dimensional visible ridge lines and the two-dimensional fixed feature lines. The two-dimensional target contour lines and the two-dimensional visible ridge lines together constitute the boundary lines of the target objects in the two-dimensional image.

[0189] The 3D visible ridge is the intersection of the 3D target surface contour of the target object and the complete ridge of the 3D model. The 3D target contour is the 3D target surface contour without the 3D visible ridge.

[0190] Furthermore, in yet another possible implementation of Embodiment 4 of this application, the registration device may further include the following modules:

[0191] The first acquisition module is used to detect the first editing command input by the user through the visual editing interface and acquire the first line segment drawing information carried in the first editing command.

[0192] The first generation module is used to generate three-dimensional fixed feature lines on the three-dimensional model in response to the first line segment drawing information in the first editing instruction.

[0193] Furthermore, in another possible implementation of Embodiment 4 of this application, the registration device may further include the following modules:

[0194] The second acquisition module is used to acquire a three-dimensional reference model, wherein the three-dimensional reference model includes three-dimensional reference feature lines.

[0195] The first execution module is used to map the three-dimensional reference feature lines onto the three-dimensional model by utilizing the mapping relationship between the three-dimensional model and the three-dimensional reference model, thereby obtaining three-dimensional fixed feature lines.

[0196] Furthermore, in another possible implementation of Embodiment 4 of this application, the registration device may further include the following modules:

[0197] The first detection module is used to detect the second editing command input by the user through the visual editing interface, which carries the second line segment drawing information.

[0198] The second generation module is used to generate at least one of the following on a two-dimensional image: a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line, in response to the second line segment drawing information in the second editing instruction.

[0199] Furthermore, in yet another possible implementation of Embodiment 4 of this application, the registration device may further include the following modules:

[0200] The second execution module is used to extract multiple candidate feature lines from the two-dimensional image.

[0201] The second detection module is used to detect the third editing command input by the user through the visual editing interface, which carries the start and end point information of the line segment.

[0202] The third generation module is used to generate at least one of the following in response to the start and end point information of the line segment in the third editing instruction: a two-dimensional target contour line, a two-dimensional visible ridge line, and a two-dimensional fixed feature line.

[0203] Furthermore, in another possible implementation of Embodiment 4 of this application, the registration device may further include the following modules:

[0204] The fourth generation module is used to call a pre-trained deep learning model to perform line segment extraction processing on the two-dimensional image to generate at least one of two-dimensional target contour lines, two-dimensional visible ridge lines, and two-dimensional fixed feature lines.

[0205] Furthermore, in another possible implementation of Embodiment 4 of this application, the registration device may further include the following modules:

[0206] The first determining module is used to determine the surface of the 3D model facing a preset direction based on the adjacency matrix of the surface patches of the 3D model and using the region growing method.

[0207] Furthermore, in yet another possible implementation of Embodiment 4 of this application, the first determining module may specifically include the following sub-modules:

[0208] The first determination submodule is used to intersect the preset local coordinate axis corresponding to the 3D model with the 3D model to determine the seed point of the surface of the 3D model facing the preset direction, wherein the positive direction of the preset local coordinate axis faces the preset direction.

[0209] The second determination submodule is used to expand outward from the seed point based on the adjacency matrix until the expansion stops at the complete ridge of the 3D model, so as to determine the surface of the 3D model facing the preset direction.

[0210] Furthermore, in another possible implementation of Embodiment 4 of this application, the first determining module may further include the following sub-modules:

[0211] The third determination submodule is used to determine the complete ridge line of the 3D model based on the normal angle of adjacent facets on the 3D model.

[0212] Furthermore, in another possible implementation of Embodiment 4 of this application, the aforementioned preset projection image is formed by projecting the surface of the three-dimensional model facing a preset direction onto a preset plane using a virtual camera. The aforementioned registration device may further include the following modules:

[0213] The third execution module is used to render the surface of the 3D model facing a preset direction using a virtual camera, and obtain a preset projection image.

[0214] Furthermore, in yet another possible implementation of Embodiment 4 of this application, the contour extraction module 1101 may specifically include the following sub-modules:

[0215] The fourth determination submodule is used to perform binarization processing on the preset projection image in order to determine the target area containing the target object in the preset projection image.

[0216] The fourth execution submodule is used to narrow down the target area and obtain a two-dimensional image contour.

[0217] Furthermore, in another possible implementation of Embodiment 4 of this application, the aforementioned fourth execution submodule may specifically include the following units:

[0218] The fourth execution unit is used to perform morphological erosion operations on the target region to reduce the target region and obtain a two-dimensional image contour.

[0219] The registration apparatus for two-dimensional images and three-dimensional models disclosed in the above embodiments of this application achieves accurate registration of two-dimensional images and three-dimensional models by using three-dimensional feature lines on the three-dimensional model to perform two-dimensional-three-dimensional line matching. Furthermore, by iteratively registering the two-dimensional image and the three-dimensional model, a target registration matrix is ​​constructed. Based on the target registration matrix, fast, real-time, and accurate registration of two-dimensional images and three-dimensional models can be achieved.

[0220] The registration device for two-dimensional images and three-dimensional models provided in this application embodiment can be applied in the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0221] Figure 12 This is a schematic diagram of the terminal device provided in Embodiment 5 of this application. Figure 12As shown, the terminal device 1200 of this embodiment includes: at least one processor 1210 ( Figure 12 The diagram shows only one processor, a memory 1220, and a computer program 1221 stored in the memory 1220 and executable on the at least one processor 1210. When the processor 1210 executes the computer program 1221, it implements the steps in the above-described embodiments of the registration method for two-dimensional images and three-dimensional models.

[0222] The terminal device 1200 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This terminal device may include, but is not limited to, a processor 1210 and a memory 1220. Those skilled in the art will understand that... Figure 12 This is merely an example of terminal device 1200 and does not constitute a limitation on terminal device 1200. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0223] The processor 1210 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0224] In some embodiments, the memory 1220 may be an internal storage unit of the terminal device 1200, such as a hard disk or memory of the terminal device 1200. In other embodiments, the memory 1220 may be an external storage device of the terminal device 1200, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 1200. Furthermore, the memory 1220 may include both internal and external storage units of the terminal device 1200. The memory 1220 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 1220 can also be used to temporarily store data that has been output or will be output.

[0225] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0226] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0227] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0228] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0229] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0230] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0231] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0232] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the various method embodiments described above.

[0233] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for registering a two-dimensional image with a three-dimensional model, characterized in that, include: Extract the two-dimensional image contour of the preset projection image, wherein the preset projection image is formed by projecting the target surface of the three-dimensional model of the target object onto a preset plane through a virtual camera, the target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction; Light is projected onto the contour of the two-dimensional image to extract the three-dimensional feature lines of the three-dimensional model; wherein, the three-dimensional feature lines are lines in the three-dimensional model that characterize the surface features of the target object; Based on the three-dimensional feature lines, the two-dimensional image containing the target object is registered with the three-dimensional model; The step of casting light onto the two-dimensional image contour to extract the three-dimensional feature lines of the three-dimensional model includes: casting light onto the two-dimensional image contour to obtain the three-dimensional target surface contour of the target object; and extracting the three-dimensional feature lines of the three-dimensional model based on the three-dimensional target surface contour. The step of projecting light onto the two-dimensional image contour to obtain the three-dimensional target surface contour of the target object includes: generating multiple light rays based on the optical center corresponding to the virtual camera and each point included in the two-dimensional image contour; performing collision detection on each light ray and the surface patch of the three-dimensional model to determine multiple target surface patches; and using the multiple target surface patches to construct the three-dimensional target surface contour.

2. The registration method for two-dimensional images and three-dimensional models as described in claim 1, characterized in that, The registration of the two-dimensional image containing the target object with the three-dimensional model based on the three-dimensional feature lines includes: Based on the three-dimensional feature lines and the two-dimensional feature lines in the two-dimensional image corresponding to the three-dimensional feature lines, the two-dimensional image and the three-dimensional model are registered.

3. The registration method for two-dimensional images and three-dimensional models as described in claim 2, characterized in that, The registration result between the two-dimensional image and the three-dimensional model is a target registration matrix, which represents the mapping relationship between the two-dimensional image and the three-dimensional model. The registration of the two-dimensional image and the three-dimensional model based on the three-dimensional feature lines and the corresponding two-dimensional feature lines in the two-dimensional image includes: Based on the three-dimensional feature lines and the two-dimensional feature lines, the two-dimensional image and the three-dimensional model are iteratively registered to generate a reference registration matrix until the reference registration matrix meets the preset convergence condition. Then, the reference registration matrix is ​​determined as the target registration matrix.

4. The registration method for two-dimensional images and three-dimensional models as described in claim 3, characterized in that, The step of iteratively registering the two-dimensional image and the three-dimensional model based on the three-dimensional feature lines and the two-dimensional feature lines to generate a reference registration matrix, until the reference registration matrix satisfies a preset convergence condition, and then determining the reference registration matrix as the target registration matrix, includes: Determine a two-dimensional projection line on the preset projection image that corresponds to the three-dimensional feature line; Between the two-dimensional projection line and the two-dimensional feature line, multiple sets of two-dimensional matching point pairs are determined; Based on the multiple sets of two-dimensional matching point pairs, determine multiple sets of two-dimensional matching point pairs between the three-dimensional feature line and the two-dimensional feature line; Registration calculations are performed on the multiple sets of two-dimensional-three-dimensional matching point pairs to generate the reference registration matrix; When the reference registration matrix does not meet the preset convergence condition, the pose of the three-dimensional model is changed using the reference registration matrix, the preset projection image is re-determined, and the above iterative registration steps are repeated to iteratively register the two-dimensional image and the three-dimensional model. When the reference registration matrix satisfies the preset convergence condition, the reference registration matrix is ​​determined as the target registration matrix.

5. The registration method for two-dimensional images and three-dimensional models as described in claim 4, characterized in that, The step of determining multiple pairs of two-dimensional matching points between the two-dimensional projection line and the two-dimensional feature line includes: Based on multiple first starting points on the two-dimensional projection line, multiple first matching points corresponding to the two-dimensional feature line are determined, wherein the first starting point is any point on the two-dimensional projection line, and the first starting point is matched one by one with the first matching point. Based on multiple second starting points on the two-dimensional feature line, multiple second matching points corresponding to the two-dimensional projection line are determined, wherein the second starting point is any point on the two-dimensional feature line, and the second starting point is matched one by one with the second matching point; Based on multiple first starting points, multiple first matching points, multiple second starting points, and multiple second matching points, the multiple sets of two-dimensional-two-dimensional matching point pairs are determined.

6. The registration method for two-dimensional images and three-dimensional models as described in claim 4, characterized in that, The step of performing registration calculations on the multiple sets of two-dimensional-three-dimensional matching point pairs to generate the reference registration matrix includes: The Random Sampling Consistency-Perspective Point Problem (RANSAC-PnP) algorithm is used to perform registration calculations on the multiple sets of two-dimensional-three-dimensional matching point pairs to generate the reference registration matrix.

7. The registration method for two-dimensional images and three-dimensional models as described in any one of claims 1-6, characterized in that, The three-dimensional feature lines include three-dimensional target contour lines and three-dimensional visible ridge lines, or the three-dimensional feature lines include three-dimensional target contour lines, three-dimensional visible ridge lines and three-dimensional fixed feature lines. The three-dimensional target contour lines change with the pose of the three-dimensional model, while the three-dimensional visible ridge lines and three-dimensional fixed feature lines do not change with the pose of the three-dimensional model. The three-dimensional target contour lines and the three-dimensional visible ridge lines together constitute the boundary lines of the three-dimensional target surface contour. When the three-dimensional feature line includes a three-dimensional target contour line and a three-dimensional visible ridge line, the two-dimensional feature line in the two-dimensional image corresponding to the three-dimensional feature line includes a two-dimensional target contour line and a two-dimensional visible ridge line. When the three-dimensional feature line includes a three-dimensional target contour line, a three-dimensional visible ridge line and a three-dimensional fixed feature line, the two-dimensional feature line includes a two-dimensional target contour line, a two-dimensional visible ridge line and a two-dimensional fixed feature line. The two-dimensional target contour line and the two-dimensional visible ridge line together constitute the boundary line of the target object in the two-dimensional image. The three-dimensional visible ridge line is the intersection of the three-dimensional target surface contour of the target object and the complete ridge line of the three-dimensional model, and the three-dimensional target contour line is the portion of the three-dimensional target surface contour without the three-dimensional visible ridge line.

8. The registration method for two-dimensional images and three-dimensional models as described in claim 7, characterized in that, The registration method further includes: The system detects the first editing command input by the user through the visual editing interface and obtains the first line segment drawing information carried in the first editing command. In response to the first line segment drawing information in the first editing instruction, the three-dimensional fixed feature line on the three-dimensional model is generated; or, A three-dimensional reference model is obtained, wherein the three-dimensional reference model includes three-dimensional reference feature lines; By utilizing the mapping relationship between the three-dimensional model and the three-dimensional reference model, the three-dimensional reference feature lines are mapped onto the three-dimensional model to obtain the three-dimensional fixed feature lines.

9. The registration method for two-dimensional images and three-dimensional models as described in claim 7, characterized in that, The registration method further includes: The system detects a second editing command input by the user through the visual editing interface, which carries information about drawing a second line segment. In response to the second line segment drawing information in the second editing instruction, at least one of the two-dimensional target contour line, the two-dimensional visible ridge line, and the two-dimensional fixed feature line is generated on the two-dimensional image; or, Extract multiple candidate feature lines from the two-dimensional image; The system detects a third editing command input by the user through a visual editing interface, the third editing command carrying information about the start and end points of a line segment; In response to the start and end point information of the line segment in the third editing instruction, at least one of the two-dimensional target contour line, the two-dimensional visible ridge line, and the two-dimensional fixed feature line is generated; or, A pre-trained deep learning model is invoked to perform line segment extraction processing on the two-dimensional image to generate at least one of the two-dimensional target contour line, the two-dimensional visible ridge line, and the two-dimensional fixed feature line.

10. The registration method for a two-dimensional image and a three-dimensional model as described in claim 1, characterized in that, Before extracting the two-dimensional image contour of the preset projection image, the registration method further includes: Based on the adjacency matrix of the surface of the 3D model, and using the region growing method, the surface of the 3D model facing the preset direction is determined.

11. The registration method for a two-dimensional image and a three-dimensional model as described in claim 10, characterized in that, The step of determining the surface of the 3D model facing the preset direction based on the adjacency matrix of the surface patches of the 3D model and using the region growing method includes: The intersection of the preset local coordinate axis corresponding to the 3D model with the 3D model is used to determine the seed point of the surface of the 3D model facing the preset direction, wherein the positive direction of the preset local coordinate axis faces the preset direction; Based on the adjacency matrix, expansion is initiated from the seed point and continues outward until the complete ridge of the 3D model is reached, thereby determining the surface of the 3D model facing the preset direction.

12. The registration method for two-dimensional images and three-dimensional models as described in claim 11, characterized in that, The registration method, before expanding outward from the seed point based on the adjacency matrix until the expansion stops at the complete ridge line of the 3D model to determine the surface of the 3D model facing the preset direction, further includes: The complete ridge line of the 3D model is determined based on the normal angles of adjacent faces on the 3D model.

13. The registration method for two-dimensional images and three-dimensional models as described in claim 1, characterized in that, The preset projection image is formed by projecting the surface of the 3D model facing the preset direction onto the preset plane using a virtual camera. Before extracting the 2D image contour of the preset projection image, the registration method further includes: Using the virtual camera, the surface of the 3D model facing the preset direction is rendered to obtain the preset projection image.

14. The registration method for two-dimensional images and three-dimensional models as described in any one of claims 1-6 and 10-13, characterized in that, The extraction of the two-dimensional image contour of the preset projection image includes: The preset projection image is binarized to determine the target region containing the target object in the preset projection image; The target area is reduced to obtain the two-dimensional image contour.

15. The registration method for a two-dimensional image and a three-dimensional model as described in claim 14, characterized in that, The process of shrinking the target region to obtain the two-dimensional image contour includes: A morphological erosion operation is performed on the target region to reduce the size of the target region, thereby obtaining the two-dimensional image contour.

16. A registration device for a two-dimensional image and a three-dimensional model, characterized in that, include: The contour extraction module is used to extract the two-dimensional image contour of a preset projection image, wherein the preset projection image is formed by projecting the target surface of the three-dimensional model of the target object onto a preset plane through a virtual camera, the target surface of the three-dimensional model is the surface of the three-dimensional model facing a preset direction, and the preset plane is perpendicular to the preset direction. A ray projection module is used to project rays onto the contour of the two-dimensional image to extract the three-dimensional feature lines of the three-dimensional model; wherein, the three-dimensional feature lines are lines in the three-dimensional model that characterize the surface features of the target object; The registration module is used to register the two-dimensional image containing the target object with the three-dimensional model based on the three-dimensional feature lines. The ray casting module includes the following sub-modules: a second execution sub-module, used to cast rays onto the two-dimensional image contour to obtain the three-dimensional target surface contour of the target object; and a third execution sub-module, used to extract the three-dimensional feature lines of the three-dimensional model based on the three-dimensional target surface contour. The second execution submodule includes the following units: a first generation unit, used to generate multiple rays based on the optical center corresponding to the virtual camera and each point included in the two-dimensional image contour; a second execution unit, used to perform collision detection on each ray and the surface patch of the three-dimensional model to determine multiple target surface patches; and a third execution unit, used to construct a three-dimensional target surface contour using the multiple target surface patches.

17. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 15.

18. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 15.

Images