Three-dimensional model and two-dimensional picture mapping method and device, storage medium and electronic equipment

Abstract

The invention relates to a three-dimensional model and two-dimensional picture mapping method and device, a storage medium and electronic equipment. The method comprises the steps of obtaining virtual camera shooting parameters when a two-dimensional picture is generated through a virtual camera in a three-dimensional space; obtaining a projection matrix according to the camera external parameters and the camera internal parameters of the virtual camera in the projection mode, determining a direction vector coordinate system according to the camera external parameters, and storing the shooting parameters of the virtual camera, the projection matrix and the direction vector coordinate system; for each two-dimensional pixel point of the edited two-dimensional picture, converting a corresponding two-dimensional pixel coordinate into a projection coordinate, constructing a ray equation and calculating an intersection point of the ray equation and the surface of the three-dimensional model based on the virtual camera shooting parameter and the projection coordinate to obtain a three-dimensional coordinate point; and constructing a three-dimensional picture in the three-dimensional model according to each three-dimensional coordinate point, and mapping the labeling attribute of the two-dimensional pixel point in the edited two-dimensional picture to the three-dimensional coordinate point corresponding to the two-dimensional pixel point. The efficiency of cooperative communication can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, storage medium, and electronic device for mapping three-dimensional models to two-dimensional images. Background Technology

[0002] In fields such as architectural design, mechanical design, construction management, virtual simulation, and digital twins, 3D images, due to their three-dimensional spatial structure, have become a core means of communication. However, in engineering communication and information transmission, the performance of the terminal devices of the communicating parties differs. For example, one party's terminal device supports 3D images, while the other's only supports 2D images. Therefore, one party uses 2D images for annotation and interacts with the 3D images displayed by the other party to achieve engineering communication and information transmission, which remains an important means of exchange. In related technologies, 3D models can be used to generate 2D images by setting viewing angles and camera parameters and transmit them to the other party. However, for the other party's editing on the 2D image, such as selecting regions of interest for annotation, the lack of a calculation mechanism to back-map 2D markers to 3D space makes it difficult to accurately map the edited 2D image to 3D space. This results in a lack of consistency between 2D markers and 3D images, and the annotations on the 2D image cannot establish a precise correspondence with the 3D image, making the accuracy and efficiency of collaborative communication based on marked 2D and 3D images low. Furthermore, since it is based on a specific coordinate system or viewpoint by setting the observation angle and camera parameters, it cannot adapt to arbitrary angles and different coordinate systems (left-handed or right-handed coordinate systems), and its versatility is poor. Summary of the Invention

[0003] In view of this, the present invention provides a method, apparatus, storage medium, and electronic device for mapping three-dimensional models to two-dimensional images.

[0004] Specifically, the present invention is achieved through the following technical solution: According to a first aspect of the present invention, a method for mapping a three-dimensional model to a two-dimensional image is provided, the method comprising: The virtual camera shooting parameters are obtained when generating a two-dimensional image in three-dimensional space using a virtual camera. The virtual camera shooting parameters include: external camera parameters, internal camera parameters, and projection mode. Based on the camera extrinsic and intra-camera parameters of the virtual camera in projection mode, obtain the projection matrix in that projection mode; and based on the camera extrinsic parameters of the virtual camera in projection mode, determine the direction vector coordinate system of the virtual camera, and store the virtual camera shooting parameters, projection matrix, and direction vector coordinate system. For each two-dimensional pixel in the edited two-dimensional image, the two-dimensional pixel coordinates corresponding to the two-dimensional pixel are converted into projected coordinates. Based on the shooting parameters of the virtual camera and the projected coordinates, a ray equation is constructed and the intersection of the ray equation with the surface of the three-dimensional model is calculated to obtain the three-dimensional coordinate points. Based on the three-dimensional coordinates corresponding to each two-dimensional pixel in the edited two-dimensional image, a three-dimensional image in the three-dimensional model is constructed, and the annotation attributes of the two-dimensional pixels in the edited two-dimensional image are mapped to the three-dimensional coordinates corresponding to the two-dimensional pixels.

[0005] Optionally, obtaining the projection matrix in the projection mode based on the extrinsic and intrinsic parameters of the virtual camera in the projection mode includes: A virtual camera intrinsic parameter matrix is ​​constructed based on the camera intrinsic parameters, including focal length, principal point offset, sensor size, and resolution. Based on the orientation vector, upward vector, and virtual camera position parameters in the camera extrinsic parameters, construct the virtual camera extrinsic parameter matrix; Based on the virtual camera intrinsic parameter matrix and the virtual camera extrinsic parameter matrix, obtain the projection matrix in the projection mode of the virtual camera.

[0006] Optionally, generating a two-dimensional image via a virtual camera includes: Using the projection matrix, each three-dimensional point in the three-dimensional image displayed in three-dimensional space is projected and transformed to obtain the projected coordinates of the three-dimensional point. The projected coordinates include: a projection horizontal coordinate component, a projection vertical coordinate component, and a projection depth component. Obtain two-dimensional screen coordinates based on projected coordinates; A two-dimensional image is generated based on the two-dimensional screen coordinates corresponding to each three-dimensional point in the three-dimensional image, and the projection depth component corresponding to the two-dimensional image is stored.

[0007] Optionally, the step of converting the two-dimensional pixel coordinates corresponding to the two-dimensional pixel point into projected coordinates, and constructing a ray equation based on the virtual camera shooting parameters and the projected coordinates, includes: Based on the sensor size and resolution in the camera's intrinsic parameters and the virtual camera intrinsic parameter matrix, the two-dimensional pixel coordinates corresponding to the two-dimensional pixels are normalized into projected coordinates; Determine the ray direction vector based on the projected coordinates and the rotation matrix in the projection matrix; A ray equation is constructed based on the ray direction vector and the virtual camera position parameter information corresponding to the stored two-dimensional image.

[0008] Optionally, the calculation of the intersection of the ray equation and the surface of the three-dimensional model to obtain the three-dimensional coordinate points includes: Using a pre-set ray detection algorithm, the intersection points of the ray equation and the surface of the 3D image in the 3D spatial model corresponding to the 2D image are obtained.

[0009] Optionally, the step of using a pre-set ray detection algorithm to obtain the intersection points of the ray equation and the surface of the three-dimensional image in the three-dimensional spatial model corresponding to the two-dimensional image includes: Triangular facets are set on the surface of a 3D image along the ray direction vector, and the edge vectors of the triangular facets are calculated, including a first edge vector and a second edge vector. Calculate the first vector based on the first side vector and the stored virtual camera's direction vector coordinate system; Calculate the second vector based on the second side vector and the first vector; Determine that the second vector is not zero, and obtain the first comparison value based on the second vector, the first vector, the common vertices of the second vector and the first vector in the triangular facet, and the virtual camera position parameters; The first comparison value is determined to be greater than zero and less than 1. The second comparison value is obtained based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, and the stored virtual camera direction vector coordinate system. Determine that the second comparison value is not less than 0 and the sum of the first comparison value and the second comparison value is not greater than 1. Based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, the stored virtual camera direction vector coordinate system, and the second vector, obtain the third comparison value. If the third comparison value is determined to be greater than 0, the third comparison value is used as a real constant of the ray equation. Based on the ray equation with the assigned real constant, the intersection point of the ray equation and the surface of the three-dimensional image in the three-dimensional spatial model corresponding to the two-dimensional image is obtained.

[0010] Optionally, the method further includes: If the second vector is equal to zero, obtain the three-dimensional model space of the three-dimensional image corresponding to the two-dimensional image, and divide the three-dimensional model space into an octree hierarchical structure, wherein the octree hierarchical structure includes a root node and leaf nodes. For each node, record the list of triangular faces contained in that node, where the node includes the root node and leaf nodes; Starting from the root node, traverse layer by layer, performing intersection calculations only on nodes traversed by the ray equation, and discarding non-intersecting nodes; For the triangular facets in the leaf node, the Möller–Trumbore algorithm is used to calculate the intersection points of the ray equation with each triangular facet in the leaf node, and the real constants in the ray equation are obtained. From all the positive real constants, select the intersection point corresponding to the smallest positive real constant to obtain the three-dimensional coordinate point.

[0011] The 3D model-to-2D image mapping method in this technical solution obtains the virtual camera shooting parameters when generating a 2D image using a virtual camera in 3D space. These virtual camera shooting parameters include: camera extrinsic parameters, camera intrinsic parameters, and projection mode. Based on the camera extrinsic and intrinsic parameters in projection mode, the projection matrix in that projection mode is obtained. Furthermore, based on the camera extrinsic parameters in projection mode, the direction vector coordinate system of the virtual camera is determined, and the virtual camera shooting parameters, projection matrix, and direction vector coordinate system are stored. For each 2D pixel in the edited 2D image, the corresponding 2D pixel coordinates are converted to projection coordinates. Based on the virtual camera shooting parameters and projection coordinates, a ray equation is constructed, and the intersection of the ray equation with the surface of the 3D model is calculated to obtain the 3D coordinate point. Based on the 3D coordinate points corresponding to each 2D pixel in the edited 2D image, a 3D image in the 3D model is constructed, and the annotation attributes of the 2D pixels in the edited 2D image are mapped to the corresponding 3D coordinate points. In this way, by recording relevant information about the camera capturing and rendering the 2D image when generating it from 3D space, the edited 2D image can be back-converted into 3D space based on the recorded information through ray detection and matrix operations. This maintains consistency between the displayed image in 2D and 3D spaces, enabling lossless reproduction of edits made in 2D images in 3D scenes, effectively improving communication and collaboration efficiency.

[0012] According to a second aspect of the present invention, a three-dimensional model to two-dimensional image mapping device is provided, the three-dimensional model to two-dimensional image mapping device comprising: The shooting parameter acquisition module is used to acquire the virtual camera shooting parameters when generating a two-dimensional image through a virtual camera in three-dimensional space. The virtual camera shooting parameters include: external camera parameters, internal camera parameters, and projection mode. The parameter processing and storage module is used to obtain the projection matrix in the projection mode based on the camera's external and internal parameters in the projection mode, and to determine the direction vector coordinate system of the virtual camera based on the camera's external parameters in the projection mode, and to store the virtual camera's shooting parameters, projection matrix and direction vector coordinate system. The coordinate point mapping module is used to convert the two-dimensional pixel coordinates corresponding to each two-dimensional pixel in the edited two-dimensional image into projected coordinates. Based on the shooting parameters of the virtual camera and the projected coordinates, it constructs a ray equation and calculates the intersection of the ray equation with the surface of the three-dimensional model to obtain the three-dimensional coordinate point. The 3D image reconstruction module is used to construct a 3D image in a 3D model based on the 3D coordinates corresponding to each 2D pixel in the edited 2D image, and to map the annotation attributes of the 2D pixels in the edited 2D image to the corresponding 3D coordinates.

[0013] According to a third aspect of the present invention, a storage medium is provided having a computer program stored thereon, wherein when the program is executed by a processor, it implements the steps of the three-dimensional model-to-two-dimensional image mapping method in any possible implementation of the first aspect.

[0014] According to a fourth aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the three-dimensional model-to-two-dimensional image mapping method in any possible implementation of the first aspect. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for mapping a three-dimensional model to a two-dimensional image, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a process for establishing a mapping relationship in a method for mapping three-dimensional models and two-dimensional images provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating a process of mapping a three-dimensional model to a two-dimensional image in a method for mapping two-dimensional images according to an embodiment of the present invention; Figure 4 A schematic diagram of a cloud-based method for mapping three-dimensional models to two-dimensional images is provided in an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating an application scenario of a three-dimensional model-to-two-dimensional image mapping method provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of a three-dimensional model and two-dimensional image mapping device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In related technologies, methods that utilize 3D models to generate 2D images by setting viewing angles and camera parameters for engineering communication and information transfer suffer from several drawbacks. After one party edits and annotates the 2D image, the lack of a mechanism to reverse-map the 2D markers to 3D space prevents a precise correspondence between the annotations and the 3D image. This leads to inconsistencies between the 2D markers and the 3D image, resulting in low accuracy and efficiency in collaborative communication between 2D and 3D images based on annotations. Furthermore, the method of generating 2D images by setting viewing angles and camera parameters lacks versatility. Moreover, the annotation information on 2D images is often simple, incomplete, and lacks a unified storage format, making it difficult to include complete information such as marker type, number, resolution, and camera parameters, further reducing the efficiency of subsequent collaborative communication.

[0020] The 3D model-to-2D image mapping method proposed in this embodiment is used to achieve bidirectional mapping between 3D and 2D images. By recording camera shooting information and mapping matrix when generating 2D images based on 3D space, and then using the recorded information, ray detection and matrix operations are used to back-engineer the 2D image back to 3D space. This establishes a strict mathematical mapping relationship between 2D and 3D spaces, enabling lossless reproduction of editing in 2D images in 3D scenes. It supports arbitrary angles and coordinate systems, and through ray detection and camera conversion, it can accurately back-engineer 2D image markers to 3D space, thereby improving collaboration efficiency and achieving efficient engineering collaboration and information alignment.

[0021] See Figure 1 This invention provides a method for mapping three-dimensional models to two-dimensional images, which may include the following steps: S101. Obtain the virtual camera shooting parameters when generating a two-dimensional image in three-dimensional space using a virtual camera. The virtual camera shooting parameters include: external camera parameters, internal camera parameters, and projection mode. In this embodiment, a virtual camera model is constructed in three-dimensional space, and virtual camera shooting parameters are set for the virtual camera model. These virtual camera shooting parameters include, but are not limited to: Camera extrinsic parameters, including but not limited to: virtual camera position parameter C=(C x C y C z Orientation vector Above vector ; Camera parameters, including but not limited to: focal length f, principal point offset (c x , c y ), sensor size (w, h), resolution (W, H); Projection modes, including but not limited to: perspective projection or orthographic projection.

[0022] S102. Based on the camera extrinsic parameters and camera intrinsic parameters of the virtual camera in projection mode, obtain the projection matrix in the projection mode, and based on the camera extrinsic parameters of the virtual camera in projection mode, determine the direction vector coordinate system of the virtual camera, and store the virtual camera shooting parameters, projection matrix and direction vector coordinate system. In this embodiment, as an optional embodiment, the projection matrix in the projection mode is obtained based on the camera's extrinsic and intra-camera parameters in the projection mode, including: A virtual camera intrinsic parameter matrix is ​​constructed based on the camera intrinsic parameters, including focal length, principal point offset, sensor size, and resolution. Based on the orientation vector, upward vector, and virtual camera position parameters in the camera extrinsic parameters, construct the virtual camera extrinsic parameter matrix; Based on the virtual camera intrinsic parameter matrix and the virtual camera extrinsic parameter matrix, obtain the projection matrix in the projection mode of the virtual camera.

[0023] In this embodiment, the sensor size (w, h) and resolution (W, H) are used to determine the proportional relationship between pixel coordinates and actual spatial coordinates. As an optional embodiment, the proportional relationship is as follows:

[0024]

[0025] In the formula, , These are the pixel focal lengths in the two-dimensional direction, respectively. , These are the principal point coordinates in the two-dimensional direction, i.e., the principal point offset.

[0026] Camera extrinsic parameters, including but not limited to: virtual camera position parameter C=(C x C y C z Orientation vector Above vector ; Camera parameters, including but not limited to: focal length f, principal point offset (c x , c y ), sensor size (w, h), resolution (W, H); In this embodiment, as an optional implementation, the projection matrix in the projection mode is constructed using the following formula: P = K[R∣T] in: P is the projection matrix; K is the virtual camera intrinsic parameter matrix, represented as follows:

[0027] [R|T] is the virtual camera extrinsic parameter matrix composed of the camera extrinsic parameters of the virtual camera, where R is the rotation matrix and T is the translation matrix.

[0028] The rotation matrix is ​​represented as follows:

[0029] In the formula, For the orientation vector, The vector above, The right-direction vector is obtained by cross product of the direction vector and the upward vector.

[0030] In this embodiment, the rotation matrix is ​​used to represent the orientation of the virtual camera coordinate system relative to the world coordinate system, and is related to the virtual camera's orientation vector. and the vector above Directly related.

[0031] The translation matrix is ​​used to represent the positional relationship between the virtual camera's origin and the world's origin. It is a translation vector that changes accordingly when the virtual camera moves to different positions.

[0032] In this embodiment, as an optional implementation, the translation vector is obtained using the following formula: T = -R×C In this embodiment, as an optional implementation, the right direction vector of the virtual camera is obtained using the following formula:

[0033] In this embodiment, as an optional implementation, generating a two-dimensional image using a virtual camera includes: Using the projection matrix, each three-dimensional point in the three-dimensional image displayed in three-dimensional space is projected and transformed to obtain the projected coordinates of the three-dimensional point. The projected coordinates include: a projection horizontal coordinate component, a projection vertical coordinate component, and a projection depth component. Obtain two-dimensional screen coordinates based on projected coordinates; A two-dimensional image is generated based on the two-dimensional screen coordinates corresponding to each three-dimensional point in the three-dimensional image, and the projection depth component corresponding to the two-dimensional image is stored.

[0034] In this embodiment, based on 3D points in a 3D image, the 2D screen coordinates projected onto the 3D point are obtained using a projection matrix. As an optional embodiment, for any 3D point (X, Y, Z, 1), the projected coordinates are calculated using the following formula through projection matrix operations:

[0035] In the formula, u′, v′, and w′ are the projected coordinates of the 3D point in the virtual camera coordinate system after transformation by the virtual camera extrinsic matrix (rotation matrix R and translation vector T) and the virtual camera intrinsic matrix, respectively; that is, the projected horizontal coordinate component, the projected vertical coordinate component, and the projected depth component. The projected depth component is the scale factor of the depth or homogeneous coordinates, used to implement perspective division, and u′ and v′ are unnormalized pixel coordinates.

[0036] In this embodiment, as an optional embodiment, obtaining two-dimensional screen coordinates based on projected coordinates includes: Calculate the ratio of the projection horizontal coordinate component to the projection depth component in the projection coordinates to obtain the horizontal coordinate in the two-dimensional screen coordinates. Calculate the ratio of the projection vertical coordinate component to the projection depth component in the projection coordinates to obtain the vertical coordinate in the two-dimensional screen coordinates.

[0037] In this embodiment, the two-dimensional screen coordinates are obtained using the following formula: ,

[0038] u and v are the horizontal and vertical coordinates of the two-dimensional screen coordinates, respectively.

[0039] In this embodiment, a two-dimensional image is generated based on the projection mode.

[0040] In this embodiment, a virtual camera can be used to obtain rendered 2D image files and virtual camera shooting parameter files from 3D images in 3D space. The rendered 2D image file formats include, but are not limited to, PNG and JPG formats; this embodiment does not impose any limitations on these formats.

[0041] In this embodiment, as an optional implementation, the stored information includes: Virtual camera shooting parameter information (virtual camera shooting parameters) includes: Camera extrinsic parameters: camera position parameters, rotation matrix (orientation vector, upward vector), translation vector (obtained based on rotation matrix and camera position parameters); Camera parameters: focal length, principal point offset, sensor size, resolution; Projection mode (perspective projection / orthographic projection).

[0042] The projection matrix and direction vector coordinate system information (direction vector coordinate system) include: The projection matrix P from three dimensions to two dimensions; Inverse projection matrix from 2D to 3D ; Ensure cross-platform consistency of the direction vector coordinate system (left-handed or right-handed).

[0043] In this embodiment, as an optional implementation, relevant auxiliary information may also be stored, including but not limited to: Virtual camera type description information: for example, perspective camera, orthographic camera, panoramic camera, etc.; Two-dimensional image number, timestamp, and other identifying information; Users tag metadata such as points, lines, polygons, symbols, text descriptions, numbers, etc.

[0044] In this embodiment, each 3D image corresponds to one or more 2D images, and 3D images of the same structure from different viewpoints are considered as multiple 3D images. As an optional embodiment, JSON / XML format is used for storage to facilitate parsing and cross-platform access.

[0045] S103. For each two-dimensional pixel in the edited two-dimensional image, convert the two-dimensional pixel coordinates corresponding to the two-dimensional pixel into projection coordinates. Based on the shooting parameters of the virtual camera and the projection coordinates, construct the ray equation and calculate the intersection point of the ray equation with the surface of the three-dimensional model to obtain the three-dimensional coordinate point. In this embodiment, the user can add annotations to edit the 2D image, such as selecting or drawing line segments or regions, or setting text. As an optional embodiment, the 2D pixel coordinates corresponding to the 2D pixel point are converted into projected coordinates. Based on the virtual camera shooting parameters and the projected coordinates, a ray equation is constructed, including: A11 normalizes the two-dimensional pixel coordinates corresponding to two-dimensional pixels into projected coordinates based on the sensor size and resolution in the camera's intrinsic parameters and the virtual camera intrinsic parameter matrix. In this embodiment, as an optional implementation, the two-dimensional pixel coordinates (two-dimensional screen coordinates) of the target two-dimensional pixel point in the two-dimensional image are converted into projection coordinates, i.e., the virtual camera normalized coordinates (x′, y′), using the following formula:

[0046] A12, determine the ray direction vector based on the projected coordinates and the rotation matrix in the projection matrix; In this embodiment, the inverse matrix of the rotation matrix is ​​obtained based on the rotation matrix corresponding to the camera extrinsic parameters in the virtual camera shooting parameters. Then, based on the inverse matrix and the normalized coordinates of the virtual camera, the ray direction vector is calculated using the following formula:

[0047] A13. Based on the ray direction vector and the virtual camera position parameter information corresponding to the stored two-dimensional image, construct the ray equation.

[0048]

[0049] in, L(t) is a ray point on the ray equation, C = (C x C y C z ), representing the position of the virtual camera in the world coordinate system, i.e., the virtual camera position parameter information, and t is a real constant greater than 0, used to characterize the distance traveled from the origin of the virtual camera along the ray direction.

[0050] In this embodiment, as t increases, the ray point L(t) moves further away from the camera along the ray direction, thereby intersecting with the geometric surface of the 3D image in the 3D scene.

[0051] In this embodiment, as an optional implementation, the intersection of the ray equation and the surface of the three-dimensional model is calculated to obtain the three-dimensional coordinate points, including: Using a pre-set ray detection algorithm, the intersection points of the ray equation and the surface of the 3D image in the 3D spatial model corresponding to the 2D image are obtained.

[0052] In this embodiment, the process of finding the intersection of the ray equation and the 3D image can be based on the analytical geometry of triangular facets. The intersection point is the 3D coordinate point on the 3D image corresponding to the 2D coordinate point. Therefore, as an optional embodiment, a pre-set ray detection algorithm is used to obtain the intersection point of the ray equation and the 3D image surface in the 3D spatial model corresponding to the 2D image, including: C11, along the ray direction vector, set a triangular facet on the surface of the three-dimensional image, calculate the edge vector of the triangular facet, the edge vector includes a first edge vector and a second edge vector; In this embodiment, triangular facets are set in the 3D image displayed by the 3D model, and the three vertices of the triangular facets are v0, v1, and v2, respectively. In this embodiment, the intersection point of the ray equation and the triangular facet can be directly calculated using the Möller–Trumbore algorithm.

[0053] In this embodiment, the edge vector is calculated using the following formula: e1 = v1 - v0, e2 = v2 - v0; In this embodiment, the first edge vector and the second edge vector are edge vectors based on the same vertex of the triangular facet.

[0054] C12, calculate the first vector based on the first side vector and the stored virtual camera's direction vector coordinate system; In this embodiment, the first vector is calculated using the following formula: ; C13, Calculate the second vector based on the second edge vector and the first vector; In this embodiment, the second vector is calculated using the following formula: a = e2×h C14, determine that the second vector is not zero, and obtain the first comparison value based on the second vector, the first vector, the common vertex of the second vector and the first vector in the triangle patch, and the virtual camera position parameters; In this embodiment, if the second vector a≈0, it means that the ray is parallel to the triangular facet and has no intersection point, then the ray equation is adjusted subsequently. As an optional embodiment, the first comparison value is obtained using the following formula: f=1 / a, s=C v0, u=f(s×h) In the formula, the virtual camera position parameter C = (C x C y C z ), where u is the first comparison value.

[0055] In this embodiment, if u<0 or u>1, then there is no intersection point; C15, determine that the first comparison value is greater than zero and less than 1, and obtain the second comparison value based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector and the stored virtual camera direction vector coordinate system; In this embodiment, as an optional implementation, the second comparison value is calculated using the following formula:

[0056] In the formula, v is the second comparison value.

[0057] If v < 0 or u + v > 1, then there are no intersections.

[0058] C16, determine that the second comparison value is not less than 0 and the sum of the first comparison value and the second comparison value is not greater than 1. Based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, the stored virtual camera direction vector coordinate system and the second vector, obtain the third comparison value; In this embodiment, the third comparison value is calculated using the following formula: t = f(e² × q) C17, determine that the third comparison value is greater than 0, take the third comparison value as the real constant of the ray equation, and obtain the intersection point of the ray equation and the surface of the three-dimensional image in the three-dimensional space model corresponding to the two-dimensional image based on the ray equation with the assigned real constant.

[0059] In this embodiment, the intersection point of the ray equation and the surface of the three-dimensional image in the three-dimensional spatial model corresponding to the two-dimensional image is calculated using the following formula: .

[0060] In this embodiment, as an optional implementation, if multiple intersection points exist, the nearest intersection point (the smallest t) is selected. Thus, by traversing all triangular faces in the 3D model and finding the smallest positive real constant t, the coordinates P (X,Y,Z) of the nearest intersection point with the model surface can be obtained.

[0061] In this embodiment, if there is no intersection in the above processing flow, as an optional embodiment, the method further includes: Based on a depth buffer that records the minimum depth value of each pixel in a 2D image and a pre-set approximate intersection formula, the approximate 3D coordinates of the nearest intersection point between the ray equation and the triangular facet are obtained.

[0062] In this embodiment, a depth buffer containing the minimum depth value of the recorded pixels is used to accelerate intersection calculation.

[0063] In this embodiment, when projecting the two-dimensional marker back into three dimensions, the depth Zdepth(u,v) corresponding to the two-dimensional pixel coordinates (u,v) is extracted, and the approximate three-dimensional coordinates of the nearest intersection point between the ray equation and the triangular facet can be directly obtained, where the depth Zdepth(u,v) is the projection depth component w′. As an optional embodiment, the approximate three-dimensional coordinates are obtained using the following formula:

[0064] In the formula, C is the virtual camera position parameter, Rz is the rotation vector corresponding to the Z-axis in the rotation matrix, and Cz is the virtual camera position parameter on the Z-axis.

[0065] In this embodiment, the intersection method using a depth buffer can avoid traversing the entire 3D image. The intersection point can be quickly located simply by querying the depth buffer value, making it suitable for real-time application scenarios.

[0066] In this embodiment, if there is no intersection in the above processing flow, as another optional embodiment, the method further includes: Obtain the 3D model space of the 3D image corresponding to the 2D image, and divide the 3D model space into an octree hierarchical structure, wherein the octree hierarchical structure includes a root node and leaf nodes. For each node, record the list of triangular faces contained in that node, where the node includes the root node and leaf nodes; Starting from the root node, traverse layer by layer, performing intersection calculations only on nodes traversed by the ray equation, and eliminating non-intersecting nodes to reduce the number of triangular faces; For the triangular facets in the leaf node, the Möller–Trumbore algorithm is used to calculate the intersection points of the ray equation with each triangular facet in the leaf node, and the real constants in the ray equation are obtained. From all the positive real constants, select the intersection point corresponding to the smallest positive real constant to obtain the three-dimensional coordinate point.

[0067] In this embodiment, the spatial octree structure divides the 3D space into hierarchical voxels. By eliminating non-intersecting voxels layer by layer, precise intersection is performed only on a small number of triangular faces, which can significantly improve performance. The complexity of this method is O(logN) (where N is the total number of triangular faces) instead of O(N), resulting in a significant performance improvement on large-scale 3D images.

[0068] In this embodiment, as an optional implementation, for real-time visualization applications, the intersection point can be approximated using the depth buffer accelerated intersection method (Z-buffer), and then the spatial octree structure can be used for precise intersection. For high-precision measurement or simulation, only the spatial octree structure can be used for precise intersection to ensure calculation accuracy. As another optional implementation, the three-dimensional coordinate points can also be obtained directly using the depth buffer accelerated intersection method or the spatial octree structure intersection method, instead of determining that there are no intersection points in the above process. This embodiment does not limit this approach.

[0069] S104. Based on the three-dimensional coordinates corresponding to each two-dimensional pixel in the edited two-dimensional image, construct the three-dimensional image in the three-dimensional model, and map the annotation attributes of the two-dimensional pixels in the edited two-dimensional image to the three-dimensional coordinates corresponding to the two-dimensional pixels.

[0070] In this embodiment, the logical information (points, lines, regions, polygons, etc.) of the edited two-dimensional markers is projected into corresponding three-dimensional geometric entities. For example, two-dimensional points (two-dimensional pixels) in the logical information of the two-dimensional markers are selected and mapped to three-dimensional coordinate points using the above method. Two-dimensional line segments connecting each three-dimensional coordinate point are constructed based on the three-dimensional coordinate points. Two-dimensional curves (connecting ray intersections) are drawn based on the two-dimensional line segments. Two-dimensional regions are formed based on the two-dimensional curves. Three-dimensional polygon patches are formed based on the two-dimensional regions. The edited annotation attributes, such as additional text, symbols, numbers, etc., are bound as attributes to the three-dimensional polygon patches, thereby realizing the mapping collaboration between two-dimensional and three-dimensional images.

[0071] In this embodiment, as an optional implementation, the marking information and the 3D reproduction results can also be synchronized to PCs, mobile devices, and VR / AR terminals. At the same time, any 2D markings added or modified using any terminal can be synchronously mapped to 3D images in real time. It supports generating 2D images from different virtual camera positions and any angle. 2D images and 2D markings generated from different angles can all be calculated back to the same 3D coordinate system through the recorded information to ensure consistency.

[0072] In this embodiment, as an optional implementation, encrypted storage of marked files is supported to ensure data security during the collaboration process. The method of this embodiment can be applied to scenarios such as drone aerial photography, BIM modeling, medical imaging, and simulation training.

[0073] The following are several specific embodiments to illustrate this embodiment in detail.

[0074] Example 1: Establishing a mapping relationship between two-dimensional and three-dimensional dimensions Figure 2 This is a schematic diagram illustrating a process for establishing a mapping relationship in a method for mapping a three-dimensional model to a two-dimensional image, as provided in an embodiment of the present invention. Figure 2 As shown in the figure, this embodiment provides a method for generating a two-dimensional image based on a three-dimensional image and establishing a mapping relationship.

[0075] S201. Load 3D images into the 3D scene and initialize the virtual camera shooting parameters; In this embodiment, a 3D image is loaded using a BIM model, CAD model, or point cloud model. The virtual camera is initialized by selecting the virtual camera position, viewing direction, and projection mode (perspective projection / orthographic projection).

[0076] S202. Based on the initialized virtual camera shooting parameters, render the loaded 3D image to obtain a 2D image; In this embodiment, a 3D image is rendered to generate a 2D image based on the virtual camera's shooting parameters. Simultaneously, the virtual camera's shooting parameters are automatically recorded, such as the virtual camera's intrinsic parameter matrix, rotation matrix, translation vector, projection matrix, coordinate system type, and image resolution.

[0077] S203. Extract two-dimensional feature points from the two-dimensional image corresponding to the three-dimensional image, and extract projection feature points from the projection view corresponding to the three-dimensional image, and establish a mapping relationship between two-dimensional feature points and projection feature points. In this embodiment, as an optional embodiment, feature points are extracted based on the generated two-dimensional image using a feature extraction algorithm. The feature extraction algorithm includes, but is not limited to, the Scale-Invariant Feature Transform (SIFT) algorithm and the Oriented Fast and Rotated BRIEF (ORB) algorithm.

[0078] In this embodiment, projection feature points are extracted from the projection view corresponding to the 3D image. As an optional embodiment, 3D feature points are determined in the 3D image, and the feature points of these 3D feature points in the corresponding 2D image and projection view are obtained respectively.

[0079] In this embodiment, the projected view and the generated 2D image have a one-to-one correspondence. The projected view is the geometric representation of the 3D image in the 3D scene projected in the camera coordinate system, while the 2D image is the final rendering result of the projected view after rasterization, lighting, and material processing. Therefore, any pixel in the 2D image can find its geometric correspondence in the projected view. As an optional embodiment, when extracting projected feature points, the projection matrix P=K[R|t] of the virtual camera can be directly used to project the 3D features forward, ensuring a one-to-one correspondence between the 2D feature points and the 3D points.

[0080] Establish a mapping relationship between two-dimensional pixel coordinates (u,v) and three-dimensional spatial points (X,Y,Z), and store it in matrix form: x_{2D} = K [R|t] X_{3D} S204. Back projection and verification using two-dimensional images; In this embodiment, a back-projection ray equation is constructed using two-dimensional points and parameters captured by a virtual camera; a ray detection algorithm is used to calculate the intersection point between the back-projection ray equation and the three-dimensional image; the intersection point position is compared with the original three-dimensional point, and if the error is less than a threshold, it indicates that the established mapping relationship is effective.

[0081] If the error exceeds the preset threshold, the mapping relationship will be dynamically adjusted using the following steps: Camera parameter fine-tuning: Based on the error direction, make small corrections to the camera translation vector (T) or rotation matrix (R), and recalculate the projection matrix P.

[0082] Feature point re-matching: Re-extract the two-dimensional features corresponding to the three-dimensional points and update the pixel coordinates.

[0083] Iterative convergence: Repeat the correction-matching process until the error is below the threshold or the maximum number of iterations is reached.

[0084] Weight optimization: If multiple feature points have errors, the least squares method is used to optimally fit the global mapping matrix to ensure the overall mapping accuracy.

[0085] S205, Stores virtual camera shooting parameters and mapping relationships.

[0086] In this embodiment, the virtual camera shooting parameters and the two-dimensional-three-dimensional mapping relationship (point pairs, matrices, camera parameters) are stored in the database, which can be directly called in subsequent annotation and collaboration to achieve accurate projection and back projection.

[0087] Example 2: Synchronous Interaction Between 3D and 2D Annotation Figure 3 This is a schematic diagram illustrating a process of mapping a 3D model to a 2D image in a 3D model mapping method provided in an embodiment of the present invention. For example... Figure 3 As shown, this embodiment provides a method for annotating a 2D image generated in 3D and synchronizing it to 3D space in real time.

[0088] S301. Generate a 2D image based on a 3D image, and store the virtual camera shooting parameters and mapping relationships; In this embodiment, the user sets the viewpoint in the 3D image, the camera renders and generates a 2D image, and at the same time records the virtual camera matrix (K, R, t, P) and the image resolution.

[0089] S302. Perform two-dimensional annotation operations on two-dimensional images; In this embodiment, the user draws annotations such as points, lines, polygons, rectangles, and text on a two-dimensional image. The annotations are represented by pixel coordinates (u,v) and are accompanied by a unique number, type, and text description.

[0090] S303. Back-project the two-dimensional annotations to three-dimensional space; In this embodiment, the projection matrix P of the virtual camera is used to construct the equation of the back-projection ray emanating from the camera position, calculate the intersection point of the back-projection ray with the three-dimensional image surface, and convert the two-dimensional annotations into three-dimensional annotations. For example, point annotations are converted into three-dimensional coordinate point annotations, line annotations are converted into three-dimensional polyline annotations composed of multiple points, rectangles are converted into corresponding three-dimensional plane ranges, and text is converted into three-dimensional labels attached to the intersection points.

[0091] S304. Implement a two-way synchronization mechanism between two-dimensional and three-dimensional images.

[0092] In this embodiment, when the 2D image annotations are updated, the back projection is recalculated and the 3D annotations are updated; and when the 3D annotations are edited, the 2D image is projected back in real time and the corresponding annotations are updated.

[0093] In this embodiment, all annotations are stored in JSON format, including two-dimensional pixel coordinates, three-dimensional coordinates, camera parameters, annotation type, text, and timestamp. Users can quickly retrieve annotations by number or keyword and locate them in the two-dimensional or three-dimensional interface.

[0094] Example 3: Cross-terminal collaboration Figure 4 This is a schematic diagram illustrating a cloud-based method for mapping 3D models to 2D images, as provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a cross-terminal two-dimensional-three-dimensional collaboration method based on cloud storage.

[0095] S401. Store the 3D images in the cloud so that the cloud can manage the 3D images uniformly. In this embodiment, the 3D images are stored in the cloud and can be accessed by PCs, mobile devices, and VR / AR devices. The cloud provides a unified camera rendering service, and users can obtain 2D images by calling the rendering interface through different terminals. At the same time, virtual camera parameters and projection matrices are sent out.

[0096] S402. The terminal performs adaptation rendering of the two-dimensional image and annotates the two-dimensional image; In this embodiment, for PC, high-precision 3D images and full-resolution 2D images can be rendered; for mobile, optimized and simplified low-poly models and compressed images are used; for VR / AR, real-time rendered scenes and panoramic textures with LOD (Level of Detail) optimization are used.

[0097] S403. Synchronize the marked areas in the 2D image to the cloud.

[0098] In this embodiment, when any terminal generates annotations on a 2D image, it saves the virtual camera matrix and pixel coordinates. After calculating the 3D position through back projection, the annotations are immediately synchronized to the cloud database. After receiving the update event, other terminals project the annotations onto the corresponding 2D or 3D image, thus achieving cross-platform synchronization.

[0099] In this embodiment, the cloud supports simultaneous operation by multiple users. All annotations, comments, and voice descriptions are bound to 3D objects, and the cloud uses a message queue for broadcasting to ensure real-time synchronization.

[0100] In this embodiment, when multiple users modify the same label, a version number mechanism is used to save all versions. Users can choose to roll back or merge, ensuring that the collaboration record is complete and traceable.

[0101] Figure 5 This is a schematic diagram illustrating an application scenario of a three-dimensional model-to-two-dimensional image mapping method provided by an embodiment of the present invention. For example... Figure 5As shown, a core collaboration system is constructed according to the method of this embodiment. In this embodiment, the core collaboration system can support multiple coordinate systems and multiple camera models, support automatic switching between left-handed and right-handed coordinate systems, and support pinhole cameras, fisheye cameras, and panoramic cameras. Accurate projection is achieved through distortion model correction.

[0102] In this embodiment, the core collaboration system supports high-resolution image processing. For example, when generating two-dimensional images, it supports block rendering and hierarchical storage. On low-performance devices, it uses downsampling and resolution pyramid to ensure smooth interaction.

[0103] In this embodiment, as an optional implementation, the annotation may also include files, hyperlinks, or video streams. For example, clicking on an annotation in a 3D image can open the associated file or real-time video.

[0104] In this embodiment, as an optional implementation, the core collaboration system can also implement intelligent assistance functions. For example, through built-in AI detection, it can automatically identify the outlines of components (such as beams, columns, and pipes) in two-dimensional images and quickly align them to three-dimensional images. The specific process includes: Data preprocessing: Denoising, enhancement, and edge detection are performed on the 2D images to improve the contrast of component outlines; Semantic segmentation model inference: Using deep neural networks (such as U-Net, Mask R-CNN), pixel-level segmentation is performed on the component regions in the image to obtain the corresponding mask; Contour extraction: Extract precise component geometric boundaries using contour tracking algorithms (such as the Canny+Suzuki algorithm); Geometric feature matching: Using the constraints of feature points and geometric shapes (such as rectangles and cylinders), the extracted contours are matched with the geometric information of components in the 3D image database; Inverse projection calculation: The two-dimensional contour of the matching result is back-calculated into the three-dimensional coordinate system through camera parameters to obtain the actual position of the component in three dimensions; Automatic alignment: The system automatically performs geometric fitting and alignment between the 3D object corresponding to the contour and the component entity in the model, reducing manual operation.

[0105] In this embodiment, the core collaboration system supports timeline-based annotation and playback, allowing users to view the two-dimensional and three-dimensional status at any given time; all operations are logged, and administrators can export audit reports.

[0106] The method in this embodiment records camera capture information and a mapping matrix when generating a 2D image based on 3D space. Then, based on the recorded information, it uses ray detection and matrix operations to reverse-engineer the 2D image back into 3D space, thereby establishing a strict mathematical mapping relationship between 2D and 3D spaces. This method has the following beneficial technical effects: Two-way mapping capability: It realizes two-way mapping from 3D to 2D and from 2D to 3D, and the 3D position can be automatically deduced from the 2D mark.

[0107] High precision and robustness: By utilizing projection matrices and ray detection algorithms, the marking positions are ensured to be accurate and remain stable even in complex models.

[0108] Arbitrary angle support: It does not depend on a fixed viewpoint and supports any camera position and any coordinate system (left-hand or right-hand).

[0109] Information integrity: Tag information is stored in JSON format, including type, text, number, camera parameters, etc., to ensure traceability and reproducibility.

[0110] Cross-platform collaboration: It can be synchronized on PCs, mobile devices, and VR / AR devices, making it suitable for remote collaboration scenarios.

[0111] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention also provides a three-dimensional model and two-dimensional image mapping device, the device comprising: The shooting parameter acquisition module 601 is used to acquire the virtual camera shooting parameters when generating a two-dimensional image through a virtual camera in three-dimensional space. The virtual camera shooting parameters include: external camera parameters, internal camera parameters, and projection mode. In this embodiment, the virtual camera shooting parameters include, but are not limited to: Extrinsic parameters of the camera, including but not limited to: virtual camera position parameters, orientation vector, and upward vector; Camera parameters, including but not limited to: focal length, principal point offset, sensor size, and resolution; Projection modes, including but not limited to: perspective projection or orthographic projection.

[0112] In this embodiment, as an optional embodiment, the shooting parameter acquisition module 601 is specifically used for: Using a projection matrix, each 3D point in a 3D image displayed in 3D space is projected and transformed to obtain the projected coordinates of that 3D point. The projected coordinates include: a horizontal projection coordinate component, a vertical projection coordinate component, and a projection depth component. Based on the projected coordinates, 2D screen coordinates are obtained. Based on the 2D screen coordinates corresponding to each 3D point in the 3D image, a 2D image is generated, and the projection depth component corresponding to the 2D image is stored.

[0113] The parameter processing and storage module 602 is used to obtain the projection matrix in the projection mode based on the camera extrinsic parameters and camera in-camera parameters of the virtual camera in the projection mode, and to determine the direction vector coordinate system of the virtual camera based on the camera extrinsic parameters of the virtual camera in the projection mode, and to store the virtual camera shooting parameters, projection matrix and direction vector coordinate system. In this embodiment, as an optional embodiment, the parameter processing and storage module 602 is specifically used for: Based on the camera's intrinsic parameters, such as focal length, principal point offset, sensor size, and resolution, a virtual camera intrinsic parameter matrix is ​​constructed. Based on the camera's extrinsic parameters, such as the orientation vector, upward vector, and virtual camera position parameters, a virtual camera extrinsic parameter matrix is ​​constructed. Based on the virtual camera's intrinsic and extrinsic parameter matrices, the projection matrix of the virtual camera in projection mode is obtained.

[0114] The coordinate point mapping module 603 is used to convert the two-dimensional pixel coordinates corresponding to each two-dimensional pixel point of the edited two-dimensional image into projected coordinates, and to construct ray equations and calculate the intersection points of the ray equations and the surface of the three-dimensional model based on the shooting parameters of the virtual camera and the projected coordinates to obtain three-dimensional coordinate points. In this embodiment, as an optional embodiment, the coordinate point mapping module 603 is specifically used for: Based on the sensor size and resolution in the camera's intrinsic parameters and the virtual camera intrinsic parameter matrix, the two-dimensional pixel coordinates corresponding to the two-dimensional pixels are normalized to projected coordinates; based on the projected coordinates and the rotation matrix in the projection matrix, the ray direction vector is determined; based on the ray direction vector and the virtual camera position parameter information corresponding to the stored two-dimensional image, the ray equation is constructed.

[0115] In this embodiment, as another optional embodiment, the coordinate point mapping module 603 is further used for: Using a pre-set ray detection algorithm, the intersection points of the ray equation and the surface of the 3D image in the 3D spatial model corresponding to the 2D image are obtained.

[0116] In this embodiment, as an optional implementation, a pre-set ray detection algorithm is used to obtain the intersection points of the ray equation and the surface of the three-dimensional image in the three-dimensional spatial model corresponding to the two-dimensional image, including: Triangular facets are set on the surface of a 3D image along the ray direction vector, and the edge vectors of the triangular facets are calculated, including a first edge vector and a second edge vector. Calculate the first vector based on the first side vector and the stored virtual camera's direction vector coordinate system; Calculate the second vector based on the second side vector and the first vector; Determine that the second vector is not zero, and obtain the first comparison value based on the second vector, the first vector, the common vertices of the second vector and the first vector in the triangular facet, and the virtual camera position parameters; The first comparison value is determined to be greater than zero and less than 1. The second comparison value is obtained based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, and the stored virtual camera direction vector coordinate system. Determine that the second comparison value is not less than 0 and the sum of the first comparison value and the second comparison value is not greater than 1. Based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, the stored virtual camera direction vector coordinate system, and the second vector, obtain the third comparison value. If the third comparison value is determined to be greater than 0, the third comparison value is used as a real constant of the ray equation. Based on the ray equation with the assigned real constant, the intersection point of the ray equation and the surface of the three-dimensional image in the three-dimensional spatial model corresponding to the two-dimensional image is obtained.

[0117] In this embodiment, as another optional embodiment, the coordinate point mapping module 603 is further used for: If the second vector is equal to zero, obtain the three-dimensional model space of the three-dimensional image corresponding to the two-dimensional image, and divide the three-dimensional model space into an octree hierarchical structure, wherein the octree hierarchical structure includes a root node and leaf nodes. For each node, record the list of triangular faces contained in that node, where the node includes the root node and leaf nodes; Starting from the root node, traverse layer by layer, performing intersection calculations only on nodes traversed by the ray equation, and discarding non-intersecting nodes; For the triangular facets in the leaf node, the Möller–Trumbore algorithm is used to calculate the intersection points of the ray equation with each triangular facet in the leaf node, and the real constants in the ray equation are obtained. From all the positive real constants, select the intersection point corresponding to the smallest positive real constant to obtain the three-dimensional coordinate point.

[0118] The 3D image reconstruction module 604 is used to construct a 3D image in a 3D model based on the 3D coordinates corresponding to each 2D pixel in the edited 2D image, and to map the annotation attributes of the 2D pixels in the edited 2D image to the 3D coordinates corresponding to the 2D pixels.

[0119] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the three-dimensional model and two-dimensional image mapping method in any of the above possible implementations.

[0120] Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0121] Based on the same inventive concept, see [link to inventive concept] Figure 7This invention also provides an electronic device, including a memory 101 (e.g., non-volatile memory), a processor 102, and a computer program stored on the memory 101 and executable on the processor 102. When the processor 102 executes the program, it implements the steps of the three-dimensional model-to-two-dimensional image mapping method described in any possible implementation, which is equivalent to the aforementioned three-dimensional model-to-two-dimensional image mapping device. Of course, the processor can also be used to process other data or perform calculations. This electronic device can be a PC, server, terminal, or other similar device.

[0122] like Figure 7 As shown, the electronic device may also include: memory 103, network interface 104, and internal bus 105. In addition to these components, other hardware may also be included, which will not be described in detail here.

[0123] It should be noted that the above-mentioned three-dimensional model and two-dimensional image mapping device can be implemented by software. As a device in a logical sense, it is formed by the processor 102 of the electronic device in which it is located reading the computer program instructions stored in the non-volatile memory into the memory 103 for execution.

[0124] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0125] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by special-purpose logic circuitry—such as FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), and the device can also be implemented as special-purpose logic circuitry.

[0126] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0127] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0128] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0129] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0130] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0131] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0132] The above are merely specific embodiments of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for mapping a three-dimensional model to a two-dimensional image, characterized in that, include: The virtual camera shooting parameters are obtained when generating a two-dimensional image in three-dimensional space using a virtual camera. The virtual camera shooting parameters include: external camera parameters, internal camera parameters, and projection mode. Based on the camera extrinsic and intra-camera parameters of the virtual camera in projection mode, obtain the projection matrix in that projection mode; and based on the camera extrinsic parameters of the virtual camera in projection mode, determine the orientation vector coordinate system of the virtual camera, and store the virtual camera shooting parameters, projection matrix, and orientation vector coordinate system. For each two-dimensional pixel in the edited two-dimensional image, the two-dimensional pixel coordinates corresponding to the two-dimensional pixel are converted into projected coordinates. Based on the shooting parameters of the virtual camera and the projected coordinates, a ray equation is constructed and the intersection of the ray equation with the surface of the three-dimensional model is calculated to obtain the three-dimensional coordinate points. Based on the three-dimensional coordinates corresponding to each two-dimensional pixel in the edited two-dimensional image, a three-dimensional image in the three-dimensional model is constructed, and the annotation attributes of the two-dimensional pixels in the edited two-dimensional image are mapped to the three-dimensional coordinates corresponding to the two-dimensional pixels.

2. The method for mapping three-dimensional models to two-dimensional images according to claim 1, characterized in that, The step of obtaining the projection matrix in the projection mode based on the extrinsic and intrinsic parameters of the virtual camera in projection mode includes: A virtual camera intrinsic parameter matrix is ​​constructed based on the camera intrinsic parameters, including focal length, principal point offset, sensor size, and resolution. Based on the orientation vector, upward vector, and virtual camera position parameters in the camera extrinsic parameters, construct the virtual camera extrinsic parameter matrix; Based on the virtual camera intrinsic parameter matrix and the virtual camera extrinsic parameter matrix, obtain the projection matrix in the projection mode of the virtual camera.

3. The method for mapping three-dimensional models to two-dimensional images according to claim 2, characterized in that, The process of generating a two-dimensional image using a virtual camera includes: Using the projection matrix, each three-dimensional point in the three-dimensional image displayed in three-dimensional space is projected and transformed to obtain the projected coordinates of the three-dimensional point. The projected coordinates include: a projection horizontal coordinate component, a projection vertical coordinate component, and a projection depth component. Obtain two-dimensional screen coordinates based on projected coordinates; A two-dimensional image is generated based on the two-dimensional screen coordinates corresponding to each three-dimensional point in the three-dimensional image, and the projection depth component corresponding to the two-dimensional image is stored.

4. The method for mapping a three-dimensional model to a two-dimensional image according to claim 1, characterized in that, The step of converting the two-dimensional pixel coordinates corresponding to the two-dimensional pixel point into projected coordinates, and constructing a ray equation based on the virtual camera shooting parameters and the projected coordinates, includes: Based on the sensor size and resolution in the camera's intrinsic parameters and the virtual camera intrinsic parameter matrix, the two-dimensional pixel coordinates corresponding to the two-dimensional pixels are normalized into projected coordinates; Determine the ray direction vector based on the projected coordinates and the rotation matrix in the projection matrix; A ray equation is constructed based on the ray direction vector and the virtual camera position parameter information corresponding to the stored two-dimensional image.

5. The method for mapping a three-dimensional model to a two-dimensional image according to any one of claims 1 to 4, characterized in that, The calculation of the intersection of the ray equation and the surface of the three-dimensional model yields three-dimensional coordinate points, including: Using a pre-set ray detection algorithm, the intersection points of the ray equation and the surface of the 3D image in the 3D spatial model corresponding to the 2D image are obtained.

6. The method for mapping three-dimensional models to two-dimensional images according to claim 5, characterized in that, The step of using a pre-set ray detection algorithm to obtain the intersection points of the ray equation and the surface of the 3D image in the 3D spatial model corresponding to the 2D image includes: Triangular facets are set on the surface of a 3D image along the ray direction vector, and the edge vectors of the triangular facets are calculated, including a first edge vector and a second edge vector. Calculate the first vector based on the first side vector and the stored virtual camera's direction vector coordinate system; Calculate the second vector based on the second side vector and the first vector; Determine that the second vector is not zero, and obtain the first comparison value based on the second vector, the first vector, the common vertices of the second vector and the first vector in the triangular facet, and the virtual camera position parameters; The first comparison value is determined to be greater than zero and less than 1. The second comparison value is obtained based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, and the stored virtual camera direction vector coordinate system. Determine that the second comparison value is not less than 0 and the sum of the first comparison value and the second comparison value is not greater than 1. Based on the common vertex of the second vector and the first vector in the triangular facet, the virtual camera position parameters, the first edge vector, the stored virtual camera direction vector coordinate system, and the second vector, obtain the third comparison value. If the third comparison value is determined to be greater than 0, the third comparison value is used as a real constant of the ray equation. Based on the ray equation with the assigned real constant, the intersection point of the ray equation and the surface of the three-dimensional image in the three-dimensional spatial model corresponding to the two-dimensional image is obtained.

7. The method for mapping three-dimensional models to two-dimensional images according to claim 6, characterized in that, The method further includes: If the second vector is equal to zero, obtain the three-dimensional model space of the three-dimensional image corresponding to the two-dimensional image, and divide the three-dimensional model space into an octree hierarchical structure, wherein the octree hierarchical structure includes a root node and leaf nodes. For each node, record the list of triangular faces contained in that node, where the node includes the root node and leaf nodes; Starting from the root node, traverse layer by layer, performing intersection calculations only on nodes traversed by the ray equation, and discarding non-intersecting nodes; For the triangular facets in the leaf node, the Möller–Trumbore algorithm is used to calculate the intersection points of the ray equation with each triangular facet in the leaf node, and the real constants in the ray equation are obtained. From all the positive real constants, select the intersection point corresponding to the smallest positive real constant to obtain the three-dimensional coordinate point.

8. A device for mapping a three-dimensional model to a two-dimensional image, characterized in that, The 3D model to 2D image mapping device includes: The shooting parameter acquisition module is used to acquire the virtual camera shooting parameters when generating a two-dimensional image through a virtual camera in three-dimensional space. The virtual camera shooting parameters include: external camera parameters, internal camera parameters, and projection mode. The parameter processing and storage module is used to obtain the projection matrix in the projection mode based on the camera's external and internal parameters in the projection mode, and to determine the direction vector coordinate system of the virtual camera based on the camera's external parameters in the projection mode, and to store the virtual camera's shooting parameters, projection matrix and direction vector coordinate system. The coordinate point mapping module is used to convert the two-dimensional pixel coordinates corresponding to each two-dimensional pixel in the edited two-dimensional image into projected coordinates. Based on the shooting parameters of the virtual camera and the projected coordinates, it constructs a ray equation and calculates the intersection of the ray equation with the surface of the three-dimensional model to obtain the three-dimensional coordinate point. The 3D image reconstruction module is used to construct a 3D image in a 3D model based on the 3D coordinates corresponding to each 2D pixel in the edited 2D image, and to map the annotation attributes of the 2D pixels in the edited 2D image to the corresponding 3D coordinates.

9. A storage medium, characterized in that, The storage medium stores a program or instructions, which are executed by a processor to implement the steps of the three-dimensional model and two-dimensional image mapping method as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the three-dimensional model and two-dimensional image mapping method as described in any one of claims 1 to 7.

Images