Screen positioning method for target object in three-dimensional scene

Abstract

The invention provides a screen positioning method for a target object in a three-dimensional scene, and the method comprises the steps: simulating a rendering pipeline of graphic hardware through software, and sequentially converting a three-dimensional object coordinate of the target object into a word coordinate, an eye frame coordinate, a clipping coordinate, a normal device coordinate and a viewport coordinate. And performing matching operation on the two-dimensional screen based on the viewport coordinate of the target object, and determining the actual three-dimensional position of the target object selected by the user. Accurate mapping from the three-dimensional space target object to the two-dimensional screen can be achieved, the problem that it is difficult for a user to correspond the picture of the target object to the actual position is solved, and the method is applied to the field of pipeline detection and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics and three-dimensional visualization technology, and more particularly to a method for positioning a target object on a screen in a three-dimensional scene. Background Art

[0002] In fields such as pipeline inspection, target objects usually refer to foreign substances or objects left in the pipeline that affect the normal operation of the pipeline, such as metal tools, screws, nuts, etc., most of which are made of metal. The existing solution is to use a computer image recognition system to detect target objects and take photos for users to identify. However, when the pipeline is long or the number of target objects is large, the number of photos taken will also increase, making it difficult for users to match the range of a certain target object photo with the actual part of the pipeline. The existing technology uses computer graphics technology to display three-dimensional simulated pipeline graphics, and displays the target object in the pipeline graphic according to coordinates. The user can select the target object with the mouse, and then the positioning software locates and displays the photo containing the target object. However, this method has technical difficulties, namely how to accurately map the target object on the two-dimensional screen to the actual three-dimensional target object.

[0003] During the implementation of the present invention, the inventors discovered that the prior art suffers from at least the following problems or deficiencies: The prior art is unable to directly obtain the accurate mapping relationship between a target object from three-dimensional space to a two-dimensional screen. This is because after the graphics hardware's rendering pipeline completes the coordinate transformation, the target object's two-dimensional screen coordinates are not directly known to the outside world. This makes it difficult for users to accurately map the target object on the two-dimensional screen to the actual three-dimensional target object when selecting it, affecting the convenience and accuracy of the operation. Summary of the Invention

[0004] The present invention provides a method for positioning a target object on a screen in a three-dimensional scene, comprising:

[0005] By simulating the rendering pipeline of graphics hardware through software, the 3D object coordinates of the target object are converted into world coordinates, eye frame coordinates, clipping coordinates, normalized device coordinates and viewport coordinates in sequence;

[0006] Based on the viewport coordinates of the target object, a matching operation is performed on the two-dimensional screen to determine the actual three-dimensional position of the target object selected by the user;

[0007] Wherein, the coordinate transformation includes:

[0008] (a) Convert object coordinates to world coordinates using an affine transformation matrix A, where the affine transformation matrix A contains a linear transformation submatrix L and a translation transformation submatrix T;

[0009] (b) Convert the world coordinates to eye frame coordinates through the affine transformation matrix M;

[0010] (c) Convert the eye frame coordinates to clipping coordinates through the projection matrix P;

[0011] (d) Convert the clipping coordinates to normalized device coordinates through normalization operation;

[0012] (e) Convert the normalized device coordinates to viewport coordinates using the viewport transformation matrix.

[0013] Furthermore, the expression of the affine transformation matrix A is:

[0014]

[0015] Where L is a 3×3 linear transformation matrix, T is a 3×1 translation vector, and 0 is a 1×3 zero matrix.

[0016] Furthermore, the expression of the projection matrix P is:

[0017]

[0018] Where n is the near plane distance, f is the far plane distance, l, r, b, and t are the left, right, bottom, and top boundary values ​​of the projection plane, respectively.

[0019] Furthermore, the normalization operation includes:

[0020] Divide the x, y, and z components of the clipping coordinates by their homogeneous w components to obtain the normalized device coordinates.

[0021] Furthermore, the expression of the viewport transformation matrix is:

[0022]

[0023] Where W is the viewport width and H is the viewport height.

[0024] Furthermore, the matching operation includes:

[0025] Model the target object as a three-dimensional sphere and calculate the viewport coordinates and mapping radius of its center.

[0026] Determine whether the target object is selected based on the distance between the mouse's viewport coordinates and the sphere's center viewport coordinates.

[0027] Furthermore, the mapping radius is obtained by scaling the radius of the three-dimensional sphere to the two-dimensional viewport according to the projection ratio.

[0028] Furthermore, the conversion of the eye frame coordinates includes:

[0029] The camera position and viewing angle are adjusted by the affine transformation matrix M, where M is a 4×4 matrix containing the camera's rotation, translation, and scaling parameters.

[0030] Furthermore, the method further comprises:

[0031] In the depth test, the occlusion relationship of the target object is determined by the z component of the normalized device coordinates.

[0032] Furthermore, the method is applied to a pipeline detection system to match a target object photo on a two-dimensional screen with the actual position of the pipeline.

[0033] The above-described embodiments of the present invention have at least the following beneficial effects: By simulating the graphics hardware rendering pipeline in software, the present invention can achieve precise mapping from 3D target objects to 2D screen coordinates, effectively resolving the difficulty users face in accurately aligning 2D on-screen objects with their actual 3D positions in complex 3D scenes. This method can be widely applied in engineering fields requiring precise 3D object positioning, such as pipeline inspection systems, helping users quickly and accurately locate target objects and improving work efficiency.

[0034] In addition, the present invention can also use the z component of the normalized device coordinates through the depth test function to determine the occlusion relationship of the target object, thereby further optimizing the visualization effect of the three-dimensional scene and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0036] Figure 1 A schematic diagram of a coordinate transformation process implemented by a hardware pipeline of a method for screen positioning of a target object in a three-dimensional scene provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the relative relationship of coordinate systems of a method for screen positioning of a target object in a three-dimensional scene provided by an embodiment of the present invention;

[0038] Figure 3A schematic diagram of a 3D scene from an eye perspective of a method for screen positioning a target object in a 3D scene provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of eye field of view selection for a method for locating a target object on a screen in a three-dimensional scene according to an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a selected projection plane for a method for screen positioning of a target object in a three-dimensional scene provided by an embodiment of the present invention;

[0041] Figure 6 A viewfrustum diagram of a method for screen positioning of a target object in a three-dimensional scene provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of the projection effect of a graphic on a projection plane of a method for screen positioning a target object in a three-dimensional scene provided by one embodiment of the present invention;

[0043] Figure 8 A schematic diagram of normalized device coordinate mapping for a method for screen positioning of a target object in a three-dimensional scene provided by an embodiment of the present invention;

[0044] Figure 9 A schematic diagram of a 12 pixel × 8 pixel window viewport for a method for screen positioning of a target object in a three-dimensional scene provided by an embodiment of the present invention;

[0045] Figure 10 A schematic diagram of the actual effect of target object selection in the screen positioning method of a target object in a three-dimensional scene provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0047] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0048] It should be noted that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0049] Example 1

[0050] A method for positioning a target object on a screen in a three-dimensional scene includes:

[0051] By simulating the rendering pipeline of graphics hardware through software, the 3D object coordinates of the target object are converted into world coordinates, eye frame coordinates, clipping coordinates, normalized device coordinates and viewport coordinates in sequence;

[0052] Based on the viewport coordinates of the target object, a matching operation is performed on the two-dimensional screen to determine the actual three-dimensional position of the target object selected by the user;

[0053] Wherein, the coordinate transformation includes:

[0054] (a) Convert object coordinates to world coordinates using an affine transformation matrix A, where the affine transformation matrix A contains a linear transformation submatrix L and a translation transformation submatrix T;

[0055] (b) Convert the world coordinates to eye frame coordinates through the affine transformation matrix M;

[0056] (c) Convert the eye frame coordinates to clipping coordinates through the projection matrix P;

[0057] (d) Convert the clipping coordinates to normalized device coordinates through normalization operation;

[0058] (e) Convert the normalized device coordinates to viewport coordinates using the viewport transformation matrix.

[0059] It should be noted that the present invention proposes a screen positioning method for a target object in a three-dimensional scene, which simulates the rendering pipeline of graphics hardware through software, converts the three-dimensional coordinates of the target object into coordinates in different coordinate systems in turn, and finally realizes precise positioning from three-dimensional space to two-dimensional screen. Among them, the three-dimensional coordinate conversion involves multiple stages, including converting from the target object's own coordinate system (object coordinates) to the world coordinate system (world coordinates), then to the view coordinate system (eye frame coordinates), followed by the clipping coordinate system (clipping coordinates), the normalized device coordinate system (normalizeddevice coordinates), and finally converting to the viewport coordinate system (viewport coordinates). These coordinate conversion processes are the key steps to achieve accurate display of three-dimensional objects on a two-dimensional screen, and the matching operation is to determine whether the user has selected the target object by calculating the position relationship between the mouse position and the target object in the viewport coordinate system.

[0060] Specifically, object coordinates are the coordinates of the target object in its own local coordinate system, world coordinates are the coordinates converted from the target object's local coordinates to the global world coordinate system, and eye frame coordinates are the coordinates converted from world coordinates to a coordinate system with the camera as the origin. Clipping coordinates are the coordinates obtained by transforming the eye frame coordinates through the projection matrix and are used to determine whether the object is within the camera's field of view. Normalized device coordinates are the coordinates normalized by the clipping coordinates, ranging from [-1, 1], and used for device-independent standardized representation. Viewport coordinates are the coordinates converted from the normalized device coordinates to the actual pixel coordinates in the screen viewport. During the coordinate transformation process, the affine transformation matrix A is used to convert object coordinates to world coordinates. It contains the linear transformation submatrix L and the translation transformation submatrix T, which are used to implement rotation, scaling, and translation operations. The projection matrix P is used to convert eye frame coordinates to clipping coordinates. The projection method is defined based on the parameters of the projection plane (such as the left, right, top, and bottom boundaries, and the distances to the near and far planes).

[0061] To improve positioning accuracy, the target object can be modeled as a 3D sphere, with the viewport coordinates of its center and the mapping radius calculated using projection scaling. During the matching operation, the user determines that the target object has been selected by comparing the distance between the mouse position and the viewport coordinates of the target object's center. If the distance is less than the mapping radius, the user is deemed to have selected the target object. Furthermore, depth testing can determine the occlusion relationship of the target object using the z component of the normalized device coordinates, thereby optimizing the display of the 3D scene.

[0062] In some embodiments, the expression of the affine transformation matrix A is:

[0063]

[0064] Where L is a 3×3 linear transformation matrix, T is a 3×1 translation vector, and 0 is a 1×3 zero matrix.

[0065] It should be noted that the affine transformation matrix A mentioned in the present invention is a 4×4 matrix used to convert the three-dimensional object coordinates of the target object into world coordinates. Among them, the object coordinates are the coordinates of the target object in its own local coordinate system, and the world coordinates are the coordinates of the target object placed in the global world coordinate system. The affine transformation matrix A contains a linear transformation submatrix L and a translation transformation submatrix T. The linear transformation submatrix L is used to implement rotation and scaling operations, while the translation transformation submatrix T is used to implement object translation operations. This matrix-based transformation can map the local coordinates of the target object to the global world coordinate system through affine transformation, providing a basis for subsequent coordinate conversion.

[0066] Specifically, the expression of the affine transformation matrix A is:

[0067]

[0068] Among them, L is a 3×3 linear transformation matrix, T is a 3×1 translation vector, and 0 is a 1×3 zero matrix. The linear transformation matrix L can include a rotation matrix (used to change the direction of an object), a scaling matrix (used to change the size of an object), etc., while the translation vector T is used to move the object from its local coordinate system origin to a specified position in the world coordinate system. In practical applications, the specific parameters of L and T can be set according to the initial position, rotation angle, and scaling ratio of the target object. For example, if an object needs to be rotated a certain angle around a certain axis, it can be achieved by calculating the rotation matrix and assigning it to L; if the object needs to be translated to a certain point in the world coordinate system, it can be done by setting the translation vector T.

[0069] Preferably, in order to achieve more flexible coordinate transformation, the affine transformation matrix A can be expanded or optimized. For example, in addition to basic rotation, scaling, and translation operations, a shear transformation can be introduced and implemented by modifying certain elements of the linear transformation matrix L.

[0070] Furthermore, for complex object transformations, multiple affine transformation matrices can be combined to perform sequential transformation operations through matrix multiplication. For example, rotating an object, then scaling and translating it, can be achieved by multiplying multiple transformation matrices. This combined transformation approach greatly enhances the functionality of the affine transformation matrix A, enabling it to adapt to more complex 3D scene transformation requirements.

[0071] In some embodiments, the projection matrix P is expressed as:

[0072]

[0073] Where n is the near plane distance, f is the far plane distance, l, r, b, and t are the left, right, bottom, and top boundary values ​​of the projection plane, respectively.

[0074] It should be noted that the projection matrix P mentioned in the present invention is a 4×4 matrix used to convert the eye frame coordinates of the target object into clipping coordinates. Eye frame coordinates are coordinates in a coordinate system with the camera as the origin, while clipping coordinates are coordinates after projection transformation, which are used to determine whether the object is within the field of view of the camera. The design of the projection matrix P is based on the parameters of the projection plane, including the near plane distance (n), the far plane distance (f), and the left, right, bottom, and top boundary values ​​of the projection plane (l, r, b, t). These parameters together define the shape and size of the view frustum, thereby determining which parts of the three-dimensional scene will be projected onto the two-dimensional screen.

[0075] Specifically, the expression of the projection matrix P is:

[0076]

[0077] Here, n and f represent the distances to the near and far planes, respectively, while l, r, b, and t represent the left, right, bottom, and top bounds of the projection plane, respectively. Setting these parameters directly affects the projection quality. For example, the values ​​of n and f determine the depth range of the viewing cone, while the values ​​of l, r, b, and t determine the size and shape of the projection plane. In practical applications, these parameters can be set based on the needs of the 3D scene and the camera's field of view (FOV). For example, for a scene with a wide field of view, the values ​​of r and t can be appropriately increased, while the values ​​of l and b can be reduced to achieve a wider field of view.

[0078] Preferably, the projection matrix P can be adjusted or optimized to accommodate different projection requirements. For example, in perspective projection, the concepts of field of view (FOV) and aspect ratio can be introduced to calculate the boundary values ​​of the projection plane. Specifically, the field of view determines the range of the field of view, while the aspect ratio determines the shape of the projection plane.

[0079] Furthermore, for orthographic projection, the projection matrix P can be simplified to a simpler form, where the values ​​of n and f no longer affect the projection result, and the object is directly projected onto a parallel plane. This projection method is suitable for scenes where the object's proportions need to be maintained, such as in architectural models or CAD designs.

[0080] In some embodiments, the normalization operation includes:

[0081] Divide the x, y, and z components of the clipping coordinates by their homogeneous w components to obtain the normalized device coordinates.

[0082] It should be noted that normalization is a key step in converting clipping coordinates into normalized device coordinates. Clipping coordinates are coordinates transformed by the projection matrix, while normalized device coordinates are coordinates that are further normalized to the range of [-1, 1]. This normalization process enables different graphics hardware to process three-dimensional graphics data in a unified manner, thereby ensuring the consistency of graphics on different devices. Normalization is achieved by dividing the x, y, and z components of the clipping coordinates by their homogeneous coordinate w component. This process is also called perspective divide.

[0083] Specifically, the formula for normalization operation is:

[0084]

[0085] Here, x, y, and z are the positional components of the clipping coordinates, and w is the homogeneous coordinate component. In 3D graphics rendering, the homogeneous coordinate w is used to represent the depth information of a point. Perspective division can be used to map points in 3D space to a normalized device coordinate system. In practical applications, the w component is typically calculated from the projection matrix P, and its value depends on the object's position in 3D space and the projection method. For example, in perspective projection, the w component is typically proportional to the object's depth. Therefore, after perspective division, distant objects appear smaller in the normalized device coordinate system, consistent with the perspective effect.

[0086] Preferably, to ensure the accuracy and stability of the normalization operation, the w component can be checked before the perspective division to avoid division by zero. In addition, for some special cases, such as when the object is located on the edge of the projection plane, a small offset can be introduced to avoid numerical instability. For example, a very small positive value (such as 1e-6) can be added to the w component to ensure that no numerical problems will occur during the division operation.

[0087] Furthermore, for different projection methods, such as orthographic projection and perspective projection, the normalization operation can be handled differently. In orthographic projection, the w component is usually 1, so the normalization operation can directly map the clipping coordinates to the normalized device coordinate system without the need for complex perspective division.

[0088] In some embodiments, the viewport transformation matrix is ​​expressed as:

[0089]

[0090] Where W is the viewport width and H is the viewport height.

[0091] It's important to note that the viewport transformation matrix converts normalized device coordinates into viewport coordinates. Viewport coordinates are the pixel coordinates ultimately displayed on the screen, while normalized device coordinates are normalized coordinates within the range [-1, 1]. The viewport transformation matrix maps standardized device coordinates to the designated viewport area on the screen, ensuring that graphics display correctly. The viewport transformation allows objects in a 3D scene to be correctly rendered onto a 2D screen, and can be flexibly adjusted based on screen resolution and viewport size.

[0092] Specifically, the expression of the viewport transformation matrix is:

[0093]

[0094] Where W and H represent the width and height of the viewport, respectively. The viewport transformation matrix scales the x and y components of the normalized device coordinates to fit within the width and height of the viewport, respectively, while leaving the z component unchanged. In practice, the values ​​of W and H can be set based on the screen resolution and viewport size. For example, if the screen resolution is 1920 × 1080 pixels and the viewport size is the entire screen, then W = 1920 and H = 1080. This allows points in the normalized device coordinates to be correctly mapped to pixel locations on the screen.

[0095] Preferably, in order to further optimize the effect of the viewport transformation, an offset and a scaling factor can be introduced into the viewport transformation matrix. For example, if multiple viewports need to be displayed on the screen (such as a multi-view display), each viewport can be positioned to a different area of ​​the screen by adjusting the offset.

[0096] Furthermore, for special display requirements, such as non-square viewports or situations requiring non-linear scaling, this can be achieved by modifying the scaling factor in the viewport transformation matrix. For example, for a viewport with an aspect ratio of 16:9, this can be achieved by setting W = 1600 and H = 900. Furthermore, additional scaling factors can be introduced to adjust the viewport size, achieving more flexible display effects.

[0097] In some embodiments, the matching operation includes:

[0098] Model the target object as a three-dimensional sphere and calculate the viewport coordinates and mapping radius of its center.

[0099] Determine whether the target object is selected based on the distance between the mouse's viewport coordinates and the sphere's center viewport coordinates.

[0100] It's important to note that matching is achieved by modeling the target object as a 3D sphere and calculating the viewport coordinates of its center and its projection radius. Viewport coordinates represent the final display position of the target object on the screen, while the projection radius is the projection of the 3D sphere's radius onto the 2D viewport. By calculating the distance between the mouse position and the viewport coordinates of the target object's center, it's possible to determine whether the mouse has selected the target object. This method leverages the geometric properties of a 3D sphere, simplifying the complex 3D object into a spherical model and streamlining the matching process.

[0101] Specifically, the target object is modeled as a three-dimensional sphere, and the object coordinates of its center are converted to viewport coordinates after a series of coordinate transformations. The mapping radius is obtained by scaling the radius of the three-dimensional sphere onto the two-dimensional viewport according to the projection ratio. In practical applications, the radius of the three-dimensional sphere can be set according to the actual size of the target object, while the projection ratio is determined by the size of the viewport and the projection matrix. For example, if the actual radius of the target object is 1 unit length and the viewport size is 100×100 pixels, the projection ratio can be calculated based on the viewport size and the display ratio of the target object in the viewport. When the mouse moves on the screen, the program calculates the distance between the mouse position and the viewport coordinates of the center of the target object in real time, and determines whether the distance is less than the mapping radius.

[0102] Preferably, to improve matching accuracy and flexibility, the matching operation can be further optimized. For example, a tolerance range can be introduced to allow the distance between the mouse position and the viewport coordinates of the target object's center to fluctuate within a certain range, thereby avoiding matching failures caused by mouse jitter or screen resolution differences.

[0103] Furthermore, for irregularly shaped objects, multiple 3D spheres can be used for approximate modeling. The center and radius of each sphere can be adjusted based on the object's geometric characteristics. For example, for a rectangular object, three spheres can be used to approximate its dimensions in three principal directions, thereby improving matching accuracy.

[0104] In some embodiments, the mapping radius is obtained by scaling the radius of a three-dimensional sphere to a two-dimensional viewport by the projection ratio.

[0105] It should be noted that the mapping radius refers to scaling the radius of the 3D sphere to the size on the 2D viewport according to the projection ratio. This process is a key step in the matching operation, ensuring that the representation of the 3D target object on the 2D screen accurately reflects its size and position in 3D space. The mapping radius is calculated based on the parameters of the projection matrix and the planar viewport. By mapping the geometric information in 3D space onto the 2D plane, the conversion from 3D to 2D is achieved, providing an accurate basis for users to select the target object on the 2D screen.

[0106] Specifically, the calculation of the mapping radius involves the relationship between the radius of the three-dimensional sphere and the projection matrix. In three-dimensional space, the radius of the sphere is a fixed value, but in a two-dimensional viewport, its representation is affected by the projection method. For example, in perspective projection, the mapping radius of the sphere changes with the distance between the sphere and the camera. The farther the distance, the smaller the mapping radius. In orthogonal projection, the mapping radius remains unchanged. The setting of specific parameters needs to be determined according to the parameters of the projection matrix (such as field of view, aspect ratio, near plane and far plane distance, etc.) and the size of the viewport. For example, for a perspective projection with a field of view of 45 degrees, the radius of the sphere can be scaled to the viewport according to a certain ratio through the projection matrix to obtain the mapping radius.

[0107] To improve the accuracy and adaptability of the mapping radius calculation, some optimization measures can be introduced. For example, the position and orientation of the sphere in 3D space can be taken into account, and the projection matrix can be dynamically adjusted to optimize the mapping effect. For complex 3D scenes, a segmented mapping approach can be used to map different parts of the sphere to the viewport separately, thereby more accurately reflecting its shape and size in 3D space.

[0108] Furthermore, by introducing an error correction mechanism, the mapping radius can be fine-tuned to compensate for errors caused by projection distortion or differences in viewport resolution. For example, the actual size of the sphere at different distances on the viewport can be measured experimentally and the mapping radius calculation formula can be adjusted based on this data, thereby improving the accuracy and reliability of the matching operation.

[0109] In some embodiments, the conversion of the eye frame coordinates includes:

[0110] The camera position and viewing angle are adjusted by the affine transformation matrix M, where M is a 4×4 matrix containing the camera's rotation, translation, and scaling parameters.

[0111] It should be noted that eye frame coordinate conversion adjusts the camera's position and perspective using the affine transformation matrix M. Eye frame coordinates are coordinates in a coordinate system with the camera as the origin, and are used to describe the position and orientation of objects in a 3D scene relative to the camera. The affine transformation matrix M is a 4×4 matrix that contains the camera's rotation, translation, and scaling parameters. These parameters adjust the camera's position and perspective, thereby changing the display of the 3D scene on the 2D screen. This process is a key step in achieving perspective control in 3D graphics rendering, ensuring that users can observe the 3D scene from different angles.

[0112] Specifically, the structure of the affine transformation matrix M can be expressed as:

[0113]

[0114] Where R is a 3×3 rotation matrix that describes the camera's rotation angle and direction; T is a 3×1 translation vector that describes the camera's position; and 0 is a 1×3 zero matrix. In practice, the rotation matrix R can be set using Euler angles (such as pitch, yaw, and roll), and the translation vector T is determined based on the camera's position in the world coordinate system. For example, if you need to move the camera 1 unit to the right, you can set the translation vector T to

[0115]

[0116] In addition, the zoom parameter can be used to adjust the field of view of the camera, for example, to achieve different observation effects by zooming in or out of the scene.

[0117] Preferably, in order to achieve more flexible camera control, the affine transformation matrix M can be expanded or optimized. For example, quaternions can be introduced to represent rotations to avoid the universal lock problem caused by Euler angles, thereby achieving a smoother rotation effect. In addition, for complex three-dimensional scenes, the parameters of the affine transformation matrix M can be dynamically adjusted to achieve real-time movement and perspective switching of the camera. For example, the translation vector T and the rotation matrix R can be updated in real time through user input (such as mouse or keyboard operations), thereby achieving interactive three-dimensional scene browsing. In addition, the path planning algorithm can be combined to make the camera move along a preset path, providing users with a richer visual experience.

[0118] In some embodiments, the method further comprises:

[0119] In the depth test, the occlusion relationship of the target object is determined by the z component of the normalized device coordinates.

[0120] It should be noted that depth testing is an important technology used to determine the visibility of objects in three-dimensional graphics rendering. The z component of the normalized device coordinates (NDC) is used to determine the occlusion relationship of the target object. In a three-dimensional scene, multiple objects may overlap or occlude each other. The depth test ensures that only the surface of the object closest to the camera is rendered on the screen, thereby achieving the correct visual effect. Normalized device coordinates are normalized coordinates, and their z component represents the depth information of the object in the viewing cone, usually ranging from [0,1], where 0 represents the near plane and 1 represents the far plane. Through depth testing, the occlusion problem in the three-dimensional scene can be effectively solved, and the realism and efficiency of rendering can be improved.

[0121] Specifically, the depth test process is to determine whether it is occluded by comparing the depth value of each pixel. During the rendering process, the depth value of each pixel is stored in the depth buffer (also known as the Z-Buffer). When a new pixel is rendered, its depth value is compared with the depth value of the corresponding position in the depth buffer. If the new depth value is less than the value in the depth buffer, it means that the new pixel is closer to the camera and should be rendered to the screen, and the value in the depth buffer should be updated; otherwise, the new pixel is occluded and will not be rendered.

[0122] Furthermore, in practical applications, the precision of the depth buffer (such as 16-bit, 24-bit, or 32-bit) affects the accuracy of depth testing. For example, for complex 3D scenes, using a higher-precision depth buffer can reduce the problem of Z-fighting, thereby improving rendering quality.

[0123] Preferably, in order to further optimize the effect of depth testing, some advanced techniques can be used. For example, depth bias can be used to resolve depth conflicts caused by floating-point precision issues. Depth bias adds a small offset to the depth value to ensure that certain objects (such as meshes or transparent objects) do not conflict with adjacent objects when rendering.

[0124] Furthermore, for transparent objects, sorted rendering can be used, rendering opaque objects first, followed by transparent objects in order from far to near, to correctly handle the occlusion relationship of transparent objects. Furthermore, hardware-accelerated depth testing can be combined to improve rendering efficiency, such as using the graphics processing unit (GPU)'s depth buffer to automatically perform depth testing operations.

[0125] In some embodiments, the method is applied to a pipeline detection system to match a target object photo on a two-dimensional screen with the actual position of the pipeline.

[0126] It should be noted that the method of the present invention is particularly suitable for pipeline inspection systems, used to match images of target objects on a two-dimensional screen with the actual location of the pipeline. Pipeline inspection systems often need to quickly locate and identify target objects, such as foreign objects or damaged parts within the pipeline, within complex three-dimensional pipeline environments. The method of the present invention accurately maps the position of the target object in three-dimensional space onto a two-dimensional screen, thereby helping inspectors intuitively identify and locate the target object. This method not only improves inspection efficiency but also reduces misidentifications caused by mismatches in viewing angle and position.

[0127] Specifically, target objects in pipeline inspection systems are usually detected through image recognition technology, and corresponding photos are taken for users to identify. In a three-dimensional scene, the position of the target object is represented by its three-dimensional coordinates, and these coordinates need to be mapped to the viewport coordinates on the two-dimensional screen through a series of transformations. In actual applications, the pipeline inspection system can set corresponding parameters according to the geometric shape and size of the pipeline, such as the viewport size, the parameters of the projection matrix (such as the field of view angle, aspect ratio, near plane and far plane distance, etc.), to ensure that the display effect of the target object on the screen meets actual needs. For example, for longer pipelines, the field of view angle can be appropriately adjusted to obtain a wider field of view, and the depth test function can be used to ensure that the occlusion relationship of the target object is correctly displayed.

[0128] Preferably, to further improve the practicality and accuracy of the pipeline inspection system, the method of the present invention can be optimized. For example, a real-time feedback mechanism can be introduced. When a user selects a target object on a two-dimensional screen, the system can immediately display the target object's three-dimensional position in the pipeline and provide relevant inspection data, such as the target object's size and material.

[0129] Furthermore, for complex pipeline networks, segmented inspection can be used to divide the pipeline into multiple areas, allowing for detection and location of target objects separately, thereby improving system processing efficiency. Virtual reality (VR) or augmented reality (AR) technology can also be combined to provide inspectors with a more intuitive 3D visualization interface, further enhancing inspection effectiveness and user experience.

[0130] The aforementioned embodiments of the present invention have the following beneficial effects: By simulating the rendering pipeline of graphics hardware through software, the present invention can achieve precise mapping from three-dimensional target objects to two-dimensional screen coordinates, effectively resolving the difficulty users face in accurately aligning target objects on a two-dimensional screen with their actual three-dimensional positions in complex three-dimensional scenes. This method can be widely applied to engineering fields requiring precise positioning of three-dimensional objects, such as pipeline inspection systems, helping users quickly and accurately locate target objects and improve work efficiency. Through a series of coordinate transformations and matching operations, users can select a target object on a two-dimensional screen, accurately locate it in actual three-dimensional space, and obtain a corresponding photo of the target object.

[0131] Furthermore, the present invention utilizes depth testing to determine the occlusion relationship of target objects using the z component of normalized device coordinates, further optimizing the visualization of 3D scenes and enhancing the user experience. By modeling the target object as a 3D sphere and calculating the viewport coordinates of its center and the mapping radius, precise matching and selection of the target object can be achieved. This method not only improves the accuracy and convenience of locating target objects in 3D scenes but also plays a vital role in various applications requiring 3D object positioning.

[0132] Example 2

[0133] like Figure 1 As shown in the figure, after the hardware pipeline obtains the object coordinates of the object vertex, the vertex shader transforms it into world coordinates, then transforms it into eye frame coordinates, and finally projects it into clipping coordinates; after the clip operation, the pipeline transforms it into normalized device coordinates; after the cull operation, the pipeline finally transforms it into viewport coordinates.

[0134] like Figure 2 As shown in the figure, the object frame is represented in green, the world frame is represented in red, and the eye frame is represented in blue. When constructing a 3D object, the 3D coordinates of its vertices are based on the object frame of the 3D object itself, while the relationships between objects are based on the world frame. Therefore, the object coordinates must be transformed into world coordinates.

[0135] Assume that the object frame is The world frame is O is exist The coordinate matrix in : [Right now for Affine transformation of : ].

[0136] If you need to perform affine transformation on the object, the affine transformation matrix is:

[0137]

[0138] Among them, L implements linear transformation [such as rotate, scale], and T implements nonlinear [translate] transformation, so

[0139] If a vertex of the object exist The coordinates in are c: so exist The coordinates in are OAc, and the above matrix can be constructed by software to calculate The world frame coordinates OAc.

[0140] Converting a 3D scene into a 2D image requires placing a camera somewhere in the scene.

[0141] Figure 3 The image shows a 3D scene from the camera's position (the origin of the eye frame) and the direction of sight. Therefore, the position of each object in the scene in the 2D image depends on its position relative to the camera in the 3D scene. In computer graphics, a right-handed standard orthogonal frame is used to simulate the camera, denoted as the eye frame. E is exist The coordinate matrix in : [Right now for Affine transformation of : ].

[0142] If you need to move the eye to a different position and viewing direction, you need to perform an affine transformation on the eye frame. If the affine transformation matrix is ​​M, then If a vertex of the object exist The coordinate in is c, so so exist The coordinates in are (EM) -1 OAc.

[0143] The above matrix can be constructed by software and calculated Eye frame coordinates (EM) -1 OAc.

[0144] The transformation from the 3D eye frame coordinates of the object's vertex to the 2D clipping coordinates on the projection plane is achieved by using the perspective projection matrix.

[0145] First determine the field of view, such as Figure 4 As shown, in the projection plane z e =-1 upper bound y e In scope Then the final normalized device coordinate y nWill be mapped to the range (-1,1).

[0146] After determining the field of view angle θ, the aspect ratio α of the projection plane can be determined as required to obtain the eye frame coordinates of the top-bottom-left-right projection plane. The result is as follows: Figure 5 shown.

[0147] At this point, the projection matrix can be obtained:

[0148]

[0149] After projection operation, clipping coordinates and normalized device coordinates can be obtained:

[0150] Get z c =

[0151] 1, w n =-z e ,∴ When z e When z approaches 0, n tends to infinity, and when z e When away from eye, z n It approaches 0. Under the limited precision of computers, the distances of such vertices will become difficult to compare, causing the subsequent depth test to fail. It is necessary to specify the front and back distance limits for the projection process, that is, the near-far plane. The distances relative to the eye are n and f, which are two negative values, such as Figure 6 shown.

[0152] Depend on Figure 5 and Figure 6 The top-bottom-left-right positions and the near-far plane shown here constitute the view frustum used for projection. This gives us the complete form of the projection matrix:

[0153] in

[0154] Determines the position and aspect ratio of the projection plane; Determines the shift effect of the projection plane [if no shift is done, r = -l, t = -b, both elements are 0]; So that z nThe final effect of the projection operation is to map the view frustum in the eye frame to the canonical cube in the normalized device frame. The normalized device coordinates of the upper right corner are The normalized device coordinates of the lower left corner are

[0155] For example, the eye frame coordinates of the upper right corner of the view frustum are The projection result is

[0156] The eye frame coordinates of the lower left corner of the view frustum are The projection result is

[0157] To sum up, we can get a vertex of the object Clipping coordinates:

[0158]

[0159] The above matrix can be constructed by software to obtain The clipping coordinates.

[0160] like Figure 7 As shown, evenly spaced pixels on the film plane may correspond to non-uniformly distributed points in the eye frame, so the clipping coordinates Must be transformed into normalized device coordinates Map it to the canonical cube, and the coordinates of its upper right corner are The coordinates of the lower left corner are This step is obtained by dividing the first three terms of the clipping coordinates (position coordinates) by the fourth term (homogeneous coordinates), that is, The results are as follows Figure 8 shown.

[0161] Normalized device coordinates are device-independent abstract unit coordinates that must be converted to actual viewport coordinates before they can be displayed in the color buffer. Therefore, this step of transformation is to map the canonical cube in the normalized device frame to a rectangle on the window screen, such as Figure 7 Shown is a 12pixel×8pixel window viewport.

[0162] The transformation must be done in the vertical and horizontal directions to do appropriate scaling and shifting. According to the geometric mapping relationship, if the height and width of the windowviewport are W and H, the transformation matrix can be obtained as follows:

[0163]

[0164] So we can calculate:

[0165]

[0166] in That is, a vertex of the object The viewport coordinates, and z w Will be written to the depth buffer for subsequent depth test operations.

[0167] The above matrix can be constructed by software to obtain The viewport coordinates.

[0168] In practical applications, the detected target object can be simulated as a sphere in a 3D graphic. The object frame coordinates of the sphere's center are the 3D coordinates to be matched. After the above series of transformations, the viewport coordinates are obtained. Following the same principle, the radius of the sphere in the object frame can be mapped to the viewport matching radius on the 2D screen.

[0169] When you move the mouse, the program calculates the distance based on the viewport coordinates of the mouse and the sphere center viewport coordinates of the target object and the matching radius, and can accurately match the target object selected by the mouse. The actual matching effect is as follows Figure 10 shown. Figure 10 In the middle right picture, the selected target object is marked with a cube.

[0170] The above descriptions are merely some preferred embodiments of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features having similar functions disclosed in the embodiments of the present invention.

Claims

1. A method for screen positioning of a target object in a three-dimensional scene, characterized in that: The following steps are involved: By simulating the rendering pipeline of graphics hardware through software, the 3D object coordinates of the target object are converted into world coordinates, eye frame coordinates, clipping coordinates, normalized device coordinates and viewport coordinates in sequence; Based on the viewport coordinates of the target object, a matching operation is performed on the two-dimensional screen to determine the actual three-dimensional position of the target object selected by the user; Wherein, the coordinate transformation includes: (a) Convert object coordinates to world coordinates using an affine transformation matrix A, where the affine transformation matrix A contains a linear transformation submatrix L and a translation transformation submatrix T; (b) Convert the world coordinates to eye frame coordinates through the affine transformation matrix M; (c) Convert the eye frame coordinates to clipping coordinates through the projection matrix P; (d) Convert the clipping coordinates to normalized device coordinates through normalization operation; (e) Convert the normalized device coordinates to viewport coordinates using the viewport transformation matrix.

2. The method according to claim 1, characterized in that The expression of the affine transformation matrix A is: Where L is a 3×3 linear transformation matrix, T is a 3×1 translation vector, and 0 is a 1×3 zero matrix.

3. The method according to claim 1, characterized in that The expression of the projection matrix P is: Where n is the near plane distance, f is the far plane distance, l, r, b, and t are the left, right, bottom, and top boundary values ​​of the projection plane, respectively.

4. The method according to claim 1, wherein The normalization operation includes: Divide the x, y, and z components of the clipping coordinates by their homogeneous w components to obtain the normalized device coordinates.

5. The method according to claim 1, characterized in that The expression of the viewport transformation matrix is: Where W is the viewport width and H is the viewport height.

6. The method according to claim 1, characterized in that The matching operation includes: Model the target object as a three-dimensional sphere and calculate the viewport coordinates and mapping radius of its center. Determine whether the target object is selected based on the distance between the mouse's viewport coordinates and the sphere's center viewport coordinates.

7. The method according to claim 6, characterized in that The mapping radius is obtained by scaling the radius of the 3D sphere to the 2D viewport by the projection ratio.

8. The method according to claim 1, characterized in that The eye frame coordinate conversion includes: The camera position and viewing angle are adjusted by the affine transformation matrix M, where M is a 4×4 matrix containing the camera's rotation, translation, and scaling parameters.

9. The method according to claim 1, characterized in that The method further comprises: In the depth test, the occlusion relationship of the target object is determined by the z component of the normalized device coordinates.

10. The method according to any one of claims 1 to 9, characterized in that: The method is applied to a pipeline detection system and is used to match a target object photo on a two-dimensional screen with the actual position of the pipeline.

Images