Method and device for displaying object contour in panorama

Abstract

The embodiment of the invention discloses an object contour display method and device in a panorama, and the method comprises the steps: obtaining a three-dimensional model of a target object in a preset scene under a world coordinate system; determining a conversion matrix between a panoramic cube chartlet of a panoramic image corresponding to the preset scene and the world coordinate system; performing coordinate alignment on the three-dimensional model and the panorama based on the conversion matrix, and embedding the three-dimensional model into the panorama; rendering the three-dimensional model based on color information in the panorama to obtain a target three-dimensional model embedded in the panorama; performing dynamic lighting effect rendering on the target three-dimensional model, and displaying at least one geometric surface contour in the target three-dimensional model; according to the embodiment, each point position does not need to be identified respectively, and the object identification cost is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method and apparatus for displaying the outline of objects in a panoramic image. Background Technology

[0002] Image-based object contour delineation has become a common requirement, and many image-based object segmentation methods exist for this purpose. However, all these image-based object recognition methods can only perform object recognition on a single image. For the same object in multiple images, it needs to be recognized separately, meaning the number of object recognition attempts equals the number of images. In panoramic VR, the same object may be captured at multiple points. Therefore, when it is necessary to identify the contour of this object in a panoramic image, it is necessary to recognize the object in the panoramic image at all points, leading to increased costs. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method and apparatus for displaying object outlines in a panoramic image.

[0004] According to one aspect of the present disclosure, a method for displaying the outline of an object in a panoramic image is provided, including: Obtain a 3D model of the target object in the preset scene in the world coordinate system; Determine the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system; Based on the transformation matrix, the 3D model and the panoramic image are aligned in coordinates, and the 3D model is embedded into the panoramic image; The 3D model is rendered based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image. Dynamic lighting effects are applied to the target 3D model to display the outline of at least one geometric surface in the target 3D model.

[0005] Optionally, determining the transformation matrix between the panoramic cubemap of the panoramic image corresponding to the preset scene and the world coordinate system includes: Obtain the camera pose parameters of the acquisition points corresponding to the panoramic image; Based on the camera pose parameters, determine the transformation matrix between the panoramic cube map of the panoramic image and the world coordinate system.

[0006] Optionally, aligning the 3D model with the panoramic image based on the transformation matrix and embedding the 3D model into the panoramic image includes: Based on the transformation matrix, the coordinates in the panoramic image are transformed to the world coordinate system; In the world coordinate system, each coordinate point corresponding to the 3D model is matched to the panoramic image, and the 3D model is embedded in the panoramic image.

[0007] Optionally, rendering the 3D model based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image includes: Construct at least one spatial ray vector based on the fragment shader; Based on the intersection of the at least one spatial ray vector and the three-dimensional model, the color information corresponding to the three-dimensional model is determined; The 3D model is rendered based on the determined color information to obtain a target 3D model embedded in the panoramic image.

[0008] Optionally, the step of dynamically rendering the target 3D model with lighting effects to display the outline of at least one geometric surface in the target 3D model includes: Determine the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model; the first scanning plane and the second scanning plane are the two extreme planes of the target 3D model with the target scanning direction as the normal vector. Dynamic lighting effects are rendered on the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, displaying at least one geometric surface contour in the target 3D model.

[0009] Optionally, determining the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model includes: Principal component analysis is performed on the normal vector corresponding to at least one vertex of the surface of the target 3D model. Based on the first two principal component directions obtained from the principal component analysis, a plane in the target 3D model is determined as the scanning principal plane. The target scanning direction is determined based on the main scanning plane; The first scanning plane and the second scanning plane are determined based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model.

[0010] Optionally, determining the target scanning direction based on the main scanning plane includes: Determine at least one triangular facet corresponding to the main scanning plane; The target scanning direction is determined based on at least one normal vector corresponding to the at least one triangular facet.

[0011] Optionally, determining the first scanning plane and the second scanning plane based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model includes: Determine the vector formed by each surface point in the target 3D model and the center point of the target 3D model; The projection of the vector corresponding to each surface point onto the target scanning direction is determined to obtain multiple projection values; The first scanning plane and the second scanning plane are determined based on the first set of surface points corresponding to the set of maximum values ​​in the projected values ​​and the second set of surface points corresponding to the set of minimum values ​​in the projected values.

[0012] Optionally, the step of dynamically rendering the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, to display at least one geometric surface contour in the target 3D model, includes: Based on the difference between the first scanning plane and the second scanning plane in the target scanning direction, the scanning speed and the light band width are determined; The light band with the width of the light band moves on the target 3D model at the scanning speed to complete the dynamic light effect rendering and display the outline of at least one geometric surface in the target 3D model.

[0013] According to another aspect of the present disclosure, a device for displaying the outline of an object in a panoramic image is provided, comprising: The model acquisition module is used to obtain a 3D model of the target object in the preset scene in the world coordinate system. The transformation matrix module is used to determine the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system. A coordinate alignment module is used to align the coordinates of the 3D model and the panoramic image based on the transformation matrix, and to embed the 3D model into the panoramic image; The model embedding module is used to render the 3D model based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image. The lighting effect rendering module is used to perform dynamic lighting effect rendering on the target 3D model, displaying the outline of at least one geometric surface in the target 3D model.

[0014] Optionally, the transformation matrix module is specifically used to obtain the camera pose parameters of the acquisition points corresponding to the panoramic image; and based on the camera pose parameters, to determine the transformation matrix between the panoramic cube map of the panoramic image and the world coordinate system.

[0015] Optionally, the coordinate alignment module is specifically used to transform the coordinates in the panoramic image to the world coordinate system based on the transformation matrix; match each coordinate point corresponding to the 3D model to the panoramic image in the world coordinate system, and embed the 3D model into the panoramic image.

[0016] Optionally, the model embedding module is specifically used to construct at least one spatial ray vector based on a fragment shader; determine the color information corresponding to the three-dimensional model based on the intersection of the at least one spatial ray vector and the three-dimensional model; and render the three-dimensional model based on the determined color information to obtain a target three-dimensional model embedded in the panoramic image.

[0017] Optionally, the light effect rendering module includes: The scanning parameter unit is used to determine the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model; the first scanning plane and the second scanning plane are two extreme planes of the target 3D model with the target scanning direction as the normal vector. The contour display unit is used to perform dynamic light effect rendering on the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, and to display the contour of at least one geometric surface in the target 3D model.

[0018] Optionally, the scanning parameter unit is specifically used to perform principal component analysis on the normal vector corresponding to at least one vertex of the surface of the target 3D model, determine a plane in the target 3D model as the scanning principal plane based on the first two principal component directions obtained from the principal component analysis; determine the target scanning direction based on the scanning principal plane; and determine the first scanning plane and the second scanning plane based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model.

[0019] Optionally, when determining the target scanning direction based on the main scanning plane, the scanning parameter unit is used to determine at least one triangular facet corresponding to the main scanning plane; and to determine the target scanning direction based on at least one normal vector corresponding to the at least one triangular facet.

[0020] Optionally, when determining the first scanning plane and the second scanning plane based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model, the scanning parameter unit is used to determine the vector formed by each surface point in the target 3D model and the center point of the target 3D model; determine the projection of the vector corresponding to each surface point on the target scanning direction to obtain multiple projection values; and determine the first scanning plane and the second scanning plane according to the first surface point set corresponding to the set of maximum values ​​of the projection values ​​and the second surface point set corresponding to the set of minimum values ​​of the projection values.

[0021] Optionally, the contour display unit is specifically used to determine the scanning speed and the light band width based on the difference between the first scanning plane and the second scanning plane in the target scanning direction; move the light band with the light band width on the target three-dimensional model according to the scanning speed to complete dynamic light effect rendering and display at least one geometric surface contour in the target three-dimensional model.

[0022] According to another aspect of the present disclosure, an electronic device is provided, comprising: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method for displaying object outlines in a panoramic image as described in any of the above embodiments.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method for displaying object outlines in a panoramic image as described in any of the above embodiments.

[0024] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method for displaying object outlines in a panoramic image as described in any of the above embodiments.

[0025] Based on the object outline display method and apparatus in the panoramic image provided in the above embodiments of this disclosure, a three-dimensional model of a target object in a preset scene in the world coordinate system is obtained; a transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system is determined; the three-dimensional model and the panoramic image are aligned in coordinates based on the transformation matrix, and the three-dimensional model is embedded in the panoramic image; the three-dimensional model is rendered based on the color information in the panoramic image to obtain a target three-dimensional model embedded in the panoramic image; dynamic lighting effect rendering is performed on the target three-dimensional model to display at least one geometric surface outline in the target three-dimensional model; this embodiment embeds the three-dimensional model into the panoramic image by aligning the coordinates of the three-dimensional model and the panoramic image, so that when the panoramic image recognizes the target object, it only needs to recognize the target three-dimensional model, without recognizing each point separately, thus reducing the object recognition cost; and this embodiment proposes dynamic lighting effect rendering of the target three-dimensional model to realize the display of the structural outline of the target object.

[0026] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0027] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0028] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 This is a flowchart illustrating a method for displaying object outlines in a panoramic image provided by an exemplary embodiment of this disclosure; Figure 2 This is a public announcement Figure 1 A flowchart of step 103 in the embodiment shown; Figure 3 This is a public announcement Figure 1 A flowchart illustrating step 104 in the illustrated embodiment; Figure 4 This is a public announcement Figure 1 A flowchart of step 105 in the illustrated embodiment is shown. Figure 5 This is a public announcement Figure 4 A flowchart of step 1051 in the embodiment shown; Figure 6 This is a schematic diagram of the structure of a panoramic image object outline display device provided in an exemplary embodiment of the present disclosure; Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0029] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0032] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0033] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0034] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0035] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0040] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0042] Application Overview In the process of realizing this disclosure, the inventors discovered that for the same object captured from multiple points in a panoramic image, the prior art requires the object to be identified separately for the images captured from multiple points, which leads to at least the following problems: multiple identifications of the same object will greatly increase the cost, and only the two-dimensional outline of the object can be identified, and the structure of the object cannot be represented.

[0043] Exemplary methods Figure 1 This is a flowchart illustrating a method for displaying object outlines in a panoramic image according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps: Step 101: Obtain the 3D model of the target object in the preset scene in the world coordinate system.

[0044] Optionally, the target object can be any object in a preset scene, such as a table or sofa in an indoor scene. A 3D model of the target object can be obtained based on the 3D point cloud corresponding to the preset scene. Optionally, semantic segmentation processing can be performed on the 3D model in the 3D point cloud of the preset scene based on any 3D object segmentation algorithm to obtain the axis-aligned bounding box (AABB) data of the target object in model space. This AABB is used as the 3D model of the target object in the world coordinate system. Alternatively, in this embodiment, the target object in the 3D point cloud corresponding to the preset scene can be obtained based on manual annotation. The 3D object segmentation algorithm can include, for example, a 3D segmentation algorithm based on hierarchical point cloud feature learning (PointNet++), or an efficient instance segmentation algorithm driven by bounding box regression (3D-BoNet).

[0045] Step 102: Determine the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system.

[0046] In one embodiment, the construction of the panoramic image includes acquiring images from multiple points in a preset scene using a camera, and then aligning and stitching the images based on the camera pose parameters corresponding to each point to obtain the panoramic image; therefore, the camera pose parameters of each of the multiple acquisition points that constitute the panoramic image of the preset scene are obtained; and a transformation matrix between the panoramic image cubetexture and three-dimensional world coordinates is established based on the camera pose parameters.

[0047] Step 103: Align the 3D model with the panoramic image based on the transformation matrix, and embed the 3D model into the panoramic image.

[0048] Optionally, after determining the transformation matrix between the two coordinate systems (panoramic image coordinate system and world coordinate system), an image in one coordinate system can be transformed to the other coordinate system based on the transformation matrix. For example, a panoramic image can be transformed to the world coordinate system, or a 3D model can be transformed to the panoramic image coordinate system.

[0049] Step 104: Render the 3D model based on the color information in the panoramic image to obtain the target 3D model embedded in the panoramic image.

[0050] This embodiment uses coordinate system transformation to place the panoramic image and the 3D model in the same coordinate system. At this time, the coordinates of the 3D model corresponding to the target object in the panoramic image can be determined. Based on these coordinates, the corresponding color information can be obtained from the panoramic image. Based on the color information, the 3D model can be rendered to obtain the target 3D model.

[0051] Step 105: Perform dynamic lighting effect rendering on the target 3D model to display the outline of at least one geometric surface in the target 3D model.

[0052] After determining the target 3D model to be embedded in the panoramic image, dynamic lighting effects are used to achieve cross-dimensional visualization of the target 3D model, presenting the 3D structural information of the target object with spatial depth on the 2D panoramic image.

[0053] This disclosure provides a method for displaying object outlines in a panoramic image, which obtains a 3D model of a target object in a preset scene in the world coordinate system; determines the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system; aligns the 3D model with the panoramic image based on the transformation matrix, and embeds the 3D model into the panoramic image; renders the 3D model based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image; and performs dynamic lighting effect rendering on the target 3D model to display at least one geometric surface outline in the target 3D model. This embodiment embeds the 3D model into the panoramic image by aligning the coordinates of the 3D model with the panoramic image, so that when the panoramic image identifies the target object, it only needs to identify the target 3D model, without needing to identify each point separately, thus reducing the object recognition cost. Furthermore, this embodiment proposes dynamic lighting effect rendering of the target 3D model, realizing the display of the structural outline of the target object.

[0054] In some alternative embodiments, step 102 includes: Obtain the camera pose parameters of the acquisition points corresponding to the panoramic image; Based on the camera pose parameters, determine the transformation matrix between the panoramic cubemap of the panoramic image and the world coordinate system.

[0055] The camera pose parameters can include translation parameters (describing the coordinates of the camera's optical center in the world coordinate system, i.e., the vector from the origin of the world coordinate system to the origin of the camera coordinate system) and rotation parameters (describing the rotation direction of the camera coordinate system relative to the world coordinate system, i.e., the camera's orientation (such as horizontal rotation, pitch, and roll)). After obtaining the camera pose parameters, optionally, by defining the fixed pose of the cube map virtual camera, its correspondence with the real camera pose can be established. The initial transformation matrix can be derived using the coordinate system transformation chain, and then the error can be reduced through multi-camera pose optimization. Finally, the transformation matrix between the cube map and the world coordinate system can be determined.

[0056] like Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 103 may include the following steps: Step 1031: Transform the coordinates in the panoramic image to the world coordinate system based on the transformation matrix.

[0057] Optionally, after determining the transformation matrix, each coordinate in the panoramic image can be substituted into the transformation matrix to transform the panoramic image into the world coordinate system.

[0058] Step 1032: Match each coordinate point corresponding to the 3D model to the panoramic image in the world coordinate system, and embed the 3D model into the panoramic image.

[0059] In this embodiment, when both the panoramic image and the 3D model are in the world coordinate system, the position of the 3D model in the panoramic image can be determined by matching and searching the coordinate points corresponding to the 3D model in the panoramic image. Embedding the 3D model into the panoramic image eliminates the need for separate identification at multiple points in the panoramic image for a target object. Instead, the 3D model corresponding to the target object is displayed in 3D within the panoramic image, reducing the display cost of the same object corresponding to multiple points in a preset scene. This embodiment achieves coordinate unification from the panoramic image space to the model space, ensuring geometric consistency between the boundary of the 3D object (corresponding to the 3D model) and the 2D panoramic projection, as well as consistency in light effect projection under different viewpoints.

[0060] like Figure 3 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 104 may include the following steps: Step 1041: Construct at least one spatial ray vector based on the fragment shader.

[0061] The fragment shader is a crucial stage in the computer graphics rendering pipeline, responsible for calculating the final color of each pixel. It is a core component for achieving visual effects such as lighting, texture mapping, and color blending. The input to the fragment shader comes from the interpolation results of the rasterization stage. Starting from the viewpoint center (the representation of the origin of the panoramic coordinate system in the world coordinate system; for example, if the preset scene is a room, a corner of the room can be used as the viewpoint center), multiple ray vectors are formed by connecting the interpolated world coordinate points.

[0062] Step 1042: Determine the color information corresponding to the three-dimensional model based on the intersection of at least one spatial ray vector and the three-dimensional model.

[0063] Based on obtaining multiple spatial ray vectors, all ray vectors intersecting the surface of the 3D model can be obtained. The coordinates of these intersection points can be used to determine their corresponding coordinates in the panoramic image. The corresponding color information can then be obtained from the coordinates in the panoramic image, which can be used as the color information for the 3D model's surface. Optionally, a samplerCube sampler (input parameters: ray and sampling direction, corresponding color) can be used to perform ray tracing sampling on the cube map, achieving a bidirectional mapping between panoramic image pixels and 3D spatial coordinates to obtain the color information of the coordinate points on the 3D model's surface. The samplerCube is a texture sampler specifically used for cube map sampling in graphics programming interfaces such as OpenGL and WebGL, defined and used in GLSL (Shader Language). Its core function is to sample a cube map composed of six orthogonal faces using 3D direction vectors, and it is widely used in scenarios such as environment mapping and panoramic rendering.

[0064] Step 1043: Render the 3D model based on the determined color information to obtain the target 3D model embedded in the panoramic image.

[0065] In this embodiment, the color information of each coordinate point on the surface of the 3D model is determined based on the color information in the panoramic image. The surface of the 3D model is rendered with this color information to obtain a colored target 3D model. This target 3D model is embedded in the panoramic image. Since the target 3D model is 3D, it can be viewed in the panoramic image from any perspective (corresponding to different points) without the need for re-identification, which greatly reduces the recognition cost of objects in the panoramic image.

[0066] like Figure 4 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 105 may include the following steps: Step 1051: Determine the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model.

[0067] Among them, the first scanning plane and the second scanning plane are two extreme planes of the target 3D model with the target scanning direction as the normal vector.

[0068] Optionally, the target 3D model includes multiple planes, each corresponding to a different angle. To more clearly display the structure of the target 3D model, it is necessary to obtain directions perpendicular to as many planes as possible as scanning directions. Optionally, the target scanning direction can be determined based on any existing technology, or the target scanning direction can be manually set. This embodiment defines the display of the target 3D model by determining the first and second scanning planes. The display extends from the first scanning plane to the second scanning plane without exceeding the target 3D model, overcoming the problem of blurred object boundaries.

[0069] Step 1052: Perform dynamic lighting effect rendering on the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, and display the outline of at least one geometric surface in the target 3D model.

[0070] Dynamic lighting effect rendering refers to the technique in computer graphics of simulating and rendering light that changes over time (movement of light sources, changes in light intensity / color, and light and shadow interactions caused by object movement, etc.). Its core objective is to dynamically update the lighting and shadow effects in a virtual scene over time to present a more realistic and immersive visual experience. In this embodiment, a light strip of a certain width can be used to move from the first scanning plane of the target 3D model along the target scanning direction to the second scanning plane to achieve dynamic lighting effect rendering. During the movement of the light strip, at least one geometric contour surface in the target 3D model is highlighted, making the structural outline of the target 3D model more clearly visible in the panoramic view, without being confused with other parts of the panoramic view.

[0071] like Figure 5 As shown above, in the above Figure 4 Based on the illustrated embodiment, step 1051 may include the following steps: Step 501: Perform principal component analysis on the normal vector corresponding to at least one vertex of the surface of the target 3D model, and determine a plane in the target 3D model as the scanning principal plane based on the first two principal component directions obtained from the principal component analysis.

[0072] Principal Component Analysis (PCA) is a linear transformation technique for data dimensionality reduction. It projects high-dimensional data into a low-dimensional space, preserving the main variance characteristics of the data while eliminating redundancy and noise. Principal components are ordered by variance, with the first principal component having the largest variance, and subsequent components exhibiting decreasing variance and orthogonality. This embodiment uses PCA to extract the two orthogonal directions (the first two principal component directions) with the largest variance from the vertex normal vector, defining the plane as the scanning principal plane. The vertex normal vector is a unit vector perpendicular to the surface where the vertex is located, used to describe the orientation of the model surface at that vertex.

[0073] Step 502: Determine the target scanning direction based on the main scanning plane.

[0074] Optionally, at least one triangular facet corresponding to the main scanning plane is determined; the target scanning direction is determined based on at least one normal vector corresponding to at least one triangular facet.

[0075] In this embodiment, the scanning master plane is decomposed into at least one triangular facet, and the normal vector of each triangular facet is determined. The target scanning direction is determined by weighted averaging of the at least one normal vector. The weight value in the weighted averaging is determined based on the area of ​​the corresponding triangular facet; triangular facets with larger areas have larger weight values, and triangular facets with smaller areas have smaller weight values. For example, the areas of all triangular facets are normalized using the area of ​​the largest triangular facet, and the normalized value is used as the weight value for each triangular facet. For example, if the areas of three triangular facets are 7 cm², ... 2 8cm 2 and 10cm 2 Normalization yields 0.7, 0.8, and 1, meaning the weights of the three triangular faces are 0.7, 0.8, and 1.

[0076] Step 503: Determine the first scanning plane and the second scanning plane based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model.

[0077] In some optional examples, the vector formed by each surface point in the target 3D model and the center point of the target 3D model is determined; the projection of the vector corresponding to each surface point onto the target scanning direction is determined to obtain multiple projection values; the first scanning plane and the second scanning plane are determined according to the first surface point set corresponding to the set of maximum values ​​in the projection values ​​and the second surface point set corresponding to the set of minimum values ​​in the projection values.

[0078] Optionally, a spatial relationship equation can be established between the scanning direction θ and the bounding box formed by the target 3D model, for example, as shown in the following formula (1): Formula (1) Where O is the center point of the bounding box; The scanning direction is denoted by ; P is any surface point of the target 3D model; P –O represents the vector from the center point O of the bounding box to point P; :vector The dot product of the vector (P - O), that is, the vector from point O to point P, is... The projection length in the direction; t is a preset value, and different scanning planes can be determined by setting different values ​​of t. The bounding box is in There are two extreme planes in the direction (the first scan plane and the second scan plane, corresponding to the maximum value t_max and the minimum value t_min of t, respectively), and all points P on the bounding box satisfy... That is, in this embodiment, by determining the values ​​of t as the maximum and minimum values, two sets of P points are determined to obtain the first scanning plane and the second scanning plane. For example, all P points of θ•(P - O) = t_max are obtained to obtain the first surface point set, and the first scanning plane is obtained from the first surface point set; the first scanning plane is obtained from the first surface point set. • The set of points P (P - O) = t_min is used to obtain the second surface point set, and the second scanning plane is obtained from the second surface point set. This embodiment, by determining the scanning direction and limiting the first and second scanning planes, ensures that the scanning light band always moves along the main structural direction of the target object, thus enhancing the visual guidance effect.

[0079] In some alternative embodiments, step 1052 may include: The scanning speed and light band width are determined based on the difference between the first and second scanning planes in the target scanning direction.

[0080] Optionally, along the scanning direction By performing ray intersection on the bounding box of the target 3D model, the intersection points of the ray with the first scanning plane (entry point t_start) and the second scanning plane (exit point t_end) can be obtained. The difference between the two intersection points can determine the total height (or length, width) of the target 3D model being scanned. In order to improve the dynamic rendering effect, a certain proportion of the total height of the scan is used as the light band width, for example, 1 / 10 of the difference is used as the light band width. The scanning speed is determined in combination with the total height of the scan, so that the time to scan the entire target 3D model is controlled within a preset range, that is, the scanning speed is equal to the difference divided by the preset scanning time.

[0081] The light band, with a width corresponding to the scanning speed, moves across the target 3D model to complete dynamic lighting effect rendering and display the outline of at least one geometric surface in the target 3D model.

[0082] Optionally, the smoothstep function is used to achieve the transition at the edge of the light band, where the smoothstep function is used to create a smooth transition effect. It generates a smooth S-shaped curve in the interval from 0 to 1. For example, a smooth curve is obtained based on the smoothstep function shown in the following formula (2): Formula (2) Where fragPos is the position vector of the currently rendered pixel in 3D space (e.g., the coordinates of a point on the surface of the target 3D model); float intensity represents the brightness of the current position (i.e., fragPos) in the dynamic scanning light band; dot is the unit vector of the scanning direction (e.g., along the x-axis, diagonally at 45°, etc.), which determines the direction of movement of the light band; fragPos) denotes the vector dot product, which is the result of fragPos in the vector dot product. Projected length along the direction (intuitively, this value represents the length of the point along the scanning direction) The "distance" from the origin is used to determine the position of a point in the scanning direction. For the light band width parameter, 2 It is the total width of the light band ( The larger the value, the wider the light band. This width parameter can be set according to the scene or determined according to a certain proportion of the length of the target 3D model in the scanning direction. This is the "center position" parameter of the light band, and The light band changes linearly with time, causing it to move along... Directional movement; optionally, = (t_current - t_start) / (t_end - t_start); where t_current is the current point in the scanning direction. The projected coordinates on the surface, where t_start is the scan direction vector. The intersection point with the first scanning plane, where t_end is the scanning direction vector. The intersection point with the second scanning plane; this formula is implemented based on different values ​​of the current point. The linear variation over time ensures the continuous movement of the scanning light band on the surface of the target 3D model. Furthermore, the movement of the light band reveals the subtle variations in brightness and darkness across the complex geometric surfaces of the target 3D model.

[0083] Any of the object outline display methods provided in the embodiments of this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the object outline display methods provided in the embodiments of this disclosure can be executed by a processor, such as by a processor executing any of the object outline display methods mentioned in the embodiments of this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0084] Exemplary device Figure 6 This is a schematic diagram of the structure of a panoramic image object outline display device provided in an exemplary embodiment of this disclosure. For example... Figure 6 As shown, the apparatus provided in this embodiment may include: Model acquisition module 61 is used to obtain a 3D model of the target object in the preset scene in the world coordinate system; Transformation matrix module 62 is used to determine the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system; The coordinate alignment module 63 is used to align the coordinates of the 3D model and the panoramic image based on the transformation matrix, and to embed the 3D model into the panoramic image; The model embedding module 64 is used to render the 3D model based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image. The light effect rendering module 65 is used to perform dynamic light effect rendering on the target 3D model and display at least one geometric surface outline in the target 3D model.

[0085] The panoramic image object outline display device provided in the above embodiments of this disclosure obtains a three-dimensional model of a target object in a preset scene in the world coordinate system; determines the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system; aligns the three-dimensional model with the panoramic image based on the transformation matrix, and embeds the three-dimensional model into the panoramic image; renders the three-dimensional model based on the color information in the panoramic image to obtain a target three-dimensional model embedded in the panoramic image; performs dynamic lighting effect rendering on the target three-dimensional model to display at least one geometric surface outline in the target three-dimensional model; this embodiment embeds the three-dimensional model into the panoramic image by aligning the coordinates of the three-dimensional model with the panoramic image, so that when the panoramic image recognizes the target object, it only needs to recognize the target three-dimensional model, without recognizing each point separately, and this embodiment proposes to perform dynamic lighting effect rendering on the target three-dimensional model, realizing the display of the structural outline of the target object.

[0086] In some optional embodiments, the transformation matrix module 62 is specifically used to obtain the camera pose parameters of the acquisition points corresponding to the panoramic image; based on the camera pose parameters, to determine the transformation matrix between the panoramic cube map of the panoramic image and the world coordinate system.

[0087] In some optional embodiments, the coordinate alignment module 63 is specifically used to transform the coordinates in the panoramic image to the world coordinate system based on the transformation matrix; and to match each coordinate point corresponding to the 3D model to the panoramic image in the world coordinate system, thereby embedding the 3D model into the panoramic image.

[0088] In some optional embodiments, the model embedding module 64 is specifically used to construct at least one spatial ray vector based on a fragment shader; determine the color information corresponding to the three-dimensional model based on the intersection of the at least one spatial ray vector and the three-dimensional model; and render the three-dimensional model based on the determined color information to obtain a target three-dimensional model with an embedded panoramic image.

[0089] In some optional embodiments, the lighting effect rendering module 65 includes: The scanning parameter unit is used to determine the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model; the first scanning plane and the second scanning plane are two extreme planes of the target 3D model with the target scanning direction as the normal vector. The contour display unit is used to dynamically render the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, and to display the contour of at least one geometric surface in the target 3D model.

[0090] Optionally, the scanning parameter unit is specifically used to perform principal component analysis on the normal vector corresponding to at least one vertex of the surface of the target 3D model, determine a plane in the target 3D model as the scanning principal plane based on the first two principal component directions obtained from the principal component analysis, determine the target scanning direction based on the scanning principal plane, and determine the first scanning plane and the second scanning plane based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model.

[0091] Optionally, when determining the target scanning direction based on the scanning principal plane, the scanning parameter unit is used to determine at least one triangular facet corresponding to the scanning principal plane; and to determine the target scanning direction based on at least one normal vector corresponding to at least one triangular facet.

[0092] Optionally, when determining the first and second scanning planes based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model, the scanning parameter unit is used to determine the vector formed by each surface point in the target 3D model and the center point of the target 3D model; determine the projection of the vector corresponding to each surface point on the target scanning direction to obtain multiple projection values; and determine the first and second scanning planes according to the first set of surface points corresponding to the set of maximum values ​​in the projection values ​​and the second set of surface points corresponding to the set of minimum values ​​in the projection values.

[0093] Optionally, the contour display unit is specifically used to determine the scanning speed and light band width based on the difference between the first scanning plane and the second scanning plane in the target scanning direction; move the light band with the light band width on the target 3D model according to the scanning speed to complete dynamic light effect rendering and display the contour of at least one geometric surface in the target 3D model.

[0094] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0095] Exemplary electronic devices Below, for reference Figure 7 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0096] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0097] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0098] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0099] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the panoramic image object outline display method described above in the various embodiments of this disclosure, and / or other desired functions.

[0100] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0101] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0102] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0103] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0104] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods for displaying object outlines in panoramic images according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0105] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0106] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for displaying object outlines in panoramic images according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0107] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0108] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0110] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0111] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0112] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0113] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0114] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for displaying the outline of an object in a panoramic image, characterized in that, include: Obtain a 3D model of the target object in the preset scene in the world coordinate system; Determine the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system; Based on the transformation matrix, the 3D model and the panoramic image are aligned in coordinates, and the 3D model is embedded into the panoramic image; The 3D model is rendered based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image. Dynamic lighting effects are applied to the target 3D model to display the outline of at least one geometric surface in the target 3D model.

2. The method according to claim 1, characterized in that, The step of determining the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system includes: Obtain the camera pose parameters of the acquisition points corresponding to the panoramic image; Based on the camera pose parameters, determine the transformation matrix between the panoramic cube map of the panoramic image and the world coordinate system.

3. The method according to claim 1 or 2, characterized in that, The step of aligning the 3D model with the panoramic image based on the transformation matrix and embedding the 3D model into the panoramic image includes: Based on the transformation matrix, the coordinates in the panoramic image are transformed to the world coordinate system; In the world coordinate system, each coordinate point corresponding to the 3D model is matched to the panoramic image, and the 3D model is embedded in the panoramic image.

4. The method according to any one of claims 1-3, characterized in that, The process of rendering the 3D model based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image includes: Construct at least one spatial ray vector based on the fragment shader; Based on the intersection of the at least one spatial ray vector with the three-dimensional model, the color information corresponding to the three-dimensional model is determined; The 3D model is rendered based on the determined color information to obtain a target 3D model embedded in the panoramic image.

5. The method according to any one of claims 1-4, characterized in that, The dynamic lighting effect rendering of the target 3D model, displaying the outline of at least one geometric surface in the target 3D model, includes: Determine the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model; the first scanning plane and the second scanning plane are the two extreme planes of the target 3D model with the target scanning direction as the normal vector. Dynamic lighting effects are rendered on the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, displaying at least one geometric surface contour in the target 3D model.

6. The method according to claim 5, characterized in that, Determining the target scanning direction, the first scanning plane, and the second scanning plane corresponding to the target 3D model includes: Principal component analysis is performed on the normal vector corresponding to at least one vertex of the surface of the target 3D model. Based on the first two principal component directions obtained from the principal component analysis, a plane in the target 3D model is determined as the scanning principal plane. The target scanning direction is determined based on the main scanning plane; The first scanning plane and the second scanning plane are determined based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model.

7. The method according to claim 6, characterized in that, Determining the target scanning direction based on the main scanning plane includes: Determine at least one triangular facet corresponding to the main scanning plane; The target scanning direction is determined based on at least one normal vector corresponding to the at least one triangular facet.

8. The method according to claim 6 or 7, characterized in that, The step of determining the first scanning plane and the second scanning plane based on the vector formed by each surface point in the target 3D model and the center point of the target 3D model includes: Determine the vector formed by each surface point in the target 3D model and the center point of the target 3D model; The projection of the vector corresponding to each surface point onto the target scanning direction is determined to obtain multiple projection values; The first scanning plane and the second scanning plane are determined based on the first set of surface points corresponding to the set of maximum values ​​in the projected values ​​and the second set of surface points corresponding to the set of minimum values ​​in the projected values.

9. The method according to any one of claims 5-8, characterized in that, The step of dynamically rendering the target 3D model from the first scanning plane to the second scanning plane according to the target scanning direction, and displaying the contour of at least one geometric surface in the target 3D model, includes: Based on the difference between the first scanning plane and the second scanning plane in the target scanning direction, the scanning speed and the light band width are determined; The light band with the width of the light band moves on the target 3D model at the scanning speed to complete the dynamic light effect rendering and display the outline of at least one geometric surface in the target 3D model.

10. A device for displaying the outline of an object in a panoramic image, characterized in that, include: The model acquisition module is used to obtain a 3D model of the target object in the preset scene in the world coordinate system. The transformation matrix module is used to determine the transformation matrix between the panoramic cube map of the panoramic image corresponding to the preset scene and the world coordinate system. A coordinate alignment module is used to align the coordinates of the 3D model and the panoramic image based on the transformation matrix, and to embed the 3D model into the panoramic image; The model embedding module is used to render the 3D model based on the color information in the panoramic image to obtain a target 3D model embedded in the panoramic image. The lighting effect rendering module is used to perform dynamic lighting effect rendering on the target 3D model, displaying the outline of at least one geometric surface in the target 3D model.

Images