Display method, device, display, medium and product

Abstract

The application provides a display method, device, display, medium and product. The method comprises: determining a pose relationship of an observer viewpoint relative to a display surface; generating a display image for presentation on the display surface according to the pose relationship; wherein a presentation viewing angle of scene content in the display image matches the pose relationship; and displaying the display image on the display surface. The technical solution of the embodiment of the application determines the pose relationship of the observer viewpoint relative to the display surface, and generates a display image matching the viewing angle of the observer based on the pose relationship, so as to dynamically adjust the display content according to the pose of the observer, so that the presentation viewing angle of the scene content in the display image changes with the change of the observer viewpoint, and the interaction between the display content and the behavior of the observer is realized. The sense of reality of the display content is enhanced, and the immersive experience of the observer is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a display method, device, display, medium, and product. Background Technology

[0002] Traditional display methods typically only display static images, meaning the content is presented in a fixed way. This results in a lack of realism and an inability to interact with the observer's behavior, making it difficult for the observer to have an immersive experience. Summary of the Invention

[0003] In view of this, the embodiments of this application are committed to providing a display method, device, display, medium and product to enhance the realism of the displayed content and improve the immersive experience of the observer.

[0004] In a first aspect, one embodiment of this application provides a display method, comprising: determining the pose relationship of an observer's viewpoint relative to a display surface; generating a display image for presentation on the display surface based on the pose relationship; wherein the presentation angle of the visual content in the display image matches the pose relationship; and displaying the display image on the display surface.

[0005] In conjunction with the first aspect, in some implementations of the first aspect, generating a display image for presentation on a display surface based on pose relationships includes: constructing a virtual surface model that matches the shape of the display surface; mapping the view content onto the virtual surface model based on pose relationships to form an image adapted to the shape of the display surface; and using the image adapted to the shape of the display surface as the display image.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the visual content is mapped onto a virtual surface model according to the pose relationship to form an image adapted to the shape of the display surface, including: determining a virtual viewpoint according to the pose relationship; and projecting a three-dimensional scene containing the visual content from the virtual viewpoint onto the virtual surface model to form an image adapted to the shape of the display surface.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, projecting a 3D scene containing visual content onto a virtual surface model from a virtual viewpoint to form an image adapted to the shape of the display surface includes: determining the texture coordinates of the virtual surface model; reconstructing the spatial coordinates of the virtual surface model based on the texture coordinates; mapping the spatial coordinates to corresponding positions in the 3D scene based on the virtual viewpoint to establish a mapping relationship between the spatial coordinates and the 3D scene; and sampling colors from the 3D scene based on the mapping relationship to generate an image adapted to the shape of the display surface.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, before projecting a 3D scene containing visual content onto a virtual surface model from a virtual viewpoint to form an image adapted to the shape of the display surface, the method further includes: performing validity checks on points on the virtual surface model based on the virtual viewpoint, and removing points that fail the validity check.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the validity detection includes at least one of the following: detecting whether a point on the virtual surface model is located within the visible space of the virtual viewpoint; and detecting whether the normal of the surface to which the point on the virtual surface model belongs is oriented toward the virtual viewpoint.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the view content is mapped onto a virtual surface model according to the pose relationship to form an image that matches the shape of the display surface, including: projecting the view content onto the virtual surface model according to the pose relationship; and unfolding the surface of the projected virtual surface model to obtain an image that matches the shape of the display surface.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the surface of the projected virtual surface model is unfolded to obtain an image that matches the shape of the display surface, including: determining the correspondence between the texture coordinates of points on the virtual surface model and the planar coordinates after the surface is unfolded; sampling colors from the projected virtual surface model, and generating an image that matches the shape of the display surface based on the correspondence and the sampled color values.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the shape of the virtual surface model is scaled proportionally to the shape of the display surface.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the display surface is the screen of the display.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, an image acquisition device is deployed on the display; determining the pose relationship of the observer's viewpoint relative to the display surface includes: acquiring a facial image of the observer through the image acquisition device, determining the observer's facial features; and determining the pose relationship of the observer's viewpoint relative to the display surface based on the facial features.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the pose relationship includes orientation information and distance of the observer's viewpoint relative to the display surface.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, facial features include the observer's facial center and facial feature scale; the facial feature scale is used to characterize the geometric correspondence between the observer's face and the facial image; based on the facial features, the pose relationship of the observer's viewpoint relative to the display surface is determined, including: based on the facial center, determining the orientation information of the observer's viewpoint relative to the display surface; and based on the facial feature scale, determining the distance between the observer and the display surface.

[0017] In conjunction with the first aspect, some implementations of the first aspect also include: performing scale compensation on the orientation information based on the facial feature scale to eliminate the orientation calculation deviation caused by the change in distance between the observer and the display surface, and obtaining the compensated orientation information.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, determining the orientation information of the observer's viewpoint relative to the display surface based on the face center includes: determining the pixel offset of the face center relative to the face image center; calculating a first deflection angle of the observer in the horizontal direction and a second deflection angle in the vertical direction based on the pixel offset and the field of view of the image acquisition device; and composing the orientation information by the first deflection angle and the second deflection angle.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, determining the distance between the observer and the display surface based on the facial feature scale includes: determining the ratio of a reference scale to the facial feature scale; wherein the reference scale is the feature scale of the facial image when the distance between the observer and the display surface is a standard distance; and calculating the distance between the observer and the display surface based on the ratio and the focal length of the image acquisition device.

[0020] In conjunction with the first aspect, some implementations of the first aspect further include: acquiring a real environment image that is occluded by the display surface; displaying the display image on the display surface, including: fusing the display image with the real environment image, and displaying the fused display image on the display surface.

[0021] Secondly, this application provides a display device, comprising: a relationship determination module for determining the pose relationship of an observer's viewpoint relative to a display surface; an image generation module for generating a display image for presentation on the display surface based on the pose relationship; wherein the presentation angle of the visual content in the display image matches the pose relationship; and an image display module for displaying the display image on the display surface.

[0022] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the method in the first aspect or any possible implementation of the first aspect.

[0023] Fourthly, one embodiment of this application provides a display, the display comprising: a screen; and a processor for performing the method in the first aspect or any possible implementation thereof.

[0024] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on a display, cause the display to implement the method in the first aspect or any possible implementation of the first aspect.

[0025] The technical solution of this application determines the pose relationship between the observer's viewpoint and the display surface, and generates a display image that matches the observer's viewing angle based on the pose relationship. This dynamically adjusts the display content according to the observer's pose, so that the presentation angle of the visual content in the display image changes with the change of the observer's viewpoint, realizing the interaction between the display content and the observer's behavior. This solves the problem of fixed display content and lack of realism in traditional display methods, enhances the realism of the display content, and improves the observer's immersive experience.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 The diagram shown is a flowchart of a display method provided in an embodiment of this application.

[0029] Figure 2 The image shown is a schematic diagram of the main scene provided in an embodiment of this application.

[0030] Figure 3 The diagram shown is a schematic diagram of a virtual cylindrical mesh model provided in an embodiment of this application.

[0031] Figure 4 The image shown is a two-dimensional image obtained after unfolding according to an embodiment of this application.

[0032] Figure 5 The diagram shown is a schematic diagram of the structure of a display device provided in an embodiment of this application.

[0033] Figure 6The diagram shown is a structural schematic of a display provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0035] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0036] It should be understood that the term "and / or" used in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Additionally, the term "based on" used in this document is not limited to relying solely on one object. For example, determining B based on A can mean: determining B based solely on A, or determining B partially based on A.

[0037] Before introducing the technical solution, the application scenarios can be illustrated by example. The display method of this application can be applied to various display scenarios in which display images are displayed on a display surface. These display scenarios include, but are not limited to: curved screen displays, flat screen displays, automotive displays, head-mounted displays, augmented reality displays, holographic displays, wall projections, ground projections, and transparent displays. The display surface can be any shape used to present the display image, including planes, curved surfaces, cylindrical surfaces, spherical surfaces, and irregular free-form surfaces. The visual content to be presented through the display image can be any type of visual content, such as virtual characters, 3D models, 2D images, video streams, and any combination thereof.

[0038] In traditional display methods, the visual content presented by the displayed image is usually only a static image. That is, the display position, viewing angle, and perspective of the content are fixed and cannot be dynamically adjusted according to the observer's real-time position or behavior. This static display method results in a lack of realism in the image, making it difficult for the observer to have an immersive experience and to interact naturally with the displayed content.

[0039] For example, when an observer tilts their head to view a 3D scene displayed on a curved screen at different angles, the image output by traditional display methods remains fixed and cannot match the changing perspectives caused by different viewing positions, thus reducing immersion. Similarly, in in-vehicle display scenarios, when occupants in different seats view the same screen, the displayed content cannot provide an adapted viewing angle based on their respective positions, affecting the user experience.

[0040] Based on this, the display method proposed in this application can dynamically adjust the presentation angle of the visual content in the displayed image according to the observer's actual viewing posture, so that the content presented in the displayed image matches the current observer's viewpoint, thereby enhancing interaction with the observer and making the viewing experience more realistic and immersive. The following is a combination of... Figures 1 to 4 The display method provided in the embodiments of this application will be described in detail.

[0041] Figure 1 The diagram shown is a flowchart illustrating a display method provided in an embodiment of this application; as follows: Figure 1 As shown, the method includes the following steps.

[0042] Step S110: Determine the pose relationship between the observer's viewpoint and the display surface.

[0043] Here, the observer's viewpoint refers to the observer's position when observing the display surface, which can be indirectly represented based on feature points extracted from a face image. The pose relationship refers to the position and orientation information of the observer's viewpoint relative to the display surface in three-dimensional space. This pose relationship describes the spatial relative position between the observer's viewpoint and the display surface. The pose relationship can be based on six degrees of freedom parameters of a specific coordinate system, or it can be simplified to include the orientation information and / or distance of the observer's viewpoint relative to the display surface.

[0044] A display surface refers to any physical surface that can be used to display images. Its shape can be a regular plane, a sphere, or an irregular curved surface. There are many types of display surfaces. For example, display surfaces can be electronic displays (such as curved display screens, flat panel display screens, and head-mounted display screens), projection surfaces (such as projection screens, walls, floors, desktops, and projectors), transparent display surfaces (such as car windows, shop windows, and mirrors), or holographic display media (such as holographic films, photopolymer films, and holographic waveguides). It should be noted that the shape and type of the display surface can be determined according to the actual display scenario to meet the different display needs of various scenarios.

[0045] In specific implementations, the methods for determining pose relationships include at least one of the following: acquiring facial images of the observer using an image acquisition device and determining pose relationships by analyzing facial features; measuring the distance between the observer and the display surface using a depth sensor; measuring the distance and triangulating the observer's viewpoint using ultrasonic or lidar; obtaining the observer's eye orientation and gaze point using an eye-tracking device; and performing 3D reconstruction using multi-camera stereo vision to determine the spatial position of the observer's viewpoint, thereby determining the pose relationship. These various pose determination methods can be used individually or in combination to improve the accuracy and robustness of pose detection.

[0046] Step S120: Generate a display image for presentation on the display surface based on the pose relationship; wherein the presentation view of the visual content in the display image is matched with the pose relationship.

[0047] The visual content refers to the content intended to be presented to the observer, including at least one of virtual objects, virtual characters, images, videos, background environments, and user interface elements. The viewing angle refers to the angle from which the visual content is viewed within the displayed image. The displayed image is a two-dimensional image used to display on a display plane.

[0048] It should be noted that matching the presentation viewpoint with the pose can include various situations. For example, the presentation viewpoint can be completely consistent with the pose, meaning the visual content in the displayed image is presented from the perspective of whichever direction the observer is viewing. For instance, when the observer moves to the right, the right side of an object is visible, while the left side is obscured.

[0049] Alternatively, a pre-defined mapping relationship can be established between the presentation viewpoint and the pose, which can be used to convert the pose relationship into a corresponding presentation viewpoint. For example, some key content in the visual scene needs to be within the observer's field of vision or presented to the observer from a fixed, optimal perspective. Therefore, a mapping relationship can be used to ensure that the presentation angle of the visual scene is mapped to a fixed angle regardless of the observer's viewing direction, ensuring that the key content faces the observer from the best angle. For example, in a navigation scenario, regardless of the observer's viewing direction, ensure that the navigation sign faces the observer directly; in a video call scenario, ensure that the face of the person being spoken to is presented from the front; and in an advertising display scenario, ensure that the main display surface of the product faces the observer directly.

[0050] In practice, there are several ways to generate and display images. For example, a 3D scene containing visual content can be constructed, and a virtual camera can be set up to render the image based on the pose relationship. Alternatively, the pose relationship can be input into a generative model trained by machine learning to obtain a display image that meets the display viewpoint requirements. Or, based on pre-stored visual materials from multiple viewpoints, the visual material that matches the pose relationship can be selected and fused to obtain the display image.

[0051] Step S130: Display the image on the display surface.

[0052] In actual display, the corresponding display method can be adopted according to the type of display surface. For example, for an electronic display screen, the display image is written to the screen buffer pixel by pixel; for a projection carrier surface, the display image is sent to the projector and projected onto the projection carrier surface; for a holographic display medium, the display image is encoded into a hologram and then output.

[0053] The technical solution of this application determines the pose relationship between the observer's viewpoint and the display surface, and generates a display image that matches the observer's viewing angle based on the pose relationship. This dynamically adjusts the display content according to the observer's pose, so that the presentation angle of the visual content in the display image changes with the change of the observer's viewpoint, realizing the interaction between the display content and the observer's behavior. This solves the problem of fixed display content and lack of realism in traditional display methods, enhances the realism of the display content, and improves the observer's immersive experience.

[0054] To better meet the adaptation requirements of non-planar display surfaces and improve the matching accuracy of the presentation viewpoint, some implementations generate a display image for presentation on the display surface based on the pose relationship. This includes: constructing a virtual surface model that matches the shape of the display surface; mapping the view content onto the virtual surface model based on the pose relationship to form an image that matches the shape of the display surface; and using the image that matches the shape of the display surface as the display image.

[0055] Virtual surface models refer to any digital model that can mathematically or graphically describe the geometry of a physical display surface, used to simulate the geometry of the display surface.

[0056] In practice, a virtual surface model corresponding to the real surface can be constructed by using the stored geometric parameters of the display surface; alternatively, a virtual surface model can be obtained by real-time scanning and reconstruction of the display surface using a 3D scanning device.

[0057] It should be noted that matching the shape of the virtual surface model with the display surface means that there is a geometric correspondence between the two, so that the visual content projected onto the virtual surface model can be correctly adapted to the actual display surface.

[0058] To simplify calculations and reduce implementation complexity, in some embodiments, the shape of the virtual surface model is scaled proportionally to the shape of the display surface. Specifically, proportional scaling means that the virtual surface model and the display surface have the same geometry, with only the overall size scaled by a fixed ratio. For example, the scaling ratio between the virtual surface model and the display surface can be 1:1. Virtual surface models established according to proportional scaling relationships can achieve a geometric correspondence between the virtual surface model and the display surface through simple scaling operations, eliminating the need for complex transformations and simplifying calculations while improving processing efficiency.

[0059] In other embodiments, the geometric correspondence between the two can also be any one or a combination of topological equivalence, conformal mapping, and surface fitting. Those skilled in the art will understand that as long as the actual display position of the corresponding physical display surface can be determined from any position of the virtual surface model through geometric correspondence, it falls under the category of the virtual surface model and display surface shape matching as described in this application.

[0060] In some embodiments, the method of mapping the visual content onto a virtual surface model according to the pose relationship to form an image adapted to the shape of the display surface is as follows: a multi-view image set is pre-constructed; the multi-view image set includes images corresponding to the visual content acquired from different viewpoints; according to the observer's pose relationship, the image corresponding to the viewpoint closest to the pose relationship is found from the multi-view image set, and the image is mapped onto the virtual surface model to form an image adapted to the shape of the display surface.

[0061] It should be noted that by mapping the visual content onto the virtual surface model, the generated display image is adapted to the geometry of the virtual surface model, which in turn ensures that the display image is adapted to the actual display surface. Thus, even when the display surface is non-planar, the geometric distortion caused by the non-planar shape can be offset, ensuring that the perspective relationship of the final presented visual content is correct and there will be no deformation or distortion, further enhancing the realism of the display effect.

[0062] For example, if the display surface is a cylinder, in practical applications, a virtual cylindrical mesh model (virtual surface model) scaled proportionally to that cylinder will be created in another spatial location within the 3D engine. For instance, it can be scaled at a 1:1 ratio. Then, the 3D scene containing visual content (such as a virtual character) is projected onto the inner surface of this virtual cylindrical mesh model, based on the observer's current pose. This digitally corrects image distortion, resulting in a pre-distorted image displayed on the cylindrical screen that appears normal and undistorted from the observer's perspective.

[0063] This application embodiment constructs a virtual surface model that matches the shape of the display surface and maps the view content onto the virtual surface model, so that the generated display image is adapted to the shape of the display surface. When the display surface is non-planar, this method is used to obtain a display plane, which solves the problem of image distortion caused by directly displaying view content on a non-planar display surface and ensures that the view content can be displayed accurately.

[0064] In other embodiments, the method of mapping the visual content onto a virtual surface model based on the pose relationship to form an image adapted to the shape of the display surface is as follows: a virtual viewpoint is determined based on the pose relationship; from the virtual viewpoint, a three-dimensional scene containing the visual content is projected onto the virtual surface model to form an image adapted to the shape of the display surface.

[0065] The virtual viewpoint is a simulated virtual observation position in a 3D scene based on the observer's pose. The 3D scene is a 3D space containing the visual content.

[0066] Specifically, based on the pose relationship between the observer's viewpoint and the display surface, the observer's viewpoint is transformed into a virtual viewpoint in virtual space through coordinate transformation, ensuring that the relative position of the virtual viewpoint and the virtual surface model is consistent with the relative position of the observer's viewpoint and the display surface. Then, the 3D scene is observed from this virtual viewpoint, and the observed 3D scene image is projected onto the virtual surface model like a projector, thus obtaining an image adapted to the shape of the display surface.

[0067] This real-time rendering-based approach eliminates the need for pre-constructing multi-view image sets, offering greater flexibility. Furthermore, by setting virtual viewpoints based on pose relationships and projecting the 3D scene onto a virtual surface model according to these virtual viewpoints, the observer's viewpoint can be accurately integrated into the image generation process. This ensures that the generated image possesses the correct perspective and stereoscopic feel that matches the observer's viewpoint, thereby enhancing the immersive visual experience.

[0068] To accurately execute the process of projecting a 3D scene onto a virtual surface model and address the issue of inaccurate positioning during projection, this application also provides a projection method. Specifically, from a virtual viewpoint, a 3D scene containing the visual content is projected onto a virtual surface model to form an image adapted to the shape of the display surface. This includes: determining the texture coordinates of the virtual surface model; reconstructing the spatial coordinates of the virtual surface model based on the texture coordinates; mapping the spatial coordinates to corresponding positions in the 3D scene based on the virtual viewpoint to establish a mapping relationship between the spatial coordinates and the 3D scene; and sampling colors from the 3D scene based on the mapping relationship to generate an image adapted to the shape of the display surface.

[0069] Texture coordinates refer to the positional markers used to map a 2D image onto the surface of a 3D model. For example, when the display surface is a cylindrical surface, the texture coordinates can be the UV coordinates of the vertices of the cylindrical surface mesh, where the U coordinate corresponds to the normalized angle value in the circumferential direction and the V coordinate corresponds to the normalized height value in the vertical direction, both ranging from [0, 1]. Spatial coordinates refer to the coordinates of points on the virtual surface model in 3D space, which can be 3D coordinates in the world coordinate system or coordinates in the local coordinate system of the virtual surface model itself.

[0070] In practice, the projection process is not forward rasterization, but rather a precise calculation method using inverse mapping. Based on the virtual viewpoint, spatial coordinates are mapped to their corresponding positions in the 3D scene to establish a mapping relationship between spatial coordinates and the 3D scene. The specific steps are as follows: First, determine the texture coordinates of the virtual surface model by traversing the texture coordinates of each mesh vertex on its surface. Then, based on the texture coordinates, use geometric formulas to inversely reconstruct the spatial coordinates (e.g., world coordinates in the world coordinate system) of each point on the virtual surface model in 3D space. Simultaneously, determine the view matrix of the virtual viewpoint based on the pose relationship. Based on the view matrix and projection matrix of the virtual viewpoint, transform the reconstructed spatial coordinates to clip space, which is an intermediate coordinate space in the 3D graphics rendering pipeline. In clip space, coordinates are represented in homogeneous coordinates. The coordinates of a point on the virtual surface model within clip space can be used to determine whether the point is located within the view frustum. If it is within the view frustum, it is retained; otherwise, it is discarded.

[0071] Furthermore, perspective division is performed on the homogeneous coordinates of each point retained in clipping space to convert them into normalized device coordinates. Finally, the normalized device coordinates are mapped to two-dimensional texture coordinates through mathematical transformation. The resulting two-dimensional texture coordinates are used to represent the position of the 3D scene rendered by the virtual viewpoint, thereby mapping the spatial coordinates to the corresponding positions in the 3D scene and establishing a mapping relationship between the spatial coordinates and the 3D scene.

[0072] Furthermore, based on the mapping relationship, the method for sampling colors from the 3D scene to generate an image adapted to the shape of the display surface is as follows: For the spatial coordinates corresponding to each point on the virtual surface model, the texture coordinates of the 3D scene are obtained according to the mapping relationship. Color is then directly sampled from the 3D scene using these texture coordinates to obtain the color value corresponding to that point. All sampled color values ​​are stored according to the texture coordinates of the corresponding points on the virtual surface model, thus obtaining a 2D image adapted to the shape of the display surface.

[0073] For example, for a virtual cylindrical mesh model, its world coordinates are constructed based on the UV coordinates (UV.x, UV.y) of at least one point on its surface. Specifically, UV.x is used to calculate the circumferential angle, and UV.y is used to calculate the height value, thereby reconstructing the world coordinates of the corresponding point on the virtual cylindrical mesh model. Next, using the view matrix and projection matrix corresponding to the virtual viewpoint, the reconstructed world coordinates are mapped to the clipping space corresponding to the virtual viewpoint; and further converted to normalized device coordinates, new texture coordinates are obtained based on the normalized device coordinates. Thus, the mapping relationship between the world coordinates of the corresponding point on the virtual cylindrical mesh model and the point on the 3D scene is established. Finally, for at least one point on the virtual cylindrical mesh model, the corresponding point can be found in the 3D scene based on this mapping relationship, and the color value of that point can be collected; based on all collected color values, an image adapted to the shape of the display surface is generated.

[0074] The embodiments of this application accurately establish the mapping relationship between spatial coordinates and three-dimensional scene, which can ensure that each point on the virtual surface model can accurately correspond to the correct position in the three-dimensional scene; and, based on the mapping relationship, accurately sample colors from the three-dimensional scene and assign the color values ​​to the corresponding points on the virtual surface model, thereby improving the projection accuracy and ensuring that accurate display images can be generated subsequently.

[0075] To avoid erroneous display content (such as back perspective or artifacts) caused by projecting onto the back of the virtual surface model or areas outside the field of view during the projection process from a virtual viewpoint to a virtual surface model, embodiments of this application also provide an optimization scheme. Specifically, before projecting a 3D scene containing visual content onto the virtual surface model from a virtual viewpoint to form an image adapted to the shape of the display surface, the method further includes: performing validity detection on points on the virtual surface model based on the virtual viewpoint, and removing points that fail the validity detection.

[0076] It should be noted that validity testing refers to a preliminary screening and judgment process performed on the elements in the target object set before a graphics processing operation is executed. Its purpose is to exclude elements that do not meet specific geometric or optical constraints in order to avoid errors or unexpected results in subsequent processing.

[0077] In practice, each grid vertex on the virtual surface can be validated. If it passes the validation, the grid vertex is retained for subsequent projection calculations. If it fails, the grid vertex is invalid and is removed. In subsequent calculations, the invalid vertex is not sampled for color and is directly assigned a default color (such as black) or marked as transparent.

[0078] By performing validity checks on points on the surface of the virtual model and removing invalid points before projection, points that should not be seen by the observer can be filtered out in advance. This solves the problem of displaying back views or incorrect perspective content during projection, ensuring that only visible content that is allowed to be seen is projected, thereby avoiding image errors.

[0079] In some embodiments, validity detection includes at least one of the following: detecting whether a point on a virtual surface model is located within the visible space of a virtual viewpoint; and detecting whether the normal of the surface to which the point on the virtual surface model belongs is oriented toward the virtual viewpoint.

[0080] The visible space refers to the area of ​​the virtual surface that can be seen from a virtual viewpoint.

[0081] To improve convenience, the normalized device coordinates of points on the virtual surface model can be checked in normalized device coordinate space. For example, check if the z-coordinate (depth) in the normalized device coordinates is greater than 0. If z > 0, the point is in front of the virtual viewpoint and belongs to the visible space; if z ≤ 0, the point is behind the virtual viewpoint (i.e., on an occluded surface) and is invisible, failing the validity check. Furthermore, check if the x-coordinate and y-coordinate in the normalized device coordinates are within the range [-1, 1]. If they exceed this range, the point is outside the horizontal or vertical field of view boundary of the virtual viewpoint and is also invisible, failing the validity check.

[0082] Additionally, it can detect whether the normal of the surface where the point is located is facing the virtual viewpoint. If it is facing the virtual viewpoint, it means that the surface where the point is located is facing the observer and should be seen, thus passing the validity test. If it is not facing the virtual viewpoint, it means that the surface where the point is located is not facing the observer and should not be seen, thus failing the validity test.

[0083] For example, the normal vector of a point on the virtual surface model and the line vector connecting the point to the virtual viewpoint can be calculated first; the dot product of the normal vector and the line vector can be determined, and whether the normal is oriented towards the virtual viewpoint can be determined by judging whether the dot product is greater than 0.

[0084] Alternatively, it can be checked whether the final projected 2D texture coordinates are within the valid range of [0, 1]. These 2D texture coordinates represent the position of the 3D scene rendered by the virtual viewpoint. If they exceed the range of [0, 1], the content corresponding to that point is outside the sampleable area of ​​the 3D scene, and is therefore deemed to have failed the validity check.

[0085] This application embodiment detects whether a point is within the visible space and whether its normal resembles a virtual viewpoint. It can determine the visibility of a point from two key dimensions: position and orientation. This allows for a more comprehensive screening of invalid points, ensuring that the displayed image does not contain visual errors.

[0086] To convert an image projected onto a virtual surface model into a planar image that can be displayed on a display surface, embodiments of this application also provide an unfolding method to unfold the image mapped onto the virtual surface model into an image that can be displayed on a physical screen. Specifically, mapping the view content onto the virtual surface model according to the pose relationship to form an image adapted to the shape of the display surface includes: projecting the view content onto the virtual surface model according to the pose relationship; and unfolding the surface of the projected virtual surface model to obtain an image adapted to the shape of the display surface.

[0087] In practice, after the visual content is projected onto the virtual surface model, each point on the projected virtual surface model already has color values ​​sampled from the 3D scene attached to it. However, at this point, the virtual surface model is still a 3D model and cannot be directly output to a 2D display surface for display. Therefore, it is necessary to unfold its surface into a 2D planar image.

[0088] In some embodiments, the coordinate positions of each point on the virtual surface model on a two-dimensional plane can be calculated based on the geometric parameters of the virtual surface model. Then, the color values ​​attached to each point are filled into the coordinates of the plane to obtain a two-dimensional image adapted to the shape of the display surface. This image can be directly output to a physical display device for display.

[0089] This application embodiment performs a two-step process of first projecting onto the model and then unfolding the model, accurately mapping complex three-dimensional surface texture information onto a two-dimensional plane, ensuring that the resulting two-dimensional image can be directly displayed normally on the display surface after output.

[0090] To address the issue of color mismatch during the unfolding process, this application also provides an optimization scheme. Optionally, the surface of the projected virtual surface model is unfolded to obtain an image that matches the shape of the display surface. This includes: determining the correspondence between the texture coordinates of points on the virtual surface model and the planar coordinates of the unfolded surface; sampling colors from the projected virtual surface model; and generating an image that matches the shape of the display surface based on the correspondence and the sampled color values.

[0091] In practice, the correspondence between the texture coordinates and planar coordinates of points on the virtual surface model can be determined based on the geometry of the virtual surface model.

[0092] Furthermore, for each planar pixel that needs to be output, based on the correspondence established above, the corresponding mesh vertex on the virtual surface model is found and its color is sampled; the color value of the sampled mesh vertex is read and assigned to the planar pixel, thus generating the final unfolded two-dimensional image.

[0093] For example, when the virtual surface model is a virtual cylindrical mesh model, for each mesh vertex on the mesh of the virtual surface model, the shader calculates the angle θ of the mesh vertex in the circumferential direction using the arctangent function and normalizes it to U coordinates; using the y coordinate of the mesh vertex, its position in the height direction is calculated and normalized to V coordinates. In this way, the mesh vertex obtains a unique texture coordinate. Then, the texture coordinate is directly linearly mapped to normalized device coordinates. It should be noted that, to adapt to the texture coordinate system conventions of different graphics engines (such as Godot), the mapped coordinates can also be flipped along the Y-axis. Based on the normalized device coordinates obtained from the mapping, the mesh vertices on the virtual surface model can be positioned to the corresponding positions in the output image, that is, a correspondence is established between the texture coordinates of points on the virtual surface model and the planar coordinates after the surface is unfolded. In the fragment shader stage, the mesh vertex corresponding to the planar pixel to be output can be determined according to this correspondence. The color is directly sampled from the projected virtual surface model according to the texture coordinates of the mesh vertex, and then the sampled color value is assigned to the planar pixel to obtain the unfolded 2D output image.

[0094] This application embodiment establishes a correspondence between texture coordinates and unfolded planar coordinates, thus finding the correct position of the color of each point on the model surface in the final unfolded image. This solves the problem of color mismatch during the unfolding process, accurately maps the image content projected on the virtual surface model onto the plane, and ensures that the generated image is distortion-free and color-accurate.

[0095] The display methods in the foregoing embodiments can have their display surfaces applied to various displays. Optionally, the display surface is the screen of the display. The screen can be a flexible screen adapted to the corresponding display shape, or it can be multiple rigid display units spliced ​​together to form a corresponding curved surface shape.

[0096] In practical applications, there are various ways to determine the pose relationship of an observer's viewpoint relative to a display surface. In some embodiments, an image acquisition device is deployed on the display; determining the pose relationship of the observer's viewpoint relative to the display surface includes: acquiring a facial image of the observer through the image acquisition device, determining the observer's facial features; and determining the pose relationship of the observer's viewpoint relative to the display surface based on the facial features.

[0097] Among them, facial images are two-dimensional image data containing the observer's face, directly captured by an image acquisition device, which can be deployed in the center of the top border of the screen. Image acquisition devices include visible light cameras, infrared cameras, depth cameras, structured light cameras, etc.

[0098] Facial features refer to any geometric information extracted from an observer's facial image used to calculate viewpoint pose. For example, facial features include the facial center determined by the coordinates of facial keypoints (such as the nose and corners of the mouth), facial dimensions, and facial feature scales. Facial feature scales characterize the geometric correspondence between the observer's face and the facial image. Facial feature scales are calculated based on the pixel distances between multiple facial feature points (such as inter-eye distance, inter-corner-of-mouth distance, and the vertical distance from the eyes to the mouth). For example, the pixel distance between two facial feature points in the acquired facial image can be directly used as the facial feature scale; alternatively, the average of the pixel distances between multiple facial feature points can be calculated as the facial feature scale.

[0099] In practice, after determining facial features, a preset pose estimation algorithm is used to calculate the pose relationship between the observer's viewpoint and the display surface in three-dimensional space. This pose estimation algorithm may include two-dimensional image processing based on a monocular camera, three-dimensional point cloud registration algorithms based on a depth camera, and eye-tracking processing based on infrared light reflection.

[0100] For example, a cylindrical display has a screen that is a cylindrical organic light-emitting diode (OLED) panel. A visible light camera is integrated into the bezel or below the screen of this display, serving as an image acquisition device. The optical axis of this camera is aligned with the central axis of the display, enabling stable imaging of the observer's face. Facial features are determined from the image of the observer acquired by this image acquisition device, and then pose relationships are calculated.

[0101] This application embodiment integrates an image acquisition device on the display and processes the acquired facial images, eliminating the need for the observer to wear additional tracking devices. It can non-contactly determine the observer's facial features and accurately determine the pose relationship between the observer's viewpoint and the display surface in real time.

[0102] In some embodiments, the pose relationship includes orientation information and distance of the observer's viewpoint relative to the display surface. For example, the orientation information can be further decomposed into a first yaw angle in the horizontal direction and a second yaw angle in the vertical direction, while the distance is the straight-line distance from the center of the observer's face to the center of the display surface. By explicitly decomposing the pose relationship into two components, orientation and distance, the three-dimensional spatial state of the observer's viewpoint can be comprehensively and quantitatively described, solving the problem of incomplete pose description and facilitating subsequent accurate projection calculations based on this pose relationship.

[0103] To accurately determine pose relationships from a two-dimensional image, embodiments of this application specifically define the observer's facial features. Specifically, facial features include the observer's facial center and facial feature scale; the facial feature scale is used to characterize the geometric correspondence between the observer's face and the facial image; based on the facial features, determining the pose relationship of the observer's viewpoint relative to the display surface includes: determining the orientation information of the observer's viewpoint relative to the display surface based on the facial center; and determining the distance between the observer and the display surface based on the facial feature scale.

[0104] In some embodiments, the facial center is determined by fusing multiple feature points. For example, the average of the coordinates of the nose, the left corner of the mouth, and the right corner of the mouth is used as the coordinates of the facial center. This method is more stable than using a single feature point to determine the facial center and is less affected by changes in facial expressions or partial occlusion.

[0105] In some embodiments, the deflection and pitch angles of the observer's viewpoint relative to the display surface can be directly calculated using the facial center, the intrinsic parameters of the image acquisition device itself, and the mounting parameters on the display surface. Furthermore, based on the facial feature scale, the straight-line distance between the observer and the display surface can be determined using the principle of similar triangles in pinhole camera imaging. It should be noted that a larger facial feature scale indicates a closer observer to the display surface, while a smaller facial feature scale indicates a farther observer.

[0106] This application embodiment utilizes the face center to solve the orientation and the facial feature scale to solve the distance, which reduces the complexity of pose calculation, improves the speed and accuracy of pose calculation, and can meet the performance requirements of real-time rendering and display.

[0107] To eliminate orientation calculation deviations caused by changes in observer distance, the display method provided in this application further includes: performing scale compensation on the orientation information based on facial feature scales to eliminate orientation calculation deviations caused by changes in the distance between the observer and the display surface, thereby obtaining compensated orientation information.

[0108] It's important to note that due to perspective effects, objects appear larger when closer to the screen than when farther away. When an observer is closer to the screen, the same head rotation at a given angle will result in a greater pixel shift in the image compared to when they are farther away. Without compensation, the angle calculated directly from this pixel shift will be too large. Therefore, scale compensation for orientation information is necessary.

[0109] Scale compensation refers to dynamically correcting orientation information to eliminate calculation errors caused by distance changes.

[0110] In practice, after determining the orientation information based on the facial center, scale compensation can be performed on the orientation information according to the facial feature scale. For example, if the facial feature scale is the pixel distance between two facial feature points on the facial image, the ratio of the reference scale to the facial feature scale is used as the scale compensation coefficient. The orientation information is scaled proportionally based on this scale compensation coefficient, which can eliminate the orientation calculation deviation introduced by the distance change and obtain the compensated orientation information.

[0111] The reference scale is the feature scale of the facial image when the distance between the observer and the display surface is at a standard distance. It should be noted that the facial feature scale and the reference scale are calculated in the same way; the facial feature scale is the pixel distance between feature point A and feature point B in the actual captured facial image. When determining the reference scale, the observer can be pre-instructed to be at a standard distance (e.g., 1 meter) from the display surface to capture the facial image, and the pixel distance between feature point A and feature point B in the captured image can be used as the reference scale.

[0112] This application embodiment utilizes facial feature scales that reflect distance to dynamically compensate for orientation information, which can eliminate the interference of perspective effects caused by the observer's movement at different distances, thereby solving the problem of inaccurate orientation calculation caused by distance factors, and realizing stable orientation information consistent with the real physical angle at any distance.

[0113] In some embodiments, determining the orientation information of the observer's viewpoint relative to the display surface includes: determining a pixel offset of the face center relative to the face image center; calculating a first deflection angle of the observer in the horizontal direction and a second deflection angle in the vertical direction based on the pixel offset and the field of view of the image acquisition device; and forming the orientation information by the first deflection angle and the second deflection angle.

[0114] Here, the facial image center is the geometric center of the image captured by the image acquisition device. The pixel offset is the difference between the pixel coordinates of the facial center and the pixel coordinates of the facial image center, which can be decomposed into a horizontal offset dx and a vertical offset dy.

[0115] In practical applications, since the image acquisition device is pre-fixed at a fixed position on the display surface, the correspondence between the center of the facial image and the center of the display surface is fixed. When the center of the observer's face deviates from the center of the facial image, it can be assumed that the observer's line of sight has shifted relative to the center of the display surface, resulting in pixel offsets in both the horizontal and vertical directions. Based on the proportion of the pixel offset to the total image size, combined with the field of view of the image acquisition device, the orientation information can be calculated.

[0116] For example, let the horizontal field of view of the image acquisition device be fov_h, and the pixel width of the facial image be W. Then, the angle value v1 corresponding to each pixel in the horizontal direction is the ratio of fov_h to W. The angle value v2 corresponding to each pixel in the vertical direction is calculated as follows: v2=(2×fov_h) / W Then, the first deflection angle and the second deflection angle can be determined based on the pixel offset, the angle value corresponding to each pixel in the vertical direction, and the angle value corresponding to each pixel in the horizontal direction. Furthermore, the first deflection angle and the second deflection angle are corrected according to a scale compensation coefficient. Specifically, the corrected first deflection angle (Yaw) and the second deflection angle (Pitch) can be calculated using the following formulas: yaw=v1×dx×scale pitch=-v2×dy×scale Where scale is the scale compensation coefficient. It should be noted that the scale compensation coefficient can be the ratio of the reference scale to the facial feature scale; to simplify the calculation process, the scale compensation coefficient can also be set to a fixed value (such as 0.5) depending on the actual application.

[0117] This application embodiment, by combining pixel offset and known field of view angle, can accurately convert planar pixel offset in a two-dimensional image into a physical rotation angle in three-dimensional space. The calculation process is simple and clear, does not require complex iterative solution calculations, and can quickly obtain the orientation information of the observer's viewpoint.

[0118] To accurately estimate the absolute distance from an observer to the screen from a facial image, embodiments of this application provide a method for calculating distance. Optionally, determining the distance between the observer and the display surface based on facial feature scales includes: determining a ratio of a reference scale to a facial feature scale; wherein the reference scale is the feature scale of the facial image when the distance between the observer and the display surface is a standard distance; and calculating the distance between the observer and the display surface based on the ratio and the focal length of the image acquisition device.

[0119] It should be noted that the calculation method of the reference scale is similar to that of the facial feature scale. If the facial feature scale is the pixel distance between facial feature point A and facial feature point B in a facial image, then the reference scale is the pixel distance between facial feature point A and facial feature point B in an image captured by an observer at a standard distance from the display surface.

[0120] In some embodiments, the ratio of the reference scale to the facial feature scale is positively correlated with the distance from the observer to the display surface. Specifically, the distance between the observer and the display surface is equal to the product of this ratio and the focal length. The focal length f can be determined based on the horizontal field of view fov_h of the image acquisition device and the pixel width W of the facial image, calculated as follows: f=W / (2×tan(fov_h / 2)) This embodiment introduces a reference scale based on a known standard distance as a benchmark and combines it with focal length to calculate distance, enabling the conversion of scale ratios on an image into distances in real space based on the principle of similar triangles. This embodiment can quickly and accurately determine the distance from the observer to the display surface without relying on special hardware such as depth sensors.

[0121] To further enhance the immersive experience of the displayed content, the display method provided in this application embodiment further includes: acquiring a real environment image that is occluded by the display surface; displaying the display image on the display surface, including: fusing the display image with the real environment image, and displaying the fused display image on the display surface.

[0122] Among them, the real environment image refers to the image of the real environment that is obscured by the display surface.

[0123] Optionally, the display is equipped with a rear-mounted image sensor to capture images of the real-world environment behind the display, which is obscured by the screen. Before displaying the actual image, the rear-mounted image sensor captures real-time images of the real environment, which are then overlaid and blended together.

[0124] In some embodiments, visual content (such as a character model) in the display image is placed on a layer with a real environment image as the background using an alpha channel or by direct overlay, thereby generating a blended display image. In this blended display image, the visual content serves as the foreground and the real environment image serves as the background, creating a visual overlay effect that makes the display surface appear transparent.

[0125] This application embodiment integrates the image of the obscured real environment with the visual content for display. This allows the real scene behind the display to be presented while the visual content is displayed, visually eliminating the physical presence of the display and solving the problem of insufficient immersion caused by the isolation of the visual content and the environment, thereby enhancing the observer's sense of realism and presence.

[0126] The embodiments of the display method have been described in detail above. To enable those skilled in the art to further understand the technical solution of this method, the following description uses a cylindrical display screen as an example to illustrate the overall process of the display method of this application. This cylindrical display is used to provide customers with a three-dimensional product interactive experience. Its screen covers the entire cylindrical surface, and a front image acquisition unit and a rear image acquisition unit are embedded in the top and bottom bezels of the screen.

[0127] Step 1: Determine the pose relationship of the observer's viewpoint relative to the display surface. As a customer approaches the cylindrical display, the front-facing image capture unit begins capturing a real-time image of the customer's face. This image is then transmitted to a processor inside the display, where a pose determination process is performed.

[0128] First, the processor analyzes the facial image and calculates a facial center by locating the coordinates of the tip of the nose and the left and right corners of the mouth. This method is more robust than using only the center of the eyes to determine the facial center. Simultaneously, the processor combines the pixel distances between feature points such as interocular distance, interocular distance, and the vertical distance from the eyes to the mouth to calculate a facial feature scale, which characterizes the relative size of the face in the image.

[0129] Then, based on the pixel offset of the face center relative to the image center, combined with the known horizontal field of view of the front image sensor and the pixel width of the image, the angle value per pixel in the horizontal direction and the angle value per pixel in the vertical direction are calculated. Subsequently, the pixel offset is multiplied by the corresponding angle value to obtain the customer's first deflection angle in the horizontal direction and the second deflection angle in the vertical direction, which together constitute the orientation information.

[0130] Furthermore, the facial feature scale is compared with a reference scale pre-stored in the system (i.e., the size of facial feature pixels measured when the customer is at a standard distance of 1 meter). Based on the pinhole camera model, the physical distance between the customer and the screen surface is calculated using the reference scale, facial feature scale, and focal length.

[0131] After obtaining the orientation information, the angle is also scaled according to the facial feature scale. This compensation eliminates the orientation calculation deviation caused by changes in customer distance (such as moving closer or further away), ensuring that the angle estimation remains accurate at different distances.

[0132] Therefore, the pose relationship (orientation information and distance) of the customer's viewpoint relative to the display surface can be calculated.

[0133] Step 2: Generate a display image that matches the viewpoint based on the pose relationship. like Figure 2 As shown, a main scene has been created in a 3D engine (such as Godot), containing the visual content (virtual characters) to be presented. Additionally, at another location in the world coordinate system, a virtual cylindrical mesh model scaled proportionally to the physical cylindrical screen has been constructed. This model serves as the projection carrier for the visual content, adapting to the curved shape of the cylindrical screen. This model consists of mesh vertices, each with first texture coordinates describing its relative position on the cylindrical surface.

[0134] Based on the customer's pose, a virtual viewpoint is set in the 3D scene, with its position and orientation completely consistent with the customer's viewpoint. Each mesh vertex on the virtual cylinder surface model is traversed, and the spatial coordinates of that mesh vertex in the world coordinate system are reconstructed using the cylinder's geometric parameters (radius, height) and the first texture coordinates of that mesh vertex.

[0135] Using the view matrix and projection matrix of the virtual viewpoint determined from the pose relationship, the reconstructed spatial coordinates are transformed to clip space, and then converted to normalized device coordinates through perspective division. The normalized device coordinates are then mapped to second texture coordinates in the interval [0, 1].

[0136] Before sampling colors, the mesh vertices undergo validity checks, including at least one of the following: checking whether the normalized device coordinates of the mesh vertices are within a valid range (e.g., [-1, 1]); and checking whether the surface normal of the mesh vertex is pointing towards the virtual viewpoint. Mesh vertices that fail the checks will be discarded and will not participate in subsequent sampling, thereby avoiding visual defects such as backlight leakage or intrusion of content outside the field of view.

[0137] Furthermore, for the mesh vertices that pass the validity check, their second texture coordinates are used to sample colors from the 3D scene rendering (such as a virtual character) containing the view content, obtaining the corresponding color values, and then attaching them to the corresponding mesh vertices of the virtual cylindrical mesh model. The attached virtual cylindrical mesh model is as follows: Figure 3 As shown.

[0138] After the bonding is complete, the surface of the virtual cylindrical mesh model is unfolded. Specifically, each vertex of the virtual cylindrical mesh model is traversed to obtain its spatial coordinates (x, y, z). atan2(x, z) is calculated to obtain the circumferential angle θ of the mesh vertex on the cylindrical surface. θ is normalized to a circumferential texture coordinate U in the range [0, 1], for example, U = (θ + π) / (2π). At the same time, using the vertical height coordinate y of the mesh vertex and the total height L of the cylinder, it is normalized to a height texture coordinate V in the range [0, 1], for example, V = (y + L / 2) / L. (U, V) is directly mapped to normalized device coordinates, and the Y-axis is flipped as necessary according to the texture coordinate system orientation of the graphics engine (such as Godot) to adapt to the image coordinate system requirements of the display system. In the fragment shader stage, using mapped normalized device coordinates, color sampling is performed from the surface of the pre-colored virtual cylindrical mesh model (or directly from the rendered 3D scene containing view content, i.e., the rendering result of the SubViewport). This unfolds the pixel colors originally wrapped around the cylindrical surface into a 2D image adapted for planar display output. The resulting 2D image is as follows: Figure 4As shown. Furthermore, it can be used as a display image. The display image generated in this way can adapt to the geometric characteristics of the cylindrical display surface, ensuring that the virtual object will not appear deformed or distorted when viewed from any angle.

[0139] Step 3: Integration with the Real Environment To further enhance the immersive and realistic feel of the display, a rear image capture device can be used to capture real-time images of the environment behind the cylindrical display (such as walls, shelves, and other customers) to obtain a realistic environmental image.

[0140] The display image generated in step two is then fused in real time with the real-world image captured by the rear camera. Specifically, the visual content of the display image is placed on a layer with the real-world image as its background to generate a fused display image. In this image, the virtual character serves as the foreground, and the real environment serves as the background, with the two visually superimposed. The fused display image is then transmitted to the driving circuitry of the cylindrical screen, illuminating all the pixels on the screen.

[0141] As customers move around, turn to the side, or tilt their heads to observe the cylindrical display, the 3D virtual characters on the screen appear to exist realistically in the real environment, with their perspective and occlusion relationships dynamically changing. Furthermore, thanks to the use of a virtual cylindrical mesh model and inverse mapping technology, the image on the cylindrical screen is completely free of geometric stretching, compression, or perspective errors; the lines of the characters and background remain straight and in the correct proportions.

[0142] The above text combined Figures 1 to 4 The present application describes in detail the embodiments of the display method. The following is in conjunction with... Figure 5 This application describes in detail the display device embodiments. It should be understood that the descriptions of the display method embodiments correspond to the descriptions of the display device embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0143] Figure 5 The diagram shown is a structural schematic of a display device provided in an embodiment of this application. Figure 5 As shown, the display device 50 provided in this application embodiment includes: The relationship determination module 510 is used to determine the pose relationship of the observer's viewpoint relative to the display surface; The image generation module 520 is used to generate a display image for presentation on the display surface based on the pose relationship; wherein the presentation angle of the visual content in the display image is matched with the pose relationship. Image display module 530 is used to display images on a display surface.

[0144] Based on any optional technical solution in the embodiments of this application, the image generation module 520 is optionally further configured to: construct a virtual surface model that matches the shape of the display surface; map the visual content onto the virtual surface model according to the pose relationship to form an image that matches the shape of the display surface; and use the image that matches the shape of the display surface as the display image.

[0145] Based on any optional technical solution in the embodiments of this application, the image generation module 520 is optionally further configured to determine a virtual viewpoint according to the pose relationship; and project a three-dimensional scene containing the visual content onto a virtual surface model from the virtual viewpoint to form an image that matches the shape of the display surface.

[0146] Based on any optional technical solution in the embodiments of this application, the image generation module 520 is optionally further configured to: determine the texture coordinates of the virtual surface model; reconstruct the spatial coordinates of the virtual surface model according to the texture coordinates; map the spatial coordinates to the corresponding position in the three-dimensional scene according to the virtual viewpoint, so as to establish a mapping relationship between the spatial coordinates and the three-dimensional scene; and sample colors from the three-dimensional scene according to the mapping relationship to generate an image that matches the shape of the display surface.

[0147] Based on any optional technical solution in the embodiments of this application, the image generation module 520 is optionally further configured to, before projecting the three-dimensional scene containing the visual content onto the virtual surface model from the virtual viewpoint to form an image adapted to the shape of the display surface, perform validity detection on the points on the virtual surface model based on the virtual viewpoint and remove the points that fail the validity detection.

[0148] Based on any of the optional technical solutions in the embodiments of this application, the validity detection may optionally include at least one of the following: detecting whether a point on the virtual surface model is located within the visible space of the virtual viewpoint; and detecting whether the normal of the surface to which the point on the virtual surface model belongs is oriented toward the virtual viewpoint.

[0149] Based on any optional technical solution in the embodiments of this application, the image generation module 520 is optionally further configured to project the visual content onto the virtual surface model according to the pose relationship; and to unfold the surface of the projected virtual surface model to obtain an image that matches the shape of the display surface.

[0150] Based on any optional technical solution in the embodiments of this application, the image generation module 520 is optionally further configured to: determine the correspondence between the texture coordinates of points on the virtual surface model and the planar coordinates after the surface is unfolded; sample colors from the projected virtual surface model; and generate an image that matches the shape of the display surface based on the correspondence and the sampled color values.

[0151] Based on any optional technical solution in the embodiments of this application, the shape of the virtual surface model is proportionally scaled to the shape of the display surface.

[0152] Optionally, based on any of the optional technical solutions in the embodiments of this application, the display surface is the screen of a display.

[0153] Optionally, based on any of the optional technical solutions in the embodiments of this application, an image acquisition device is deployed on the display. The relationship determination module 510 is also used to determine the facial features of the observer by acquiring the facial image of the observer through the image acquisition device; and to determine the pose relationship of the observer's viewpoint relative to the display surface based on the facial features.

[0154] Based on any optional technical solution in the embodiments of this application, the pose relationship may optionally include the orientation information and distance of the observer's viewpoint relative to the display surface.

[0155] Based on any optional technical solution in the embodiments of this application, the facial features may optionally include the observer's facial center and facial feature scale; the facial feature scale is used to characterize the geometric correspondence between the observer's face and the facial image; The relationship determination module 510 is also used to determine the orientation information of the observer's viewpoint relative to the display surface based on the center of the face; and to determine the distance between the observer and the display surface based on the facial feature scale.

[0156] Based on any optional technical solution in the embodiments of this application, the relationship determination module 510 is optionally further configured to perform scale compensation on the orientation information according to the facial feature scale, so as to eliminate the orientation calculation deviation caused by the change in distance between the observer and the display surface, and obtain the compensated orientation information.

[0157] Based on any optional technical solution in the embodiments of this application, the relationship determination module 510 is optionally further configured to: determine the pixel offset of the face center relative to the face image center; calculate the first deflection angle of the observer in the horizontal direction and the second deflection angle in the vertical direction according to the pixel offset and the field of view of the image acquisition device; and form orientation information by the first deflection angle and the second deflection angle.

[0158] Based on any optional technical solution in the embodiments of this application, the relationship determination module 510 is optionally further configured to determine the ratio of the reference scale to the facial feature scale; wherein, the reference scale is the feature scale of the facial image when the distance between the observer and the display surface is a standard distance; and calculate the distance between the observer and the display surface based on the ratio and the focal length of the image acquisition device.

[0159] Based on any optional technical solution in the embodiments of this application, the device may optionally further include: an image acquisition module for acquiring a real environment image that is obscured by the display surface; the image display module 530 is further configured to fuse the display image with the real environment image and display the fused display image on the display surface.

[0160] The display device provided in this application embodiment can execute the display method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0161] It is worth noting that in the above-described embodiments of the target detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0162] Below, for reference Figure 6 This describes a display according to an embodiment of the present application. Figure 6 The diagram shows a schematic representation of a display provided in an exemplary embodiment of this application. The display 60 includes a screen 601 for displaying images and a processor 602 for executing any of the aforementioned display methods. The processor 602 is electrically connected to the screen 601 and executes a computer program stored in a memory to implement the display method provided in this application. The display 60 can be a standalone display device or a display component integrated into other electronic devices (such as smart speakers, computers, and vehicle terminals).

[0163] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the display methods according to the various embodiments of this application described above.

[0164] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0165] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the display methods according to the various embodiments of this application described above.

[0166] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0167] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details of the above application are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0168] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0169] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0170] The above description of the claimed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be applied within the widest scope consistent with the principles and novel features of this application.

[0171] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms described herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A display method, characterized in that, include: Determine the pose relationship between the observer's viewpoint and the display surface; Based on the pose relationship, a display image is generated for presentation on the display surface; wherein the presentation angle of the visual content in the display image matches the pose relationship; The display image is displayed on the display surface.

2. The method according to claim 1, characterized in that, The step of generating a display image for presentation on the display surface based on the pose relationship includes: Construct a virtual surface model that matches the shape of the display surface; Based on the pose relationship, the visual content is mapped onto the virtual surface model to form an image that matches the shape of the display surface; An image that is adapted to the shape of the display surface is used as the display image.

3. The method according to claim 2, characterized in that, The step of mapping the visual content onto the virtual surface model based on the pose relationship to form an image adapted to the shape of the display surface includes: Based on the aforementioned pose relationship, determine the virtual viewpoint; From the virtual viewpoint, a three-dimensional scene containing the visual content is projected onto the virtual surface model to form an image that matches the shape of the display surface.

4. The method according to claim 3, characterized in that, The step of projecting a 3D scene containing the visual content from the virtual viewpoint onto the virtual surface model to form an image adapted to the shape of the display surface includes: Determine the texture coordinates of the virtual surface model; Based on the texture coordinates, the spatial coordinates of the virtual surface model are reconstructed; Based on the virtual viewpoint, the spatial coordinates are mapped to the corresponding positions in the three-dimensional scene to establish a mapping relationship between the spatial coordinates and the three-dimensional scene; Based on the mapping relationship, colors are sampled from the three-dimensional scene to generate an image that matches the shape of the display surface.

5. The method according to claim 3, characterized in that, Before projecting the 3D scene containing the visual content onto the virtual surface model from the virtual viewpoint to form an image adapted to the shape of the display surface, the method further includes: Based on the virtual viewpoint, the points on the virtual surface model are subjected to validity checks, and points that fail the validity check are removed.

6. The method according to claim 5, characterized in that, The validity test includes at least one of the following: Detect whether a point on the virtual surface model is located within the visible space of the virtual viewpoint; and Detect whether the normal of the surface to which a point on the virtual surface model belongs is oriented toward the virtual viewpoint.

7. The method according to claim 2, characterized in that, The step of mapping the visual content onto the virtual surface model based on the pose relationship to form an image adapted to the shape of the display surface includes: Based on the pose relationship, the visual content is projected onto the virtual surface model; The surface of the projected virtual surface model is unfolded to obtain the image that matches the shape of the display surface.

8. The method according to claim 7, characterized in that, The step of unfolding the surface of the projected virtual surface model to obtain the image that matches the shape of the display surface includes: Determine the correspondence between the texture coordinates of points on the virtual surface model and the planar coordinates of the unfolded surface; Colors are sampled from the projected virtual surface model, and an image adapted to the shape of the display surface is generated based on the correspondence and the sampled color values.

9. The method according to claim 2, characterized in that, The shape of the virtual surface model is scaled proportionally to the shape of the display surface.

10. The method according to any one of claims 1 to 9, characterized in that, The display surface is the screen of the monitor.

11. The method according to claim 10, characterized in that, An image acquisition device is deployed on the display; Determining the pose relationship of the observer's viewpoint relative to the display surface includes: The facial features of the observer are determined by the facial image acquired by the image acquisition device. Based on the facial features, the pose relationship of the observer's viewpoint relative to the display surface is determined.

12. The method according to claim 11, characterized in that, The pose relationship includes the orientation information and distance of the observer's viewpoint relative to the display surface.

13. The method according to claim 12, characterized in that, The facial features include the observer's facial center and facial feature scale; The facial feature scale is used to characterize the geometric correspondence between the observer's face and the facial image; Determining the pose relationship of the observer's viewpoint relative to the display surface based on the facial features includes: Based on the center of the face, determine the orientation information of the observer's viewpoint relative to the display surface; The distance between the observer and the display surface is determined based on the facial feature scale.

14. The method according to claim 13, characterized in that, Also includes: Based on the facial feature scale, the orientation information is scale-compensated to eliminate the orientation calculation deviation caused by the change in distance between the observer and the display surface, thus obtaining the compensated orientation information.

15. The method according to claim 13, characterized in that, Determining the orientation information of the observer's viewpoint relative to the display surface based on the facial center includes: Determine the pixel offset of the face center relative to the face image center; Based on the pixel offset and the field of view of the image acquisition device, calculate the first deflection angle of the observer in the horizontal direction and the second deflection angle in the vertical direction; The orientation information is composed of the first deflection angle and the second deflection angle.

16. The method according to claim 13, characterized in that, Determining the distance between the observer and the display surface based on the facial feature scale includes: Determine the ratio of the reference scale to the facial feature scale; wherein the reference scale is the feature scale of the facial image when the distance between the observer and the display surface is a standard distance; The distance between the observer and the display surface is calculated based on the ratio and the focal length of the image acquisition device.

17. The method according to any one of claims 1 to 9, characterized in that, Also includes: Acquire real-world images that are obscured by the display surface; The step of displaying the image on the display surface includes: The displayed image is fused with the real environment image, and the fused image is displayed on the display surface.

18. A display device, characterized in that, include: The relationship determination module is used to determine the pose relationship of the observer's viewpoint relative to the display surface; An image generation module is configured to generate a display image for presentation on the display surface based on the pose relationship; wherein the presentation angle of the visual content in the display image matches the pose relationship; An image display module is used to display the image on the display surface.

19. A display, characterized in that, include: Screen; A processor for performing the display method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the display method according to any one of claims 1 to 17.

21. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a display, cause the display to perform the display method according to any one of claims 1 to 17.

Images