Display method and device

Abstract

The embodiment of the invention provides a display method and equipment. The method comprises the following steps: determining a first centroid coordinate of a first object in a world coordinate system in a first picture to be displayed; determining a mapping relation between the world coordinate system and an observation coordinate system according to the first centroid coordinate; wherein the observation coordinate system is a coordinate system taking an actual viewpoint in a physical space as an original point; according to the mapping relation, mapping the first picture from the world coordinate system to the observation coordinate system to obtain a mapped first picture; and rendering the mapped first picture and then displaying the first picture.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of metaverse technology, and in particular, to a display method and device. Background Art

[0002] In the virtual digital space of metaverse scenarios such as three-dimensional, virtual reality, or augmented reality, there are a large number of two-dimensional and / or three-dimensional elements. When different elements of multiple dimensions are located in the same three-dimensional space and there are requirements such as interaction, following, rotation, binding, and positioning, situations such as jitter, jump, and mutation may occur. Summary of the Invention

[0003] Embodiments of this application provide a display method and device, which are beneficial to decoupling phenomena such as jitter, jump, and unevenness caused by changes in reference points brought about by bone actions, deformation animations, etc. within a local coordinate system.

[0004] The technical solution of the embodiments of this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a display method, which includes: determining the first centroid coordinate of a first object in a first picture to be displayed in a world coordinate system; determining the mapping relationship between the world coordinate system and an observation coordinate system according to the first centroid coordinate, where the observation coordinate system is a coordinate system with the actual viewpoint in the physical space as the origin; mapping the first picture from the world coordinate system to the observation coordinate system according to the mapping relationship to obtain a mapped first picture; and rendering and displaying the mapped first picture.

[0006] In a second aspect, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the method described in the first aspect.

[0007] In a third aspect, embodiments of this application provide a display device, which includes: a first determination module configured to determine the first centroid coordinate of a first object in a first picture to be displayed in a world coordinate system; a second determination module configured to determine the mapping relationship between the world coordinate system and an observation coordinate system according to the first centroid coordinate, where the observation coordinate system is a coordinate system with the actual viewpoint in the physical space as the origin; a mapping module configured to map the first picture from the world coordinate system to the observation coordinate system according to the mapping relationship to obtain a mapped first picture; and a display module configured to render and display the mapped first picture.

[0008] Fourthly, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, and the computer program causes a computer to execute the method described in the first aspect.

[0009] Fifthly, an embodiment of the present application provides a computer program product, including computer program instructions, and the computer program instructions cause a computer to execute the method described in the first aspect.

[0010] Sixthly, an embodiment of the present application provides a computer program, and the computer program causes a computer to execute the method described in the first aspect.

[0011] In an embodiment of the present application, the first centroid coordinate of the first object of the first screen to be displayed in the world coordinate system is determined; according to the first centroid coordinate, the mapping relationship between the world coordinate system and the viewing coordinate system is determined; according to this mapping relationship, the first screen is mapped from the world coordinate system to the viewing coordinate system; that is to say, the object in the first screen is bound to the centroid of the first object, rather than being bound to a specific bone node of the first object; thereby decoupling the jitter, jump, unevenness and other phenomena caused by the change of the reference point brought by the bone actions, deformation animations, etc. in the local coordinate system, and further improving the display effect of the first screen.

[0012] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the technical solution of the present application. Description of the Drawings

[0013] The drawings here are incorporated into the description and form a part of this description. These drawings show embodiments consistent with the present application and are used together with the description to explain the technical solution of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0014] The flowcharts shown in the drawings are only exemplary descriptions, and do not necessarily include all the content and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0015] Figure 1 Schematic diagram of the application scenario for the metaverse Figure 1 ;

[0016] Figure 2 Schematic diagram of the application scenario for the metaverse Figure 2 ;

[0017] Figure 3It is a schematic diagram for solving the view transformation matrix based on a camera;

[0018] Figure 4 It is a schematic diagram of a display method Figure 1 ;

[0019] Figure 5 It is a schematic diagram of a display method Figure 2 ;

[0020] Figure 6 It is a schematic diagram of a tree structure;

[0021] Figure 7 It is a schematic diagram of the implementation process of a display method provided by this application Figure 1 ;

[0022] Figure 8 It is a schematic diagram of the implementation process of determining the centroid coordinates provided by an embodiment of this application;

[0023] Figure 9 It is a schematic diagram of the implementation process of determining the first bounding volume provided by an embodiment of this application;

[0024] Figure 10 It is a schematic diagram of the implementation process of a display method provided by this application Figure 2 ;

[0025] Figure 11 It is a schematic diagram of the implementation process of determining the fourth coordinate provided by this application;

[0026] Figure 12 It is a schematic diagram of the first object provided by an embodiment of this application;

[0027] Figure 13 It is a schematic diagram of the first bounding volume provided by an embodiment of this application Figure 1 ;

[0028] Figure 14 It is a schematic diagram of the first bounding volume provided by an embodiment of this application Figure 2 ;

[0029] Figure 15 It is a schematic diagram of the structure of a display device provided by an embodiment of this application;

[0030] Figure 16 It is a schematic diagram of the structure of an electronic device provided by an embodiment of this application. Detailed implementation manners

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0034] In the virtual digital space of metaverse scenarios such as 3D, VR or AR, there are a large number of 2D and / or 3D elements, such as particles, 2D images, 3D geometric bodies, text, video, and even traditional Hypertext Markup Language (HTML) nodes. Solving the coordinate positions of these elements is widely used in digital scenarios of different businesses. Figure 1 Illustration of the application scenarios of the Metaverse Figure 1 ;in, Figure 1 (a) shows the 3D holographic projection and the 2D interactive board are anchored to each other; Figure 1 (b) and (c) show the 3D motorcycle product and the 2D information board being anchored to each other; Figure 1 (d) in the figure shows that the 3D digital human anchor and multiple 2D guides are anchored to each other; Figure 1 (e) in the figure shows the mutual anchoring of the 3D twin park and the 2D annotation; Figure 1 (f) in the figure shows that many elements of the industry digital twin scenario are anchored to each other.

[0035] As can be seen from the above cases, a large number of different types of objects (one-dimensional, two-dimensional, three-dimensional) will generate a large number of related requirements such as interaction, following, rotation, binding, and positioning according to the business. This is different from traditional videos and pictures, and also different from traditional application (Application, APP) development and web development, which only require a Cartesian coordinate system.

[0036] However, when different elements in multiple dimensions are located in the same three-dimensional space and generate requirements such as interaction, following, rotation, binding, and positioning, more complex spatial calculation algorithms are needed to achieve various requirements and optimizations. If the solution cannot be well obtained under the cooperation of multiple coordinate systems, situations such as jitter, jump, and mutation may occur.

[0037] In some scenarios, when 2D text is bound to a three-dimensional multi-degree-of-freedom digital human, due to the transformation of the character or text within their respective local coordinate systems, problems such as anchor jitter, jump, and misalignment may occur. For example Figure 2 as shown in (a) of , for the digital human running in 3D and the anchored 2D text (such as "test" in the figure), after executing the bone animation, the 2D nickname undergoes repeated and high-frequency jitter; another example is Figure 2 as shown in (b) of , for the 3D digital human and the anchored 2D nickname (such as "test" in the figure): when executing animations such as looking up, tilting the body, and standing naturally, the 2D nickname undergoes repeated and high-frequency jitter.

[0038] At the same time, for the follow-up of the camera, for example Figure 3 as shown in , jitter and jump phenomena also occur and cannot maintain relative stability. Because it is difficult to select the focal point of the camera. As a result, the view transformation matrix cannot be solved, that is, the target position coordinates cannot be confirmed. Among them, Figure 3 the coordinate system in is the Media Coordinate System (MCS), and the content in MCS is the video captured by the scene camera, and 301 represents the observation point.

[0039] In a related technology, as Figure 4 shown in , all objects are bound to the world coordinate system. This technology builds a state machine and manually maintains the relative relationships and transformation matrices of all objects; then it is calculated by traversing each frame.

[0040] In another related technology, as Figure 5 shown in , taking the digital human as an example, the main body such as the digital human model is managed and generated (according to the bones). Through a tree structure (as Figure 6 shown in ), the object 601 to be anchored is added to a specific bone node; when the node changes, the same transformation matrix is applied to its bound node; among them, 602 represents the parent node pointer.

[0041] However, the inventors of the present application found through research and analysis of the above related technologies that the technology of binding all objects to the world coordinate system has the following disadvantages: large amount of calculation, difficult to manage, high time consumption, low frame rate, and large performance overhead. The technology of establishing a tree structure has the following disadvantages: it can only bind attachments to a certain node of the main object and cannot bind globally (the root is also regarded as a specific node).

[0042] An embodiment of the present application provides a display method, Figure 7 which is a schematic implementation process of a display method provided by the present application Figure 1 , as Figure 7 shown, the method includes the following steps 701 to 704:

[0043] Step 701, determining the first centroid coordinate of a first object in a first screen to be displayed in a world coordinate system;

[0044] Step 702, determining a mapping relationship between the world coordinate system and an observation coordinate system according to the first centroid coordinate; wherein, the observation coordinate system is a coordinate system with an actual viewing point in a physical space as the origin;

[0045] Step 703, mapping the first screen from the world coordinate system to the observation coordinate system according to the mapping relationship to obtain a mapped first screen;

[0046] Step 704, rendering and displaying the mapped first screen.

[0047] It can be understood that in the embodiment of the present application, the first centroid coordinate of the first object in the first screen to be displayed is determined in the world coordinate system; according to the first centroid coordinate, the mapping relationship between the world coordinate system and the observation coordinate system is determined; according to this mapping relationship, the first screen is mapped from the world coordinate system to the observation coordinate system; that is to say, the object in the first screen is bound to the centroid of the first object, rather than being bound to a specific bone node of the first object; thereby decoupling the jitter, jump, unevenness, etc. caused by the change of the reference point brought by the bone action, deformation animation, etc. in the local coordinate system, and further improving the display effect of the first screen.

[0048] The following further describes optional implementation manners of the above steps and related terms, etc.

[0049] In step 701, the first centroid coordinate of the first object in the first screen to be displayed is determined in the world coordinate system.

[0050] It should be understood that in the embodiments of the present application, the first screen and the first object are not limited. In some embodiments, the first screen includes a first object and a third object that moves or changes with reference to, or follows, the first object. Exemplarily, in one possible implementation, the third object includes an object that interacts with, follows, rotates, binds, positions, etc. with the first object. In some embodiments, the first object is the object in the first screen that is referred to, referenced, or followed. Exemplarily, in one possible implementation, the first object may be a person in a virtual space.

[0051] In the embodiments of the present application, the centroid coordinates of the first object are not limited. In some embodiments, the centroid coordinates may be the approximate centroid of the first object.

[0052] In the embodiments of the present application, the world coordinate system is the absolute coordinate system of the system, also known as the global coordinate system or the absolute coordinate system. Before the user coordinate system is established, the coordinates of all points on the screen are determined by the origin of this coordinate system for their respective positions. The world coordinate system describes the position and orientation of objects in the entire scene. And in the world coordinate system, the directions of the coordinate origin, the X-axis, the Y-axis, and the Z-axis are fixed and do not change with the viewing angle or other factors.

[0053] In the embodiments of the present application, the viewing coordinate system, also known as the camera coordinate or view coordinate, refers to the coordinate system that converts an object from the world coordinate system to the camera coordinate system. The viewing coordinate system is a three-dimensional rectangular auxiliary coordinate system that can be defined in any direction and anywhere in the user coordinate system. The viewing coordinate system is mainly used to specify the clipping space and determine which part of the three-dimensional geometric shape needs to be output on the screen. Through the viewing plane, the coordinate values of the part of the three-dimensional geometric shape that needs to be output in the world coordinate system can be converted into the coordinate values in the normalized coordinate system. When observing an object at different distances and angles, the viewing coordinate system will change to adapt to the observer's perspective.

[0054] In some embodiments, Figure 8 is a schematic flow diagram of an implementation for determining centroid coordinates provided by the embodiments of the present application. As Figure 8 shown, the centroid coordinates can be determined through the following steps 801 to 803:

[0055] Step 801, determine the first bounding volume of the first object according to the first coordinates of the vertices of the first object in the first model coordinate system; wherein, the first model coordinate system is used to create the first object;

[0056] Step 802, determine the third coordinates of the centroid of the first bounding volume in the world coordinate system according to the second coordinates of the centroid of the first bounding volume in the first model coordinate system;

[0057] Step 803: Determine the first centroid coordinates of the first object in the world coordinate system according to the third coordinate.

[0058] It should be understood that in the embodiments of the present application, there is no limitation on the bounding volume, as long as the bounding volume can enclose the first object, and the bounding volume can be of any shape. There is also no limitation on the first model coordinate system. The first model coordinate system is the coordinate system defined when defining the first object, and it is usually a local Cartesian coordinate system used to describe the shape, size, and position of the first object itself.

[0059] In the embodiments of the present application, the model coordinate system is the coordinate system of the object itself, relative to the origin of the object itself, and is also called the local coordinate system. The model coordinate system is a set of three-dimensional local coordinate systems with a certain point (usually the center point) on the model as the origin. The model coordinate system is usually used in modeling software and the model editing interfaces of engine editors. Data related to the vertices of the model itself is usually stored according to the model coordinate system. When used for rendering, it needs to be transformed into the world coordinate system through its own transformation (Transform) relative to the world coordinate system.

[0060] In some embodiments, the determining the first bounding volume of the first object according to the first coordinates of the vertices of the first object in the first model coordinate system includes: determining the second bounding volume of the first part according to the first coordinates of the vertices of the first part in the first object in the first model coordinate system; determining the first bounding volume of the first object according to the second bounding volume and the first coordinates of the vertices of the second part in the first object in the first model coordinate system; the second part is different from the first part.

[0061] It should be understood that in the embodiments of the present application, there is no limitation on the first part and the second part, and the first part and the second part can be any part of the first object.

[0062] In some embodiments, the determining the second bounding volume of the first part according to the first coordinates of the vertices of the first part in the first model coordinate system includes: determining the distances of the first coordinates of the vertices of the first part in the first model coordinate system in multiple direction dimensions respectively; determining the fourth bounding volume according to the distances in the multiple direction dimensions; determining the second bounding volume according to the fourth bounding volume and the first coordinates of the vertices of the first part in the first model coordinate system.

[0063] It should be understood that in the embodiments of the present application, there is no limitation on the multiple direction dimensions, and the multiple dimension directions can be the X-axis direction, the Y-axis direction, and the Z-axis direction.

[0064] In some embodiments, determining the fourth bounding volume according to the distances in the plurality of direction dimensions includes: determining the maximum distance in the plurality of direction dimensions; and using the maximum distance as the radius of the fourth bounding volume.

[0065] It should be understood that in the embodiments of the present application, no limitation is imposed on the central position of the fourth bounding volume. In some embodiments, the central position of the fourth bounding volume may be the central position of the first part; in some other embodiments, the central position of the fourth bounding volume may be the central position of the first object; in still some other embodiments, the central position of the fourth bounding volume may be the central position in the first model coordinate system.

[0066] In some embodiments, determining the second bounding volume according to the fourth bounding volume and the first coordinates of the vertices of the first part in the first model coordinate system includes: performing a third process according to the first coordinates of the j-th vertex of the first part in the first model coordinate system; wherein, the third process includes: if the first coordinates of the j-th vertex are not within the fourth bounding volume, updating the fourth bounding volume so that the updated fourth bounding volume includes the first coordinates of the j-th vertex; where j is greater than 0 and less than or equal to the number of vertices of the first part; iteratively performing the third process until the updated fourth bounding volume includes the vertices of the first part, and determining the second bounding volume according to the updated fourth bounding volume.

[0067] It should be understood that in the embodiments of the present application, the updated fourth bounding volume obtained by finally performing the third process is used as the second bounding volume; the second bounding volume of the first part is the updated fourth bounding volume, that is to say, the updated fourth bounding volume is adjusted to obtain the second bounding volume of the first part. In the embodiments of the present application, no limitation is imposed on the second bounding volume, and the second bounding volume may be a bounding volume of any shape.

[0068] In some embodiments, the shape of the fourth bounding volume is a sphere, and updating the fourth bounding volume includes: determining the updated fourth bounding volume according to the distance between the first coordinates of the j-th vertex and the center of the sphere of the fourth bounding volume and the radius of the fourth bounding volume.

[0069] In some embodiments, determining the first bounding volume of the first object according to the first coordinates of the second part of the first object and the vertices of the second bounding volume in the first model coordinate system includes: performing a first process according to the first coordinates of the i-th vertex of the second part of the first object in the first model coordinate system; wherein, the first process includes: if the first coordinates of the i-th vertex are not within the second bounding volume, updating the second bounding volume so that the updated second bounding volume includes the first coordinates of the i-th vertex; wherein, i is greater than 0 and less than or equal to the number of vertices of the second part; iteratively performing the first process until the updated second bounding volume includes the vertices of the second part, and determining the first bounding volume according to the updated second bounding volume.

[0070] It should be understood that in the embodiments of the present application, the updated second bounding volume obtained by performing the first process for the last time is used as the first bounding volume; the first bounding volume of the first object is the updated second bounding volume, that is to say, the updated second bounding volume is adjusted to obtain the first bounding volume of the first object. In the embodiments of the present application, the second bounding volume is not limited, and the second bounding volume can be a bounding volume of any shape.

[0071] In some embodiments, the shape of the second bounding volume is a sphere, and updating the second bounding volume includes: determining the updated second bounding volume according to the distance between the first coordinates of the i-th vertex and the center of the sphere of the second bounding volume and the radius of the second bounding volume.

[0072] In some embodiments, Figure 9 The implementation flow diagram of determining the first bounding volume provided by the embodiments of the present application is shown in Figure 9 As shown, the first bounding volume can be determined through the following steps 901 to 903:

[0073] Step 901, determining the distances of the first coordinates in multiple direction dimensions in the first model coordinate system according to the vertices of the first object;

[0074] Step 902, determining a third bounding volume according to the distances in the multiple direction dimensions;

[0075] Step 903, determining the first bounding volume according to the third bounding volume and the first coordinates of the vertices of the first object in the first model coordinate system.

[0076] It should be understood that in the embodiments of the present application, the multiple direction dimensions are not limited, and the multiple dimension directions can be the X-axis direction, the Y-axis direction, and the Z-axis direction.

[0077] In some embodiments, determining the third bounding volume according to the distances in the plurality of direction dimensions includes: determining the maximum distance in the plurality of direction dimensions; and using the maximum distance as the radius of the third bounding volume.

[0078] It should be understood that in the embodiments of the present application, no limitation is imposed on the center position of the third bounding volume. In some embodiments, the center position of the third bounding volume may be the center position of the first object; in other embodiments, the center position of the third bounding volume may be the center position in the first model coordinate system.

[0079] In some embodiments, determining the first bounding volume according to the third bounding volume and the first coordinates of the vertices of the first object in the first model coordinate system includes: performing a second process according to the first coordinates of the m-th vertex of the first object in the first model coordinate system, where the second process includes: if the first coordinates of the m-th vertex are not within the third bounding volume, updating the third bounding volume so that the updated third bounding volume includes the first coordinates of the m-th vertex; where m is greater than 0 and less than or equal to the number of vertices of the first object; iteratively performing the second process until the updated third bounding volume includes the vertices of the first object, and determining the first bounding volume according to the updated third bounding volume.

[0080] In some embodiments, updating the third bounding volume includes: determining the updated third bounding volume according to the distance between the first coordinates of the m-th vertex and the center of the sphere of the third bounding volume and the radius of the third bounding volume.

[0081] It should be understood that in the embodiments of the present application, no limitation is imposed on the third bounding volume, and the third bounding volume may be a bounding volume of any shape. In some embodiments, the third bounding volume is spherical.

[0082] Exemplarily, in a possible implementation manner, the first bounding volume may be determined through the following steps 11 to 16:

[0083] Step 11, find the maximum distance maxX in the x direction of the vertices in the first object;

[0084] Step 12, find the maximum distance maxY in the y direction of the vertices in the first object;

[0085] Step 13, find the maximum distance maxZ in the z direction of the vertices in the first object;

[0086] Step 14, take the maximum of the three as the initial radius r;

[0087] Step 15: Traverse the remaining vertex coordinates. If the point P (vertex) is contained in the sphere, it is ignored; otherwise, recalculate the new radius. Let the new center coordinate of the sphere be S, and the radius of the old center be r. Then the radius of the new center is equal to:

[0088]

[0089] Step 16: Calculate the movement vector from the old center of the sphere to the new center of the sphere:

[0090]

[0091] Repeat the traversal logic, that is, repeat steps 15 and 16 until all vertex coordinates are traversed.

[0092] Among them, the point O represents the center of the old sphere, represents the distance between the vertex P and the old center O of the sphere; represents the vector from the vertex P to the old center O of the sphere; represents the distance between the new center S and the old center O of the sphere; represents the vector from the new center S to the old center O of the sphere.

[0093] In some embodiments, if there is occlusion of the first object, the centroid coordinates of the first object can be determined in the following manner:

[0094] (1) If the unoccluded part of the first object has a regular or symmetric shape, determine the vertex coordinates of the occluded part of the first object according to the shape of the unoccluded part;

[0095] (2) If the first object has been recorded or measured before occlusion, determine the vertex coordinates of the occluded part of the first object according to the information before occlusion and the current information of the first object;

[0096] (3) Determine the vertex coordinates of the occluded part of the first object according to image processing techniques (such as edge detection, contour extraction, etc.);

[0097] Thus, based on the determined vertex coordinates of the first object, determine the first centroid coordinates of the first object in the world coordinate system.

[0098] In step 702, determine the mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates; among them, the observation coordinate system is a coordinate system with the actual viewpoint in the physical space as the origin.

[0099] In some embodiments, determining the mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates includes: determining the pose information of the object to which the actual view point belongs; and determining the mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates and the pose information.

[0100] It should be understood that in the embodiments of the present application, no limitation is imposed on the pose information, and the pose information refers to the information describing the direction and position state of an object or system in space. For example, the observer stands facing forward, or the observer lies down and looks right forward.

[0101] Exemplarily, in a possible implementation, the position of the origin of the observation space coordinate system (i.e., an example of the observation coordinate system) in the world coordinate system is Pe(ex, ey, ez); the camera looks at the target position (i.e., an example of the first centroid coordinates) as T(tx, ty, tz); and the up direction vector (i.e., an example of the pose information) of the camera is upDirection(ux, uy, uz).

[0102] The solution steps of the mapping relationship are as follows:

[0103] Step 21, zAxis = Pe - T = (ex - tx, ey - ty, ez - tz);

[0104] Step 22, xAxis = zAxis × upDirection;

[0105] Step 23, yAxis = zAxis × xAxis;

[0106] Step 24, substituting into the matrix transformation framework, the matrix E for transforming the observation space coordinate system to the world coordinate system can be obtained; where E is:

[0107]

[0108] where x represents xAxis, y represents yAxis, and z represents zAxis;

[0109] Step 25, using the algorithm of the inverse matrix to find the inverse matrix E-1 of E, that is, the view matrix (i.e., an example of the mapping relationship).

[0110] In step 703, according to the mapping relationship, the first picture is mapped from the world coordinate system to the observation coordinate system to obtain the mapped first picture.

[0111] It should be understood that in the embodiments of the present application, "mapping the first picture from the world coordinate system to the observation coordinate system" means mapping the world coordinate systems of all objects in the first picture to the observation coordinate system.

[0112] In some embodiments, mapping the first screen from the world coordinate system to the viewing coordinate system according to the mapping relationship includes: multiplying the coordinates of the objects in the first screen by the mapping relationship to obtain the coordinates of the objects in the first screen in the viewing coordinate system, thereby mapping the first screen from the world coordinate system to the viewing coordinate system.

[0113] In some embodiments, Figure 10 is a schematic implementation process of a display method provided by the present application Figure 2 , such as Figure 10 shown, the method includes the following steps 1001 to 1005:

[0114] Step 1001, determining the second centroid coordinates of the first object in the second screen to be displayed in the world coordinate system;

[0115] Step 1002, determining the fourth coordinates of the second object in the second screen in the world coordinate system according to the second centroid coordinates and the first centroid coordinates; the second object is at least one object different from the first object in the second screen;

[0116] Step 1003, determining the fifth coordinates of the first object in the second screen in the world coordinate system;

[0117] Step 1004, respectively mapping the fourth coordinates of the second object and the fifth coordinates of the first object in the second screen from the world coordinate system to the viewing coordinate system according to the mapping relationship to obtain the mapped second screen;

[0118] Step 1005, rendering and displaying the mapped second screen.

[0119] It can be understood that in the embodiments of the present application, the second centroid coordinates of the first object in the second screen to be displayed are determined in the world coordinate system; according to the second centroid coordinates and the first centroid coordinates, the fourth coordinates of the second object different from the first object in the second screen are determined in the world coordinates, and the fifth coordinates of the first object in the second screen are determined in the world coordinates; according to the above mapping relationship, the fourth coordinates of the second object and the fifth coordinates of the first object in the second screen are respectively mapped from the world coordinate system to the viewing coordinate system; that is to say, when the centroid coordinates of the first object change, the coordinates of the second object are determined according to the change of the centroid coordinates of the first object, rather than being bound to specific bone nodes of the first object; thereby decoupling the jitter, jump, unevenness, etc. caused by the change of the reference point brought by the bone actions, deformation animations, etc. in the local coordinate system, and further improving the display effect of the second screen.

[0120] It should be understood that in the embodiments of the present application, the first object and the second object are not limited. In some embodiments, the second object moves or changes by referring to, referencing, or following the first object. Exemplarily, in a possible implementation manner, the first object is a virtual person, and the second object can be a pet that follows the virtual person, a label that follows the virtual person, or a vehicle driven by the virtual person, etc.

[0121] In the embodiments of the present application, according to the mapping relationship, respectively mapping the fourth coordinate of the second object and the fifth coordinate of the first object in the second picture from the world coordinate system to the observation coordinate system includes: determining the seventh coordinate of the second object in the second picture according to the transformation relationship; determining the fourth coordinate of the second object in the second picture in the world coordinate system according to the seventh coordinate.

[0122] It should be understood that in the embodiments of the present application, the seventh coordinate is actually still the coordinate in the second model coordinate system, so the fourth coordinate needs to be determined according to the relationship between the second model coordinate system and the world coordinate system.

[0123] In some embodiments, according to the mapping relationship, respectively mapping the fourth coordinate of the second object and the fifth coordinate of the first object in the second picture from the world coordinate system to the observation coordinate system includes: multiplying the fourth coordinate of the second object in the second picture by the mapping relationship to obtain the coordinate of the fourth coordinate of the second object in the second picture in the observation coordinate system, so as to map the fourth coordinate of the second object in the second picture from the world coordinate system to the observation coordinate system; multiplying the fifth coordinate of the first object in the second picture by the mapping relationship to obtain the coordinate of the fifth coordinate of the first object in the second picture in the observation coordinate system, and mapping the fifth coordinate of the first object in the second picture from the world coordinate system to the observation coordinate system.

[0124] In some embodiments, Figure 11 A schematic diagram of an implementation process for determining the fourth coordinate provided by the present application is as Figure 11 shown, and the method includes the following steps 1101 to step 1104:

[0125] Step 1101, determining the fifth coordinate of the second centroid coordinate in the second model coordinate system where the second object is located; wherein, the second model coordinate system is used to create the second object;

[0126] Step 1102, determining the sixth coordinate of the first centroid coordinate in the second model coordinate system;

[0127] Step 1103: Determine the transformation relationship between the fifth coordinate and the sixth coordinate according to the fifth coordinate and the sixth coordinate;

[0128] Step 1104: Determine the fourth coordinate of the second object in the second picture in the world coordinate system according to the transformation relationship.

[0129] It should be understood that in the embodiments of the present application, the second model coordinates are not limited. The second model coordinate system is the coordinate system used to define the second object. It is usually a local Cartesian coordinate system, which is used to describe the shape, size and position of the second object itself.

[0130] In some embodiments, the first model coordinate system and the second model coordinate system may be the same coordinate system, and this coordinate system is used to create the first object and the second object.

[0131] In step 704, the mapped first picture is rendered and then displayed.

[0132] The following describes an exemplary application of the processing method provided in the embodiments of the present application in an actual scenario.

[0133] The problems to be considered in solving the camera looking at the target position T are as follows: The model as a whole is not centered in the local coordinate system; the model will undergo matrix changes such as displacement, deformation, and rotation, such as skeletal animation and deformation animation.

[0134] Then, a fast calculation method for the approximate centroid based on the minimum bounding sphere is adopted here to solve for T:

[0135] For all spatial vertex coordinates of the first object:

[0136] Step 11: Find the maximum distance maxX in the x direction;

[0137] Step 12: Find the maximum distance maxY in the y direction;

[0138] Step 13: Find the maximum distance maxZ in the z direction;

[0139] Step 14: Take the maximum of the three as the initial radius r;

[0140] Step 15: Traverse the remaining vertex coordinates. If the point P (vertex) is included in the sphere, it is ignored. Otherwise, recalculate the new radius. Let the new center coordinate of the sphere be S, and the radius of the old center be r. Then the radius of the new center of the sphere is equal to:

[0141]

[0142] Step 16: It is easy to find the movement vector from the old center of the sphere to the new center of the sphere:

[0143]

[0144] The repeated traversal logic, that is, steps 15 and 16 are traversed repeatedly. Among them, point O represents the center of the old sphere, represents the distance between vertex P and the center O of the old sphere; represents the vector from vertex P pointing to the center O of the old sphere; represents the distance between the center S of the new sphere and the center O of the old sphere; represents the vector from the center S of the new sphere pointing to the center O of the old sphere. Among them, the first object is as Figure 12 shown, Figure 12 in (a), (b), and (c) respectively represent an example of the first object.

[0145] The overall algorithm complexity is O(n), and the final generated effect is as Figure 13 and Figure 14 shown.

[0146] The final center of the sphere is the target T. The relative coordinates of any type (2D / 3D) object can be based on this approximate centroid value. In this way, no matter where the reference object is in the local coordinate system, any deformation, any amplitude, there will no longer be situations such as following jitter, including the camera, and the rendering is always in a smooth state.

[0147] The position of the origin of the viewing space coordinate system (i.e., an example of the viewing coordinate system) in the world coordinate system is Pe(ex, ey, ez); the camera looks at the target position (i.e., an example of the first centroid coordinate) as T(tx, ty, tz); the upward direction vector of the camera (i.e., an example of the attitude information) upDirection is (ux, uy, uz).

[0148] The solution steps of the mapping relationship are as follows:

[0149] Step 21, zAxis = Pe - T = (ex - tx, ey - ty, ez - tz);

[0150] Step 22, xAxis = zAxis × upDirection;

[0151] Step 23, yAxis = zAxis × xAxis;

[0152] Step 24, substituting into the matrix transformation framework, the matrix E for transforming the viewing space coordinate system to the world coordinate system can be obtained; among them, E is:

[0153]

[0154] Among them, x represents xAxis, y represents yAxis, and z represents zAxis;

[0155] Step 25, use the algorithm of the inverse matrix to obtain the inverse matrix E-1 of E, which is the view matrix (i.e., an example of the mapping relationship).

[0156] Here, it is necessary to find the relatively fixed observation space target position to achieve the following effects:

[0157] 1. The digital space rendering is stable and smooth;

[0158] 2. There are no phenomena such as jumps;

[0159] 3. The objects that are mutually anchored can track the geometric whole of the reference target instead of the local components.

[0160] The embodiment of the present application proposes a display method. This method proposes a way to respond to all changes with constancy and uses an efficient approximation algorithm to solve the approximate geometric centroid of the object to be referenced. The centroid coordinates are converted from the local coordinate system to the world coordinate system coordinates as the constant reference coordinate points. The advantages of this method include: in terms of performance, only one approximation of the centroid needs to be solved, and no additional calculation overhead is required after binding, which greatly reduces the redundant calculation amount and improves the performance; in terms of stability and the user's visual experience, it completely isolates and decouples the jitter, jumps, unevenness, etc. caused by the change of the reference point brought about by the bone actions, deformation animations, etc. in the local coordinate system.

[0161] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that these steps must be executed in this specific order, or that all the steps shown must be executed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.; or, the steps in different embodiments may be combined into a new technical solution.

[0162] Based on the foregoing embodiments, the embodiment of the present application provides a display device. The device includes each module included and each unit included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; during the implementation process, the processor can be an AI acceleration engine (such as an NPU, etc.), a GPU, a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0163] Figure 15 It is a schematic structural diagram of a display device provided by the embodiment of the present application. As Figure 15 shown, the display device 150 includes a first determination module 1501, a second determination module 1502, a mapping module 1503, and a display module 1504, where:

[0164] The first determination module 1501 is configured to determine the first centroid coordinate of the first object in the first screen to be displayed in the world coordinate system;

[0165] The second determination module 1502 is configured to determine the mapping relationship between the world coordinate system and the viewing coordinate system according to the first centroid coordinate; wherein, the viewing coordinate system is a coordinate system with the actual viewing point in the physical space as the origin;

[0166] The mapping module 1503 is configured to map the first screen from the world coordinate system to the viewing coordinate system according to the mapping relationship to obtain the mapped first screen;

[0167] The display module 1504 is configured to render and display the mapped first screen.

[0168] In some embodiments, determining the first centroid coordinate of the first object in the first screen to be displayed in the world coordinate system includes: determining the first bounding volume of the first object according to the first coordinates of the vertices of the first object in the first model coordinate system; wherein, the first model coordinate system is used to create the first object; determining the third coordinate of the centroid of the first bounding volume in the world coordinate system according to the second coordinates of the centroid of the first bounding volume in the first model coordinate system; and determining the first centroid coordinate of the first object in the world coordinate system according to the third coordinate.

[0169] In some embodiments, determining the first bounding volume of the first object according to the first coordinates of the vertices of the first object in the first model coordinate system includes: determining the second bounding volume of the first part according to the first coordinates of the vertices of the first part in the first object in the first model coordinate system; and determining the first bounding volume of the first object according to the second bounding volume and the first coordinates of the vertices of the second part in the first object in the first model coordinate system; the second part is different from the first part.

[0170] In some embodiments, determining the first bounding volume of the first object according to the first coordinates of the second part of the first object in the first model coordinate system and the second bounding volume includes: performing a first process according to the first coordinates of the i-th vertex of the second part of the first object in the first model coordinate system; wherein, the first process includes: if the first coordinates of the i-th vertex are not within the second bounding volume, updating the second bounding volume so that the updated second bounding volume includes the first coordinates of the i-th vertex; wherein, i is greater than 0 and less than or equal to the number of vertices of the second part; iteratively performing the first process until the updated second bounding volume includes the vertices of the second part, and determining the first bounding volume according to the updated second bounding volume.

[0171] In some embodiments, determining the first bounding volume of the first object according to the first coordinates of the vertices of the first object in the first model coordinate system includes: determining the distances of the first coordinates in multiple direction dimensions in the first model coordinate system according to the first coordinates of the vertices of the first object in the first model coordinate system; determining a third bounding volume according to the distances in the multiple direction dimensions; and determining the first bounding volume according to the third bounding volume and the first coordinates of the vertices of the first object in the first model coordinate system.

[0172] In some embodiments, determining the first bounding volume according to the third bounding volume and the first coordinates of the vertices of the first object in the first model coordinate system includes: performing a second process according to the first coordinates of the m-th vertex of the first object in the first model coordinate system, wherein the second process includes: if the first coordinates of the m-th vertex are not within the third bounding volume, updating the third bounding volume so that the updated third bounding volume includes the first coordinates of the m-th vertex; wherein, m is greater than 0 and less than or equal to the number of vertices of the first object; iteratively performing the second process until the updated third bounding volume includes the vertices of the first object, and determining the first bounding volume according to the updated third bounding volume.

[0173] In some embodiments, determining the mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates includes: determining the pose information of the object to which the actual viewpoint belongs; and determining the mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates and the pose information.

[0174] In some embodiments, the display device 150 further includes a third determination module, a fourth determination module, a fifth determination module, and an obtaining module. The third determination module is configured to determine the second centroid coordinates of the first object in the second picture to be displayed in the world coordinate system. The fourth determination module is configured to determine the fourth coordinates of the second object in the second picture in the world coordinate system according to the second centroid coordinates and the first centroid coordinates. The second object is at least one object different from the first object in the second picture. The fifth determination module is configured to determine the fifth coordinates of the first object in the second picture in the world coordinate system. The obtaining module is configured to map the fourth coordinates of the second object and the fifth coordinates of the first object in the second picture from the world coordinate system to the observation coordinate system respectively according to the mapping relationship to obtain the mapped second picture. The display module is configured to render and display the mapped second picture.

[0175] In some embodiments, the determining the fourth coordinates of the second object in the second picture in the world coordinate system according to the second centroid coordinates and the first centroid coordinates includes: determining the fifth coordinates of the second centroid coordinates in the second model coordinate system where the second object is located. The second model coordinate system is used to create the second object. Determining the sixth coordinates of the first centroid coordinates in the second model coordinate system. Determining the transformation relationship between the fifth coordinates and the sixth coordinates according to the fifth coordinates and the sixth coordinates. Determining the fourth coordinates of the second object in the second picture in the world coordinate system according to the transformation relationship.

[0176] The description of the above device embodiments is similar to that of the above method embodiments and has similar beneficial effects to those of the method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0177] It should be noted that the division of modules in the embodiments of the present application is illustrative, merely a logical function division, and there may be other division methods in actual implementation. In addition, each functional unit in the various embodiments of the present application may be integrated in a processing unit, or may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware, or in the form of a software functional unit, or in the form of a combination of software and hardware.

[0178] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of software functional modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing an electronic device to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs that can store program codes. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0179] The embodiments of the present application provide an electronic device. Figure 16 As shown in the structural schematic diagram of an electronic device provided by the embodiments of the present application, Figure 16 As shown, the electronic device 160 includes a memory 1601 and a processor 1602. The memory 1601 stores a computer program that can run on the processor 1602. When the processor 1602 executes the program, it implements the steps in the method provided in the above embodiments.

[0180] It should be noted that the memory 1601 is configured to store instructions and applications executable by the processor 1602, and can also cache the data to be processed or already processed by each module in the processor 1602 and the electronic device 160, and can be implemented by flash memory (Flash) or random access memory (RAM).

[0181] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the method provided in the above embodiments.

[0182] The embodiments of the present application provide a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the steps in the method provided in the above method embodiments.

[0183] It should be pointed out here that: the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments and have beneficial effects similar to those of the method embodiments. For the technical details not disclosed in the storage medium, storage medium, and device embodiments of the present application, please refer to the descriptions of the method embodiments of the present application for understanding.

[0184] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" or "in a possible implementation" or "example" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the "in one embodiment" or "in an embodiment" or "in some embodiments" or "in a possible implementation" or "example" and the like that appear throughout the specification do not necessarily refer to the same embodiment.

[0185] In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the magnitude of the serial numbers of the above processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0186] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments. The descriptions of the above embodiments tend to emphasize the differences between the embodiments, and the same or similar parts can be referred to each other. For the sake of brevity, they will not be repeated herein.

[0187] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0188] It should be noted that in this article, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the element defined by the statement "comprising one..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0189] In several embodiments provided by the present application, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. The above-described embodiments are merely illustrative. The features disclosed in several product embodiments provided by the present application can be arbitrarily combined without conflict to obtain new product embodiments.

[0190] As described above, it is only the implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A display method, the method comprising: Determine the first centroid coordinates of the first object in the first picture to be displayed in the world coordinate system; Determine a mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates; wherein the observation coordinate system is a coordinate system with the actual viewpoint in the physical space as the origin; According to the mapping relationship, mapping the first picture from the world coordinate system to the observation coordinate system to obtain a mapped first picture; The mapped first picture is rendered and then displayed.

2. The method according to claim 1, wherein determining the first centroid coordinates of the first object in the first picture to be displayed in the world coordinate system comprises: Determine a first bounding volume of the first object according to first coordinates of vertices of the first object in a first model coordinate system; wherein the first model coordinate system is used to create the first object; determining, according to the second coordinates of the centroid of the first bounding volume in the first model coordinate system, the third coordinates of the centroid of the first bounding volume in the world coordinate system; The first centroid coordinates of the first object in the world coordinate system are determined according to the third coordinates.

3. The method according to claim 2, wherein determining the first bounding volume of the first object according to the first coordinates of the vertices of the first object in the first model coordinate system comprises: determining a second bounding volume of a first part of the first object according to first coordinates of vertices of the first part in the first model coordinate system; A first bounding volume of the first object is determined according to the second bounding volume and first coordinates of vertices of a second part of the first object in the first model coordinate system; the second part is different from the first part.

4. The method according to claim 3, wherein determining the first bounding volume of the first object according to the second bounding volume and the first coordinates of the vertices of the second part of the first object in the first model coordinate system comprises: Execute a first process according to the first coordinate of the i-th vertex of the second part of the first object in the first model coordinate system; wherein the first process includes: if the first coordinate of the i-th vertex is not in the second bounding volume, update the second bounding volume so that the updated second bounding volume includes the first coordinate of the i-th vertex; wherein i is greater than 0 and less than or equal to the number of vertices of the second part; The first process is iteratively performed until the updated second bounding volume includes the vertices of the second part, and the first bounding volume is determined according to the updated second bounding volume.

5. The method according to claim 2, wherein determining a first bounding volume of the first object according to first coordinates of vertices of the first object in a first model coordinate system comprises: Determine, according to first coordinates of vertices of the first object in the first model coordinate system, distances of the first coordinates in multiple direction dimensions in the first model coordinate system; Determine a third bounding volume according to the distances in the multiple direction dimensions; The first bounding volume is determined according to the third bounding volume and first coordinates of vertices of the first object in a first model coordinate system.

6. The method according to claim 5, wherein determining the first bounding volume according to the third bounding volume and first coordinates of vertices of the first object in a first model coordinate system comprises: According to the first coordinate of the mth vertex of the first object in the first model coordinate system, a second process is performed, wherein the second process comprises: if the first coordinate of the mth vertex is not in the third bounding volume, updating the third bounding volume so that the updated third bounding volume includes the first coordinate of the mth vertex; wherein m is greater than 0 and less than or equal to the number of vertices of the first object; The second process is iteratively performed until the updated third bounding volume includes the vertices of the first object, and the first bounding volume is determined according to the updated third bounding volume.

7. The method according to any one of claims 1 to 6, wherein determining the mapping relationship between the world coordinate system and the observation coordinate system according to the first centroid coordinates comprises: Determining posture information of the object to which the actual viewpoint belongs; A mapping relationship between the world coordinate system and the observation coordinate system is determined according to the first centroid coordinates and the posture information.

8. The method according to any one of claims 1 to 7, further comprising: Determine the second centroid coordinates of the first object in the second picture to be displayed in the world coordinate system; determining fourth coordinates of a second object in the second picture in the world coordinate system according to the second centroid coordinates and the first centroid coordinates; the second object is at least one object in the second picture that is different from the first object; determining a fifth coordinate of the first object in the second picture in the world coordinate system; According to the mapping relationship, respectively map the fourth coordinate of the second object and the fifth coordinate of the first object in the second picture from the world coordinate system to the observation coordinate system to obtain a mapped second picture; The mapped second picture is rendered and then displayed.

9. The method according to claim 8, wherein determining the fourth coordinates of the second object in the second picture in the world coordinate system according to the second centroid coordinates and the first centroid coordinates comprises: Determine a fifth coordinate of the second centroid coordinate in a second model coordinate system where the second object is located; wherein the second model coordinate system is used to create the second object; Determine a sixth coordinate of the first centroid coordinate in the second model coordinate system; Determining a transformation relationship between the fifth coordinate and the sixth coordinate according to the fifth coordinate and the sixth coordinate; According to the transformation relationship, a fourth coordinate of the second object in the second picture in the world coordinate system is determined.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to any one of claims 1 to 9 when executing the program.

Images