Virtual reality based display method, device and computer readable medium

Abstract

The application provides a virtual reality-based display method, device and computer readable medium. The method applied to a first terminal device comprises: obtaining a center point coordinate and four corner point coordinates of a real screen; obtaining position information and angle information of a real camera relative to the real screen; wherein the field of view angle of the real camera covers the real screen; determining a center point coordinate and a default matrix of a virtual screen according to the position information and the angle information, and determining a projection matrix and a view matrix of a virtual camera; determining a position coordinate of the virtual camera relative to the center point of the virtual screen according to the default matrix, the projection matrix and the view matrix; determining an offset value according to the position coordinate, the center point coordinate of the real screen and the center point coordinate of the virtual screen; performing calibration processing on the position coordinate according to the offset value; and determining a target image according to the four corner point coordinates and the result of the calibration processing, so as to reduce the error of the position offset of the picture displayed by the extended screen.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and more particularly to a display method, device and computer-readable medium based on virtual reality. Background Technology

[0002] XR, or Extended Reality, is a term used to describe environments or interactive behaviors that combine virtual and real elements. XR products are highly innovative virtual scene applications. Compared to traditional 3D technology, these products do not require 3D glasses; users can experience the stunning effect of objects seemingly leaping off the screen simply by looking at it. XR products are generally based on a design using a real camera, a large screen, and an extended screen. They employ virtual reality fusion to cleverly blend virtual scenes with reality, seamlessly integrating camera-rendered images with real-time video footage, providing users with an ultimate virtual reality interactive experience and delivering stunning 3D effects. (See also...) Figure 1 , Figure 1 This is a schematic diagram illustrating the effect of an operator wearing 3D glasses interacting with a large screen. For users not wearing 3D glasses, an extended screen connected to the large screen can also provide the same stunning effect of objects seemingly leaping off the screen, and allow them to watch the operator interact with the screen. Figure 1 (Not shown).

[0003] In the implementation of XR products based on related technologies, it is necessary to construct a virtual screen in the virtual scene that is the same size as the real screen, based on the size of the real screen. However, the inventors have found that doing so will result in a large positional offset between the screen range of the virtual image and the screen range of the real image, that is, the image displayed on the extended screen will have a large positional offset, thus affecting the display effect. Summary of the Invention

[0004] One objective of this application is to provide a display method, device, and computer-readable medium based on virtual reality, at least to solve the technical problem in the related art where the positional offset of the screen displayed on the extended screen is large, thus affecting the display effect.

[0005] To achieve the above objectives, some embodiments of this application provide a virtual reality-based display method. The method is applied to a first terminal device and includes: acquiring the center coordinates and four corner coordinates of a real screen; acquiring the position and angle information of a real camera relative to the real screen; wherein the field of view of the real camera covers the real screen; determining the center coordinates and a default matrix of a virtual screen based on the position and angle information, and determining the projection matrix and view matrix of the virtual camera; determining the position coordinates of the virtual camera relative to the center point of the virtual screen based on the default matrix, the projection matrix, and the view matrix; determining an offset value based on the position coordinates, the center coordinates of the real screen, and the center coordinates of the virtual screen; calibrating the position coordinates based on the offset value; and determining a target image based on the four corner coordinates and the calibration result, for a second terminal device to display after superimposing and fusing the real image based on the target image.

[0006] Some embodiments of this application also provide a virtual reality-based display method, which is applied to a second terminal device and is communicatively connected to the first terminal device. The method includes: acquiring a target image from the first terminal device; wherein the target image is a target image obtained according to the method described above; and displaying a real image after overlaying and fusing it based on the target image.

[0007] Some embodiments of this application also provide a computer device, the device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the method described above.

[0008] Some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods described above.

[0009] Compared to existing technologies, the solution provided in this application obtains the center point coordinates and four corner point coordinates of the real screen, as well as the position and angle information of the real camera relative to the real screen. Based on the position and angle information, the center point coordinates and default matrix of the virtual screen are determined, and the projection matrix and view matrix of the virtual camera are determined. Then, based on the default matrix, the projection matrix, and the view matrix, the position coordinates of the virtual camera relative to the center point of the virtual screen are determined. Thus, based on the position coordinates, the center point coordinates of the real screen, and the center point coordinates of the virtual screen, an offset value is determined. Based on the offset value, the position coordinates are calibrated. Due to the calibration process, the error can be greatly reduced. Since the target image can be determined directly based on the four corner point coordinates and the calibration result in this application embodiment, it is not necessary to reconstruct the screen range in the virtual scene based on the real screen. In summary, after the second terminal device overlays and fuses the real image based on the target image, a display effect that perfectly fits the screen and is seamless can be obtained. Attached Figure Description

[0010] Figure 1 This is an exemplary schematic diagram illustrating the effect of an operator wearing 3D glasses interacting with a large screen in related technologies.

[0011] Figure 2 An exemplary flowchart of a virtual reality-based display method provided in Embodiment 1 of this application;

[0012] Figure 3 This is an exemplary schematic diagram of a system corresponding to a virtual reality-based display method provided in Embodiment 1 of this application;

[0013] Figure 4 This is an exemplary flowchart of a sub-step of step S104 in a virtual reality-based display method provided in Embodiment 2 of this application.

[0014] Figure 5 This is an exemplary flowchart of a sub-step of step S105 in a virtual reality-based display method provided in Embodiment 3 of this application;

[0015] Figure 6 An exemplary flowchart of a virtual reality-based display method provided in Embodiment 5 of this application;

[0016] Figure 7 This is an exemplary structural diagram of a computer device provided in Embodiment Six of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The following terms are used in this document.

[0019] The MVP matrix consists of three matrices: ModelQ, View, and Projection.

[0020] NDC coordinates, or Normalized Device Coordinates, is a small space where the x, y, and z values ​​range from -1.0 to 1.0.

[0021] Example 1

[0022] Embodiment 1 of this application provides a display method based on virtual reality. The method is applied to a first terminal device and includes the following steps: Figure 2 As shown:

[0023] Step S101: Obtain the coordinates of the center point and the four corner points of the actual screen;

[0024] Step S102: Obtain the position and angle information of the real camera relative to the real screen; wherein the field of view of the real camera covers the real screen;

[0025] Step S103: Based on the position information and angle information, determine the center point coordinates and default matrix of the virtual screen, and determine the projection matrix and view matrix of the virtual camera;

[0026] Step S104: Determine the position coordinates of the virtual camera relative to the center point of the virtual screen based on the default matrix, the projection matrix, and the view matrix;

[0027] Step S105: Determine the offset value based on the position coordinates, the center point coordinates of the real screen, and the center point coordinates of the virtual screen;

[0028] Step S106: The position coordinates are calibrated according to the offset value;

[0029] Step S107: Based on the coordinates of the four corner points and the result of the calibration process, a target image is determined so that the second terminal device can overlay and fuse the real image based on the target image for display.

[0030] Specifically, in some examples, such as Figure 3 As shown, the first terminal device 10 can be a host that is communicatively connected to the real screen 20. The first terminal device 10 has a first application installed, and the host operates on the real screen 20 through the first application. For example, the first application is ModelViewer. The second terminal device can be an extended screen 30, which is a display interface for displaying naked-eye 3D corresponding to the real screen 20. The real camera is 40, and the field of view of the real camera 40 covers the real screen 20.

[0031] Specifically, in some examples, steps S101 to S106 described above can be processed by the CPU of the first terminal device, and step S107 described above can be processed by the GPU of the first terminal device.

[0032] Specifically, in some examples, the real screen can be any screen, such as a flat screen or a curved screen, etc., without specific limitations. The first terminal device can obtain the center point coordinates and four corner point coordinates of the real screen, as well as the position and angle information of the real camera relative to the real screen, through a second terminal device connected to it. The center point coordinates and four corner point coordinates of the real screen can be the coordinates of the center point and the four corner points in the NDC coordinate system.

[0033] Specifically, in some examples, the center coordinates (center) and default matrix (m) of the virtual screen in the virtual scene can be determined based on the position and angle information, and the projection matrix (p) and view matrix (v) of the virtual camera can also be determined. It is worth mentioning that in some examples, in addition to obtaining the position and angle information of the real camera relative to the real screen, other parameter information of the real camera can also be obtained simultaneously. This parameter information may include, but is not limited to: the field of view of the real camera, the rotation parameters of the real camera relative to the real screen, the rotation axis and angle of the real camera, the rotation axis matrix of the real camera, etc.

[0034] It is understandable that in the implementation of XR products using related technologies, it is necessary to construct a virtual screen in the virtual scene that is the same size as the real screen. In other words, assuming the real screen is 1 meter wide and 1 meter high, a virtual screen of the same size needs to be constructed in the virtual scene. This process results in a significant positional offset between the virtual and real screen areas, meaning the image displayed on the extended screen is significantly off-center, thus affecting the display quality.

[0035] In the solution provided in this application embodiment, after obtaining the center point coordinates and four corner point coordinates of the real screen, as well as the position and angle information of the real camera relative to the real screen, the center point coordinates and default matrix of the virtual screen are determined based on the position and angle information, and the projection matrix and view matrix of the virtual camera are determined. Then, based on the default matrix, the projection matrix, and the view matrix, the position coordinates of the virtual camera relative to the center point of the virtual screen are determined. Thus, based on the position coordinates, the center point coordinates of the real screen, and the center point coordinates of the virtual screen, an offset value is determined. Based on the offset value, the position coordinates are calibrated. Because calibration is performed, the error can be greatly reduced. Since the target image can be determined directly based on the four corner point coordinates and the result of calibration in this application embodiment, it is not necessary to reconstruct the screen range in the virtual scene based on the real screen. In summary, after the second terminal device superimposes and fuses the real image based on the target image, a display effect that perfectly fits the screen and is seamless can be obtained.

[0036] Example 2

[0037] Embodiment 2 of this application is an improvement on Embodiment 1. Specifically, in Embodiment 2, determining the position coordinates of the virtual camera relative to the center point of the virtual screen based on the default matrix, the projection matrix, and the view matrix may include the following steps: Figure 4 As shown:

[0038] Step S1041: Determine the MVP matrix based on the default matrix, the projection matrix, and the view matrix;

[0039] Step S1042: Determine the position coordinates of the virtual camera relative to the center point of the virtual screen based on the MVP matrix.

[0040] Specifically, in some examples, in step S1041, the default matrix m, the projection matrix p of the virtual camera, and the view matrix v can be multiplied together to obtain the MVP matrix.

[0041] Furthermore, in some examples, in step S1042, the position coordinates new_center of the virtual camera relative to the center point of the virtual screen can be determined according to the MVP matrix, and then the position coordinates new_center can be converted into coordinates in the NDC coordinate system.

[0042] Example 3

[0043] Embodiment 3 of this application is an improvement on Embodiment 1. Specifically, in Embodiment 3, the center point coordinates of the real screen are two-dimensional coordinates; determining the offset value based on the position coordinates, the center point coordinates of the real screen, and the center point coordinates of the virtual screen may include the following steps: Figure 5 As shown:

[0044] Step S1051: Obtain the distance value of the virtual camera relative to the center point of the virtual screen based on the position coordinates;

[0045] Step S1052: Determine the three-dimensional coordinates based on the distance value and the two-dimensional coordinates of the center point of the real screen;

[0046] Step S1053: Based on the three-dimensional coordinates, the default matrix, the projection matrix, and the view matrix, obtain the new center point coordinates of the virtual screen relative to the virtual camera;

[0047] Step S1054: Determine the offset value based on the difference between the center point coordinates of the new virtual screen and the center point coordinates of the virtual screen.

[0048] Specifically, in some examples, in step S1051, the z-axis coordinate in the position coordinate new_center can be used as the distance value of the virtual camera relative to the center point of the virtual screen, denoted as new_center.z.

[0049] Specifically, in some examples, in step S1052, the distance value new_center.z of the virtual camera relative to the center point of the virtual screen and the two-dimensional coordinates of the center point of the real screen can be combined to obtain the three-dimensional coordinates new_center3d.

[0050] Specifically, in some examples, in step S1053, the default matrix m, the projection matrix p of the virtual camera, and the view matrix v can be multiplied to obtain the MVP matrix. Multiplying the three-dimensional coordinates new_center3d with the inverse matrix of the MVP matrix yields the coordinates center1 of the new virtual screen relative to the virtual camera.

[0051] Specifically, in some examples, in step S1054, the offset value oft is obtained by calculating the difference between the new virtual screen center point coordinates center1 relative to the virtual camera and the center point coordinates center of the virtual screen. It can be understood that since the virtual screen and the virtual camera are in the same spatial coordinate system, the offset value oft can be used as the offset value of the virtual camera.

[0052] It should be noted that Embodiment 3 of this application may also be an improvement based on Embodiment 2.

[0053] Example 4

[0054] Embodiment 4 of this application is an improvement on Embodiment 1. Specifically, in Embodiment 4 of this application, the calibration process of the position coordinates based on the offset value may include: determining the new position coordinates of the virtual camera after calibration based on the offset value and the position coordinates, and generating a new view matrix based on the new position coordinates.

[0055] Determining the target image based on the coordinates of the four corner points and the result of calibration processing may include: determining the target image based on the coordinates of the four corner points, the default matrix, the projection matrix, and the new view matrix.

[0056] Specifically, in some examples, the offset value oft can be added to the position coordinates to obtain the new position coordinates after the virtual camera has been calibrated, and a new view matrix v' can be generated based on the new position coordinates.

[0057] Furthermore, in some embodiments of this application, determining the target image based on the coordinates of the four corner points, the default matrix, the projection matrix, and the new view matrix may include: determining the position information of the target object after projection in the virtual scene based on the default matrix, the projection matrix, and the new view matrix; and determining the target image based on the position information and the rectangular range formed by the coordinates of the four corner points.

[0058] Furthermore, in some embodiments of this application, determining the target image based on the location information and the rectangular range formed by the coordinates of the four corner points may include: removing the location coordinates outside the rectangular range from the location information and retaining the location coordinates within the rectangular range from the location information; and determining the target image based on the location coordinates within the rectangular range.

[0059] Specifically, in some examples, the target image can be determined using a rendering pipeline in related technologies. For instance, the geometric information of the virtual scene model, such as the vertex positions and normals in the virtual scene space, can be pre-input into the vertex shader. The default matrix m, the projection matrix p, and the new view matrix v' are multiplied to obtain the MV'P matrix. Then, the vertex positions can be multiplied by the MV'P matrix to determine the position information of the target object after projection into the virtual scene. Those skilled in the art will understand that the information of the triangular facets can be obtained using vector calculation methods in the art, and their positions in three-dimensional space can be determined based on this information.

[0060] Furthermore, in some examples, the result of multiplying the vertex position and the MV'P matrix can be converted into NDC coordinates to obtain ndc_pos. Then, ndc_pos is rasterized and sent to the source / pixel shader. At this point, the source / pixel shader can obtain the coordinates of the four corner points and the rasterized ndc_pos. It can then use an existing algorithm to determine whether the coordinates represented by ndc_pos are within the rectangular area formed by the coordinates of the four corner points. Coordinates outside the rectangular area are discarded, while those within the rectangular area are retained. Thus, based on the position coordinates within the rectangular area, the target image is determined.

[0061] It should be noted that Embodiment 4 of this application may also be an improvement based on Embodiment 2 and / or Embodiment 3.

[0062] Example 5

[0063] Embodiment 5 of this application provides another virtual reality display method, which is applied to a second terminal device. The second terminal device is communicatively connected to the first terminal device as provided in Embodiment 1. The method may include the following steps, such as... Figure 6 As shown:

[0064] Step S201: Obtain a target image from the first terminal device; wherein the target image is a target image obtained according to the method described in any one or more of the embodiments from Embodiment 1 to Embodiment 4;

[0065] Step S202: Based on the target image, the real image is superimposed and fused before being displayed. It can be understood that the out-of-screen effect displayed based on the superimposed and fused result is that a 3D effect can be seen even without wearing 3D glasses or other devices.

[0066] Furthermore, in some embodiments of this application, the method may further include the following steps:

[0067] Obtain the two-dimensional coordinates of the center point and the four corner points of the actual screen;

[0068] The position and angle information of the real camera relative to the real screen are obtained; wherein the field of view of the real camera covers the real screen;

[0069] The two-dimensional coordinates of the center point and the four corner points of the real screen are converted into the coordinates of the center point and the four corner points in the NDC coordinate system, respectively.

[0070] The coordinates of the center point and the four corner points in the converted NDC coordinate system, as well as the position and angle information of the real camera relative to the real screen, are sent to the first terminal device.

[0071] Specifically, in some examples, the second terminal device may include an extended screen, which may have a second application installed, such as XReal.

[0072] Specifically, in some examples, the real camera can be calibrated using a second application on the second terminal device. Since the field of view of the real camera covers the real screen, images captured by the real camera can be obtained. The two-dimensional coordinates of the center point and the four corner points of the real screen can be determined from these images. Furthermore, OpenCV can be used to obtain the position and angle information of the real camera relative to the real screen. Then, the two-dimensional coordinates of the center point and the four corner points of the real screen can be converted to the coordinates of the center point and the four corner points in the NDC coordinate system. Thus, the coordinates of the center point and the four corner points in this NDC coordinate system fall within the range of (-1, 1). Finally, the coordinates of the center point and the four corner points located in the (-1, 1) range can be saved to the registry. Furthermore, the coordinates of the center point and the four corner points in the NDC coordinate system obtained after conversion in the registry, as well as the position and angle information of the real camera relative to the real screen, can be sent to the first terminal device. Thus, the first terminal device can obtain the coordinates of the center point and the four corner points of the real screen, as well as the position and angle information of the real camera relative to the real screen, which corresponds to steps S101 and S102 in Embodiment 1.

[0073] Example 6

[0074] Embodiment Six of this application also provides a computer device, which may include the first terminal device and / or the second terminal device, and the structure of the device is as follows: Figure 7As shown, the device includes a memory 50 for storing computer-readable instructions and a processor 60 for executing the computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor is triggered to execute the method described thereon.

[0075] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0076] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0077] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0078] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] In another aspect, embodiments of this application also provide a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer-readable instructions, which may be executed by a processor to implement the steps of the methods and / or technical solutions of the various embodiments of this application.

[0081] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0083] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0084] Furthermore, this application also provides a computer program stored in a computer device, which causes the computer device to execute the method for executing the control code.

[0085] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0086] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A display method based on virtual reality, characterized in that, The method is applied to a first terminal device, and the method includes: Obtain the coordinates of the center point and four corner points of the actual screen; Acquire the position and angle information of the real camera relative to the real screen; wherein the field of view of the real camera covers the real screen; Based on the location and angle information, determine the center point coordinates and default matrix of the virtual screen, and determine the projection matrix and view matrix of the virtual camera; The position coordinates of the virtual camera relative to the center point of the virtual screen are determined based on the default matrix, the projection matrix, and the view matrix. The offset value is determined based on the position coordinates, the center point coordinates of the real screen, and the center point coordinates of the virtual screen; The position coordinates are calibrated based on the offset value; Based on the coordinates of the four corner points and the results of the calibration process, a target image is determined so that the second terminal device can overlay and fuse the real image based on the target image for display.

2. The method according to claim 1, characterized in that, Determining the position coordinates of the virtual camera relative to the center point of the virtual screen based on the default matrix, the projection matrix, and the view matrix includes: The MVP matrix is ​​determined based on the default matrix, the projection matrix, and the view matrix; Based on the MVP matrix, determine the position coordinates of the virtual camera relative to the center point of the virtual screen.

3. The method according to claim 1, characterized in that, The coordinates of the center point of the real screen are two-dimensional coordinates; The step of determining the offset value based on the position coordinates, the center point coordinates of the real screen, and the center point coordinates of the virtual screen includes: Based on the location coordinates, obtain the distance value of the virtual camera relative to the center point of the virtual screen; The three-dimensional coordinates are determined based on the distance value and the two-dimensional coordinates of the center point of the actual screen; Based on the three-dimensional coordinates, the default matrix, the projection matrix, and the view matrix, the new center point coordinates of the virtual screen relative to the virtual camera are obtained; The offset value is determined based on the difference between the center point coordinates of the new virtual screen and the center point coordinates of the virtual screen.

4. The method according to any one of claims 1 to 3, characterized in that, The step of calibrating the position coordinates based on the offset value includes: determining the new position coordinates of the virtual camera after calibration based on the offset value and the position coordinates, and generating a new view matrix based on the new position coordinates; The step of determining the target image based on the coordinates of the four corner points and the result of calibration processing includes: determining the target image based on the coordinates of the four corner points, the default matrix, the projection matrix, and the new view matrix.

5. The method according to claim 4, characterized in that, The step of determining the target image based on the coordinates of the four corner points, the default matrix, the projection matrix, and the new view matrix includes: Based on the default matrix, the projection matrix, and the new view matrix, determine the position information of the target object after projection in the virtual scene; The target image is determined based on the location information and the rectangular range formed by the coordinates of the four corner points.

6. The method according to claim 5, characterized in that, Determining the target image based on the location information and the rectangular area formed by the coordinates of the four corner points includes: Remove the position coordinates located outside the rectangular area from the position information, and retain the position coordinates located within the rectangular area from the position information; The target image is determined based on the position coordinates within the rectangular area.

7. A display method based on virtual reality, characterized in that, The method is applied to a second terminal device, which is communicatively connected to a first terminal device, and the method includes: The target image is obtained according to the first terminal device; wherein the target image is obtained according to the method described in any one of claims 1 to 6; The target image is overlaid and fused with the real image before being displayed.

8. The method according to claim 7, characterized in that, The method further includes: Obtain the two-dimensional coordinates of the center point and the four corner points of the actual screen; The position and angle information of the real camera relative to the real screen are obtained; wherein the field of view of the real camera covers the real screen; The two-dimensional coordinates of the center point and the four corner points of the real screen are converted into the coordinates of the center point and the four corner points in the NDC coordinate system, respectively. The coordinates of the center point and the four corner points in the converted NDC coordinate system, as well as the position and angle information of the real camera relative to the real screen, are sent to the first terminal device.

9. A computer device, characterized in that, The device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the method as described in any one of claims 1 to 8.

10. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as claimed in any one of claims 1 to 8.

Images