Picture display method and device, equipment and storage medium

By receiving and processing the display screen in a virtual reality device and creating a virtual display with consistent locations in a virtual reality scene, the problem of insufficient correlation between digital class cards and virtual reality devices is solved, and the immersion of virtual reality teaching is improved. Feeling and real.

CN119065544APending Publication Date: 2024-12-03GUANGZHOU SHIYUAN ELECTRONICS CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310597321.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Digital class cards and virtual reality equipment are independent devices on campus, and are not very relevant, resulting in insufficient immersion and realism in virtual reality teaching.

Method used

The virtual reality device receives the screen of the display and creates a virtual display in the virtual reality scene to ensure that the position and attitude of the virtual display are consistent with the display in the real world, thereby realizing linkage display and enhancing the display of the virtual reality device. Spatial position perception.

Benefits of technology

It improves the realism and immersion of the display screen in virtual reality devices, enhances the display effect of virtual reality scenes, and increases the correlation between the display screen and virtual reality devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065544A_ABST
    Figure CN119065544A_ABST
Patent Text Reader

Abstract

The invention provides a picture display method and device, equipment and a storage medium, the picture display method and device are applied to virtual reality equipment in a virtual-real interconnection system, the virtual-real interconnection system comprises the virtual reality equipment and a display screen, the method comprises the steps that a first picture sent by the display screen is received, and the first picture is a picture currently displayed in the display screen; a virtual display screen is created in a virtual reality scene corresponding to the virtual reality equipment, and the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world; and displaying the first picture in the virtual display screen. According to the technical scheme, the virtual reality equipment can sense the spatial position of the display screen, so that the picture in the display screen is more reasonably displayed in the virtual reality equipment, and the sense of reality of the picture displayed in the virtual reality scene is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of display, and in particular to a screen display method, device, equipment and storage medium. Background Art

[0002] Digital class signs are intelligent interactive and display devices on campus. They are usually installed near the door of the classroom and can be used to collect and transmit student big data and display various campus information (such as class information and class schedule). With the development of science and technology, virtual reality equipment has also been applied to campus. Virtual reality equipment can be used to demonstrate immersive virtual reality scenes in the classroom and conduct virtual reality teaching.

[0003] At present, digital class signs and virtual reality devices are two independent devices on campus, and the correlation between the two is not strong. Summary of the invention

[0004] The present application provides a screen display method, device, equipment and storage medium to solve the technical problem that virtual reality equipment and digital class signs are not strongly related.

[0005] In a first aspect, a method for displaying a picture is provided, which is applied to a virtual reality device in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes the virtual reality device and a display screen, and the method includes:

[0006] Receiving a first picture sent by the display screen, where the first picture is a picture currently displayed on the display screen;

[0007] Creating a virtual display screen in a virtual reality scene corresponding to the virtual reality device, wherein the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world;

[0008] The first screen is displayed on the virtual display screen.

[0009] In this technical solution, after receiving the first picture sent by the display screen, the virtual reality device creates a virtual display screen in the virtual reality scene and then displays the first picture on the virtual display screen, thereby realizing the linkage display between the virtual reality device and the display screen and increasing the correlation between the display screen and the virtual reality device; and, since the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world, the virtual reality device is able to perceive the spatial position of the display screen, making the display of the picture on the display screen in the virtual reality device more reasonable and able to enhance the realism of the picture displayed in the virtual reality scene.

[0010] In combination with the first aspect, in a possible implementation, the creating a virtual display screen in the virtual reality scene corresponding to the virtual reality device includes: obtaining first coordinates and second coordinates corresponding to the display screen, the second coordinates being used to indicate the position of the display screen in a target image, the target image being an image obtained by the virtual reality device currently photographing the display screen; determining a first projection transformation matrix corresponding to the virtual reality device according to the first coordinates and the second coordinates, the first projection transformation matrix being used to reflect the conversion relationship between a real world coordinate system and a camera coordinate system; performing coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a third coordinate, the third coordinate being used to indicate the position of the display screen in the virtual world; and creating the virtual display screen in the virtual reality scene according to the third coordinate. Determining the projection transformation matrix of the virtual reality device according to the coordinates reflecting the position of the display screen in the real world and the image, and transforming the coordinates reflecting the display screen in the real world according to the projection transformation matrix to obtain the coordinates reflecting the position of the display screen in the virtual world, and creating the virtual display screen, so that a virtual display screen having the same position and posture as the display screen is created in the virtual reality scene.

[0011] In combination with the first aspect, in a possible implementation, the first coordinate is transformed according to the first projection transformation matrix to obtain a third coordinate, including: according to the first projection transformation matrix, the first coordinate is transformed to obtain a fourth coordinate, the fourth coordinate is a coordinate in a camera coordinate system; according to the second projection transformation matrix corresponding to the virtual camera in the virtual reality scene, the fourth coordinate is transformed to obtain the third coordinate, the second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system. By first converting the coordinates reflecting the real world position to the camera coordinate system, and then converting them to the virtual world coordinate system, the conversion from the real world to the virtual world coordinates can be achieved.

[0012] In combination with the first aspect, in a possible implementation, obtaining the second coordinates corresponding to the display screen includes: identifying a preset mark in the target image; and determining the second coordinates according to a position of the preset mark in the target image. Marking the display screen with the preset mark facilitates rapid determination of the coordinates of the display screen in the image.

[0013] In combination with the first aspect, in a possible implementation, the method further includes: in response to a screen switching instruction, switching to display a next screen corresponding to the first screen in the virtual display screen, wherein the next screen belongs to a target screen set, and the target screen set is a set of screens that can be displayed in the display screen. By switching to display the screen that can be displayed in the display screen in the virtual display screen, the display and viewing of the screen that can be displayed in the display screen can be completed without affecting the display of the display screen.

[0014] In combination with the first aspect, in a possible implementation, displaying the first picture on the virtual display screen includes: acquiring a target three-dimensional model corresponding to the first picture; and displaying the target three-dimensional model in the virtual reality scene. By displaying the three-dimensional model corresponding to the picture in the virtual reality scene, the display effect of the picture displayed on the display screen can be enriched.

[0015] In combination with the first aspect, in a possible implementation, after displaying the target three-dimensional model in the virtual reality scene, the method further includes: in response to a state transformation instruction acting on the target three-dimensional model, switching the state of the target three-dimensional model, and displaying the target three-dimensional model after the switching state in the virtual reality scene. According to the model state transformation instruction, the three-dimensional model corresponding to the picture after the switching state is displayed on the virtual display screen can enrich the display effect of the picture displayed on the display screen.

[0016] In combination with the first aspect, in a possible implementation, the first picture is obtained by the display screen capturing and encoding a picture currently displayed on the display screen; and displaying the first picture on the virtual display screen includes: decoding the first picture; and displaying the decoded picture on the virtual display screen. By streaming the picture on the display screen to the virtual display screen for display, the clarity of the picture in the virtual reality scene can be guaranteed.

[0017] In conjunction with the first aspect, in a possible implementation, before receiving the first picture sent by the display screen, the method further includes: obtaining a communication address of the display screen, and establishing a communication connection with the display screen based on the communication address of the display screen. By obtaining the communication address of the display screen and establishing a communication connection with the display screen, subsequent data transmission between the virtual reality device and the display screen can be facilitated.

[0018] In a second aspect, a picture display device is provided, which is applied to a virtual reality device in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes the virtual reality device and a display screen, and the device includes:

[0019] A picture receiving module, used for receiving a first picture sent by the display screen, where the first picture is the picture currently displayed on the display screen;

[0020] A creation module, used to create a virtual display screen in a virtual reality scene corresponding to the virtual reality device, wherein the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world;

[0021] A display module is used to display the first picture on the virtual display screen.

[0022] In a third aspect, a virtual reality device is provided, comprising a memory, a display panel, and one or more processors, wherein the memory and the display panel are connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory. When the one or more processors execute the one or more computer programs, the virtual reality device implements the screen display method of the first aspect.

[0023] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the screen display method of the first aspect.

[0024] The present application can achieve the following technical effects: realize the linkage display between the virtual reality device and the display screen, and increase the correlation between the display screen and the virtual reality device; and, because the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world, the virtual reality device can perceive the spatial position of the display screen, making the display of the picture on the display screen in the virtual reality device more reasonable, and enhancing the realism of the picture displayed in the virtual reality scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the system architecture of a virtual-real interconnection system provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a flow chart of a screen display method provided in an embodiment of the present application;

[0027] Figure 3 and Figure 4 A schematic diagram of a screen display provided in an embodiment of the present application;

[0028] Figure 5 is a structural schematic diagram of a picture display device provided in an embodiment of the present application;

[0029] Figure 6 It is a structural schematic diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0031] The technical solution of the present application is applicable to teaching scenarios, wherein the present application aims to associate a display device and a virtual reality device in a teaching scenario so that the display device and the virtual reality device are mutually associated and perceived, so as to expand more new applications.

[0032] The technical solution of the present application can be applied to a virtual reality interconnection system composed of a virtual reality device and a display screen. For ease of understanding, the virtual reality interconnection system of the present application is first introduced. Figure 1 , Figure 1 A schematic diagram of the system architecture of a virtual-real interconnection system provided in an embodiment of the present application. Figure 1 As shown, the virtual-reality interconnection system 10 includes a virtual reality device 101 and a display screen 102, and the display screen 102 can be a digital class sign in the teaching scene. The virtual reality device 101 is a terminal with a virtual reality function, that is, the virtual reality device 101 has the ability to render and display virtual reality scenes. Specifically, the virtual reality device 101 can be VR glasses, VR helmets, and so on. The virtual reality device 101 can also have a camera, and the camera is used to shoot the environment outside the virtual reality device 101 to obtain an external environment picture. In this application, the camera is used to shoot the external environment picture in the teaching scene, such as shooting the environment picture inside and outside the classroom.

[0033] Virtual-reality interconnection services may be respectively run in the virtual reality device 101 and the display screen 102. The virtual-reality interconnection service refers to a service that can be perceived and interacted with by the virtual reality device 101 and the display screen 102. The display screen 102 may serve as a service provider of the virtual-reality interconnection service, and the virtual reality device 101 may serve as a service user of the virtual-reality interconnection service.

[0034] In order to enable the virtual-reality interconnection service to operate normally between the virtual reality device 101 and the display screen 102, the virtual reality device 101 and the display screen 102 need to establish a communication connection to realize the interconnection between the virtual device 101 and the display screen 102, so as to maintain the virtual interconnection service based on the communication connection. The virtual reality device 101 and the display screen 102 can be connected to the same communication network based on Ethernet communication or wireless fidelity (wire fidelity, WiFi) communication, and obtain the communication information (such as communication address, communication protocol, service type, port number, etc.) required by both parties for the virtual-reality interconnection service based on device identification technology, so as to establish a communication connection that can maintain the virtual-reality interconnection service.

[0035] The virtual reality device 101 and the display screen 102 can obtain the communication information required for both parties to perform virtual-reality interconnection services in one or more of the following ways.

[0036] (1) Based on near field communication (NFC)

[0037] In this way, an NFC chip may be provided in the display screen, which stores the communication information required for the display screen to perform virtual-reality interconnection services; when the virtual reality device is within the NFC communication range of the display screen, the virtual reality device can read the communication information of the NFC chip of the display screen; the virtual reality device can establish a communication connection with the display screen based on the communication information that can maintain the virtual-reality interconnection service.

[0038] (2) QR code-based method

[0039] In this way, a QR code may be posted on the display screen, and the QR code carries the device identification code of the display screen; the virtual reality device may obtain the device identification code of the display screen by scanning the QR code; the virtual reality device may obtain the communication information of the display screen from the server based on the device identification code, and the virtual reality device may establish a communication connection with the display screen based on the communication information that can maintain the virtual-reality interconnection service. In the case where the display screen has established an exclusive connection with the server through the device identification code, the virtual reality device may also connect to the server based on the device identification code, thereby establishing a communication connection with the display screen that can maintain the virtual-reality interconnection service.

[0040] Alternatively, a QR code may be displayed on the display screen, which may carry the communication information of the display screen. The virtual reality device may obtain the communication information of the display screen by scanning the QR code displayed on the display screen. The virtual reality device may establish a communication connection with the display screen based on the communication information to maintain the virtual-reality interconnection service.

[0041] After the communication connection that can maintain the virtual-reality interconnection service is established, the virtual reality device and the display screen can form a virtual-reality interconnection system. Based on the virtual-reality interconnection system, the technical solution of the present application can be implemented. Among them, the technical solution of the present application can be specifically applied to the virtual reality device of the virtual-reality interconnection system.

[0042] The technical solution of this application is described in detail below.

[0043] See also Figure 2 , Figure 2 A flowchart of a screen display method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0044] S201, receiving a first picture sent by a display screen.

[0045] The display screen may be a display screen in the virtual-real interconnection system, and the display screen may be, for example, the aforementioned Figure 1In the display screen 102, the first picture is the picture currently displayed on the display screen. Figure 3 or Figure 4 As shown in P1.

[0046] The display screen and virtual reality device in the virtual-reality interconnection system can transmit the picture in the display screen to the virtual reality device based on the video streaming method. Specifically, the display screen can capture the picture currently displayed in the display screen based on the full-screen capture or window capture method to obtain a screenshot picture; then use any encoding method such as H264, H265, VP8 and VP9 to encode the screenshot picture to obtain a first picture; then send the first picture to the virtual reality device through the data stream transmission channel between the virtual reality device, and after the virtual reality device receives the first picture, it uses the decoding method corresponding to the encoding method used by the display screen to decode the received first picture to obtain a decoded picture to be displayed. The data stream transmission channel between the display screen and the virtual reality device can be established based on a preset stream transmission protocol, which can be any one of the video streaming protocols such as HTTP live streaming (HLS) protocol based on HTTP, dynamic adaptive streaming over HTTP (DASH) protocol based on HTTP, real time messaging protocol (RTMP) protocol, web real time communication (WebRTC) protocol, and HTTP smooth stream protocol based on HTTP. By streaming, the picture on the display screen is displayed on the virtual display screen, which can ensure the clarity of the picture in the virtual reality scene.

[0047] Optionally, before receiving the first picture sent by the display screen, the virtual reality device may also obtain the communication address of the display screen, and establish a communication connection with the display screen based on the communication address of the display screen. Specifically, the virtual reality device may obtain the communication address of the display screen through the aforementioned NFC-based method or QR code-based method, and establish a communication connection with the display screen based on the communication address of the display screen. By establishing a communication connection with the display screen, it is possible to facilitate subsequent communication transmission between the virtual reality device and the display screen to perform virtual-reality interconnection services.

[0048] S202: Create a virtual display screen in the virtual reality scene corresponding to the virtual reality device.

[0049] Here, the virtual reality scene corresponding to the virtual reality device refers to the virtual three-dimensional scene currently being displayed in the virtual reality device. The virtual display screen refers to the display screen in the virtual reality scene used to display the image currently being displayed. The virtual display screen can be presented as a two-dimensional surface, a three-dimensional thin sheet, or a three-dimensional model in the virtual reality scene, and this application does not limit this. For example, the virtual display screen can be as follows: Figure 3 As shown in w in , it appears as a polyhedron.

[0050] Among them, the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world. The position and posture of the display screen in the real world refers to the position and posture of the display screen relative to the virtual reality device in the real world, that is, the position and posture of the display screen relative to the user wearing the virtual reality device. The position and posture of the virtual display screen in the virtual reality scene refers to the position and posture of the virtual display screen relative to the virtual camera in the virtual reality scene, that is, the position and posture of the virtual display screen relative to the user position in the virtual reality scene. For example, the display screen is set perpendicular to the ground in the real world, the display screen is located within the line of sight of the user wearing the virtual reality device, and the right front side of the user wearing the virtual reality device is 60° to the user's line of sight, then the virtual display screen is located at 60° to the right front side of the user position in the virtual reality scene. The shape of the virtual display screen can be the same as the display screen, or it can be different from the display screen.

[0051] In a feasible implementation, the relative position relationship between the display screen and the virtual reality device can be determined based on visual recognition, so as to create a virtual display screen in the virtual reality scene whose position and posture are the same as the position and posture of the display screen in the real world.

[0052] Specifically, a virtual display screen can be created through the following steps A1-A4:

[0053] A1. Obtain the first coordinate and the second coordinate corresponding to the display screen.

[0054] Here, the first coordinate is used to represent the position of the display screen in the real world. The display screen can be abstracted as a rectangle in the real world. The position coordinates of the four vertices of the rectangle can be used to represent the rectangle. Therefore, the position coordinates of the vertices corresponding to the four corners of the display screen can be used to represent the position of the display screen in the real world, that is, the first coordinate includes the position coordinates of the vertices corresponding to the four corners of the display screen. The first coordinate may include (x 11 ,y 11 ,0)、(x 12 ,y 12 ,0)、(x 13 , x 13 ,0)、(x14 ,y 14 , 0), (x1, y1, 0) represents the coordinates in the real world coordinate system. The vertex corresponding to one of the corners of the display screen can be used as the origin of the coordinates of the real world coordinate system, and the position coordinates of the vertices corresponding to the other three corners of the display screen in the real world coordinate system can be determined according to the size of the display screen.

[0055] Specifically, the first coordinate of the display screen can be pre-calibrated and saved in the display screen. The virtual reality device can obtain the first coordinate corresponding to the display screen based on the communication connection with the display screen. Alternatively, the first coordinate of the display screen can also be pre-calibrated and saved in a QR code posted on the display screen, and the virtual reality device can obtain the first coordinate corresponding to the display screen when scanning the QR code. Alternatively, the first coordinate of the display screen can also be pre-calibrated and saved in a QR code displayed on the display screen, and the virtual reality device can obtain the first coordinate corresponding to the display screen when scanning the QR code to establish a communication connection. This application does not limit the implementation method of obtaining the first coordinate.

[0056] The second coordinate is used to indicate the position of the display screen in the target image. The target image is the image obtained by the virtual reality device currently shooting the display screen. The second coordinate is a pixel coordinate. The second coordinate changes with the relative position relationship between the virtual reality device and the display screen, that is, the second coordinate changes with the shooting angle when the virtual reality device shoots the display screen. Similar to the first coordinate, the second coordinate includes the position coordinates of the vertices corresponding to the four corners of the display screen in the image. The second coordinate may include (x 21 ,y 21 )、(x 22 ,y 22 )、(x 23 ,y 23 )、(x 24 ,y 24 ), where (x2, y2) represents the coordinates in the pixel coordinate system, (x 21 ,y 21 ) can be combined with the first coordinate (x 11 , x 11 , 0) corresponds to the same angle, (x 22 ,y 22 ) can be combined with the first coordinate (x 12 ,y 12 , 0) corresponds to the same angle, (x 23 ,y 23 ) can be combined with the first coordinate (x 13 ,y 13 , 0) corresponds to the same angle, (x 24 ,y 24 ) can be combined with the first coordinate (x 14 ,y14 , 0) correspond to the same angle.

[0057] The virtual reality device can perform image recognition on the target image to identify the positions of the four corners of the display screen in the target image and obtain the second coordinates.

[0058] In some possible situations, clearly identifiable features can be set in advance for the four corners of the display screen, so that the positions of the four corners of the display screen in the image can be quickly and accurately identified. Among them, clearly identifiable features can be set in advance for the display screen or the four corners of the display screen in the following ways: a1, a special appearance can be set for the display screen, in which the four corners of the display screen are designed as clearly distinguishable preset patterns, and the preset patterns can be used as marks of the four corners of the display screen to be identified in the target image; a2, a QR code can also be pre-posted on the four corners of the display screen, and the QR code is used as a mark corresponding to the four corners of the display screen to be identified in the target image. It is not limited to setting clearly identifiable features for the display screen or the four corners of the display screen in the above manner.

[0059] When clearly identifiable features are set in advance for the four corners of the display screen, the virtual reality device can identify the preset mark in the target image after capturing the target image, and determine the second coordinate according to the position of the preset mark in the target image.

[0060] Among them, when a clearly identifiable feature is set by the above-mentioned method a1, the preset mark is a preset pattern. After the virtual reality device captures the target image, it can detect the preset pattern in the target image, and determine the position coordinates of the preset image in the target image as the second coordinates; when a clearly identifiable feature is set by the above-mentioned method a2, the preset mark is a QR code. After the virtual reality device captures the target image, it can detect the QR code in the target image, and determine the position coordinates of the QR code in the target image as the second coordinates.

[0061] Optionally, in the case where no obvious identifiable features are set in advance for the four corners of the display screen, the positions of the four corners of the display screen in the target image can be determined by a corner point detection algorithm to determine the second coordinate. Alternatively, the position of the display screen in the target image can be detected by a target recognition model, and then the positions of the four corners of the display screen in the target image can be determined to determine the second coordinate. This application does not limit the implementation method of obtaining the second coordinate.

[0062] A2. Determine a first projection transformation matrix corresponding to the virtual reality device according to the first coordinates and the second coordinates corresponding to the display screen.

[0063] The first projection transformation matrix is ​​used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system, and the first projection transformation matrix may include a rotation matrix R1 and a translation matrix T1.

[0064] In a feasible implementation, the SolvePnP function may be called to solve the projection matrix according to the first coordinates and the second coordinates corresponding to the display screen to obtain the first projection transformation matrix corresponding to the virtual reality device.

[0065] Optionally, based on other algorithms for solving PnP, the first projection transformation matrix corresponding to the virtual reality device can be obtained according to the first coordinates and the second coordinates corresponding to the display screen.

[0066] A3. According to the first projection transformation matrix corresponding to the virtual reality device, coordinate transformation is performed on the first coordinate corresponding to the display screen to obtain a third coordinate.

[0067] The third coordinate is used to represent the position of the display screen in the virtual world. Similar to the first coordinate and the second coordinate, the third coordinate includes the position coordinates of the vertices corresponding to the four corners of the display screen in the virtual world. The third coordinate may include (x 31 ,y 31 , z1), (x 32 ,y 32 , z2), (x 33 ,y 33 , z3), (x 34 ,y 34 , z4), (x3, y3, z) represents the coordinates in the virtual world coordinate system, (x 31 ,y 31 , z1) and the aforementioned (x 21 ,y 21 and (x 11 , x 11 , 0) corresponds to the same angle, (x 32 ,y 32 , z2) and the aforementioned (x 22 ,y 22 ) and (x 12 ,y 12 , 0) corresponds to the same angle, (x 33 ,y 33 , z3) and the aforementioned (x 23 ,y 23 ) and (x 13 , x 13 , 0) corresponds to the same angle, (x 34 ,y 34 , z4) and the aforementioned (x 24 ,y 24 ) and (x 14 , x14 , 0) correspond to the same angle.

[0068] Specifically, the first coordinate corresponding to the display screen can be transformed according to the first projection transformation matrix to obtain a fourth coordinate, which is a coordinate in the camera coordinate system, that is, P c =R1(P w1 -T1), P c is the fourth coordinate, P w1 is the first coordinate, P w1 Specifically, it can be the aforementioned (x 11 ,y 11 ,0)、(x 12 ,y 12 ,0)、(x 13 , x 13 ,0)、(x 14 ,y 14 , 0), (x1, y1, 0); then, according to the second projection transformation matrix corresponding to the virtual camera in the virtual reality scene, the fourth coordinate is transformed to obtain the third coordinate. The second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system, that is, P w2 =R2(P c -T2), P w2 is the third coordinate, P w1 For the aforementioned (x 11 ,y 11 , 0), P w2 For the aforementioned (x 31 ,y 31 , z1); P w1 For the aforementioned (x 11 ,y 11 , 0), P w2 For the aforementioned (x 31 ,y 31 , z1); P w1 For the aforementioned (x 12 ,y 12 , 0), P w2 For the aforementioned (x 32 ,y 32 , z2); P w1 For the aforementioned (x 13 ,y 13 , 0), P w2 For the aforementioned (x 33 ,y 33 , z3); ​​P w1 For the aforementioned (x 14 ,y 14 , 0), P w2 For the aforementioned (x 34 ,y34 , z4); R2 is the rotation matrix of the camera coordinate system relative to the virtual world coordinate system, and T2 is the translation matrix of the camera coordinate system relative to the virtual world coordinate system. The second projection transformation matrix can be obtained based on the position and posture of the virtual camera in the virtual reality scene. In this way, the conversion from the real world to the virtual world coordinates can be achieved.

[0069] A4. Create a virtual display screen in the virtual reality scene according to the third coordinate.

[0070] Specifically, the virtual reality device can create a virtual display screen in the virtual reality scene by creating a two-dimensional surface piece or a two-dimensional thin sheet in the virtual reality scene or placing a three-dimensional display screen model in the virtual reality scene. The virtual reality device can create a two-dimensional surface piece or a two-dimensional thin sheet or place a three-dimensional display screen model at a position corresponding to the third coordinate according to the third coordinate, thereby creating a virtual display screen having the same position and posture as the position and posture of the display screen.

[0071] The projection transformation matrix of the virtual reality device is determined according to the coordinates reflecting the position of the display screen in the real world and in the image, and the coordinates reflecting the display screen in the real world are transformed according to the projection transformation matrix to obtain the coordinates reflecting the position of the display screen in the virtual world, and a virtual display screen is created, so that a virtual display screen with the same position and posture as the position and posture of the display screen can be created in the virtual reality scene.

[0072] S203: Display the first picture on the virtual display screen.

[0073] Specifically, the virtual reality device can use the decoded picture obtained by decoding the first picture as the texture of the virtual display screen to display the first picture on the virtual display screen. Assume that the virtual reality scene currently displayed in the virtual reality device is as follows Figure 3 As shown in P2 in , assuming that the relative position relationship between the display screen and the virtual reality device is as follows Figure 3 As shown in S1 in FIG. 1 , the first picture can be displayed on the virtual display screen as follows: Figure 3 As shown in P3 in .

[0074] In some possible situations, when the first picture has a corresponding three-dimensional model, a target three-dimensional model corresponding to the first picture can also be obtained, and the target three-dimensional model can be displayed in the virtual reality scene.

[0075] The virtual reality device can receive the target 3D model sent by the display screen while receiving the first picture, and then insert the target 3D model into the virtual reality scene to display the target 3D model in the virtual reality scene. The virtual reality device can insert the target 3D model into the virtual display screen in the virtual reality scene to display the target 3D model on the virtual display screen. Assuming that the first picture is as follows Figure 1 As shown in P1 in FIG. 1 , the target three-dimensional model corresponding to the first picture is as follows Figure 4 As shown in P4 in FIG. 1 , the target three-dimensional model can be displayed on the virtual display screen as follows: Figure 4 Optionally, the virtual reality device may also insert the target three-dimensional model at a position other than the virtual display screen in the virtual reality scene, so as to display the target three-dimensional model at a position other than the virtual display screen.

[0076] By displaying a three-dimensional model corresponding to the picture on a virtual display screen, the picture can have a three-dimensional sense, enriching the display effect of the picture displayed on the display screen.

[0077] Furthermore, the virtual reality device can also respond to the model state transformation instruction corresponding to the target three-dimensional model, switch the state of the target three-dimensional model, and display the target three-dimensional model after the state is switched in the virtual reality scene. The model state transformation instruction can be triggered by the user and is used to instruct one or more operations of translation, rotation or scaling on the target three-dimensional model. When the model state transformation instruction is used to instruct the translation operation of the target three-dimensional model, the target three-dimensional model can be translated in the virtual reality scene and the translated target three-dimensional model can be displayed in response to the model state transformation instruction; when the model state transformation instruction is used to instruct the rotation operation of the target three-dimensional model, the target three-dimensional model can be rotated in the virtual reality scene and the rotated target three-dimensional model can be displayed in response to the model state transformation instruction. When the model state transformation instruction is used to instruct the scaling of the target three-dimensional model, the target three-dimensional model can be scaled in the virtual reality scene and the scaled target three-dimensional model can be displayed in response to the model state transformation instruction.

[0078] According to the model state conversion instruction, a three-dimensional model corresponding to the picture after the switching state is displayed on the virtual display screen, which can enrich the display effect of the picture displayed on the display screen.

[0079] exist Figure 2In the corresponding technical solution, after receiving the first picture sent by the display screen, the virtual reality device creates a virtual display screen in the virtual reality scene and then displays the first picture on the virtual display screen, thereby realizing the linkage display between the virtual reality device and the display screen and increasing the correlation between the display screen and the virtual reality device; and, since the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world, the virtual reality device is able to perceive the spatial position of the display screen, making the display of the picture on the display screen in the virtual reality device more reasonable, and can enhance the realism of the picture displayed in the virtual reality scene.

[0080] Optionally, in some possible situations, the images displayed on the display screen may also be switched in the virtual reality scene.

[0081] Above Figure 2 The corresponding method may further include: in response to a screen switching instruction, switching to display a next screen corresponding to the first screen on the virtual display screen.

[0082] The next screen belongs to the target screen set, which is a set of screens that can be displayed on the display screen. The screen switching instruction can be triggered by the user. For example, there are 5 screens that can be displayed on the display screen, namely screen 1 to screen 5, and the current screen displayed on the virtual display screen is screen 2, and the next screen of screen 2 is screen 3. When the screen switching instruction triggered by the user is obtained, screen 3 is displayed on the virtual display screen.

[0083] When receiving the first picture sent by the display screen, the virtual reality device can receive a target picture set sent by the display screen, wherein the target picture set is a collection of pictures that can be displayed by multiple display screens, and the target picture set has a sequence of picture display. When the user triggers the switching of pictures in the virtual display screen in the virtual reality device, the virtual reality device can obtain the next picture corresponding to the first picture in the target picture set according to the sequence of picture display, and then use the next picture as the texture of the virtual display screen, so as to switch and display the next picture corresponding to the first picture in the virtual display screen.

[0084] Optionally, when the user triggers the switching of the screen in the virtual display screen, the virtual reality device may also send a screen acquisition request to the display screen, requesting to obtain the next screen of the first screen, and then receive the first screen sent by the display screen in response to the screen acquisition request, and finally use the next screen as the texture of the virtual display screen, thereby switching the display screen to the next screen corresponding to the first screen.

[0085] Switching the images that can be displayed on the virtual display screen can complete the display and viewing of the images that can be displayed on the display screen without affecting the display of the display screen, so that the images seen using the virtual reality device can be different, and the operations on the images will not affect each other.

[0086] The method of the present application is introduced above, and the device of the present application is introduced below.

[0087] See also Figure 5 , Figure 5 Schematic diagram of the structure of a screen display device provided in an embodiment of the present application. Figure 5 As shown, the screen display device 30 includes:

[0088] A picture receiving module 301 is used to receive a first picture sent by the display screen, where the first picture is the picture currently displayed on the display screen;

[0089] A creation module 302 is used to create a virtual display screen in a virtual reality scene corresponding to the virtual reality device, wherein the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world;

[0090] The display module 303 is configured to display the first picture on the virtual display screen.

[0091] In a possible design, the creation module 302 is specifically used to: obtain a first coordinate and a second coordinate corresponding to the display screen, where the first coordinate is used to indicate the position of the display screen in the real world, and the second coordinate is used to indicate the position of the display screen in a target image, where the target image is an image of the display screen currently photographed by the virtual reality device; determine a first projection transformation matrix corresponding to the virtual reality device according to the first coordinate and the second coordinate, where the first projection transformation matrix is ​​used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system; perform coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a third coordinate, where the third coordinate is used to indicate the position of the display screen in the virtual world; and create the virtual display screen in the virtual reality scene according to the third coordinate.

[0092] In a possible design, the creation module 302 is specifically used to: perform coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a fourth coordinate, where the fourth coordinate is the coordinate in the camera coordinate system; perform coordinate transformation on the fourth coordinate according to the second projection transformation matrix corresponding to the virtual camera in the virtual reality scene to obtain the third coordinate, where the second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system.

[0093] In a possible design, the creation module 302 is specifically used to: identify a preset mark in the target image; and determine the second coordinate according to the position of the preset mark in the target image.

[0094] In one possible design, the above-mentioned display module 303 is also used to: in response to a screen switching instruction, switch to display the next screen corresponding to the first screen in the virtual display screen, the next screen belongs to a target screen set, and the target screen set is a set of screens that can be displayed in the display screen.

[0095] In a possible design, the display module 303 is further used to: obtain a target three-dimensional model corresponding to the first picture; and display the target three-dimensional model in the virtual reality scene.

[0096] In a possible design, the display module 303 is further used to: in response to a state change instruction acting on the target three-dimensional model, switch the state of the target three-dimensional model, and display the target three-dimensional model after the state is switched in the virtual reality scene.

[0097] In a possible design, the first picture is obtained by capturing and encoding a picture currently displayed on the display screen; the above-mentioned display module 303 is specifically used to: decode the first picture; and display the decoded picture on the virtual display screen.

[0098] In a possible design, the above-mentioned screen display device 30 also includes a connection establishment module 304, which is used to obtain the communication address of the display screen and establish a communication connection with the display screen based on the communication address of the display screen.

[0099] It should be noted that Figure 5 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0100] After receiving the first picture sent by the display screen, the above-mentioned device creates a virtual display screen in the virtual reality scene and then displays the first picture on the virtual display screen, thereby realizing the linkage display with the display screen in the virtual reality device and increasing the correlation between the display screen and the virtual reality device; and, because the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world, the virtual reality device can perceive the spatial position of the display screen, making the display of the picture in the display screen in the virtual reality device more reasonable, and can enhance the realism of the picture displayed in the virtual reality scene.

[0101] See also Figure 6 , Figure 64 is a structural diagram of a virtual reality device provided in an embodiment of the present application, wherein the virtual reality device 40 includes a processor 401, a memory 402, and a display panel 403. The memory 402 and the display panel 403 are connected to the processor 401, for example, through a bus.

[0102] The processor 401 is configured to support the virtual reality device 40 to perform the corresponding functions of the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0103] The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

[0104] The display panel 403 is used to display the virtual reality scene.

[0105] The processor 401 may call the program code to perform the following operations:

[0106] Receiving a first picture sent by the display screen, where the first picture is a picture currently displayed on the display screen;

[0107] Creating a virtual display screen in a virtual reality scene corresponding to the virtual reality device, wherein the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world;

[0108] The first screen is displayed on the virtual display screen.

[0109] Optionally, the virtual reality device may also include sensor components such as a camera, a gyroscope, and a magnetometer to support the virtual reality device in realizing virtual reality functions.

[0110] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0111] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0112] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A screen display method, characterized in that: A virtual reality device applied to a virtual-reality interconnection system, wherein the virtual-reality interconnection system comprises the virtual reality device and a display screen, and the method comprises: Receiving a first picture sent by the display screen, where the first picture is a picture currently displayed on the display screen; Creating a virtual display screen in a virtual reality scene corresponding to the virtual reality device, wherein the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world; The first screen is displayed on the virtual display screen.

2. The method according to claim 1, characterized in that The step of creating a virtual display screen in a virtual reality scene corresponding to the virtual reality device includes: Acquire a first coordinate and a second coordinate corresponding to the display screen, wherein the first coordinate is used to indicate the position of the display screen in the real world, and the second coordinate is used to indicate the position of the display screen in a target image, wherein the target image is an image obtained by the virtual reality device currently photographing the display screen; Determine a first projection transformation matrix corresponding to the virtual reality device according to the first coordinates and the second coordinates, where the first projection transformation matrix is ​​used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system; According to the first projection transformation matrix, the first coordinate is transformed to obtain a third coordinate, where the third coordinate is used to represent the position of the display screen in the virtual world; The virtual display screen is created in the virtual reality scene according to the third coordinate.

3. The method according to claim 2, characterized in that The step of performing coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a third coordinate includes: According to the first projection transformation matrix, the first coordinate is transformed to obtain a fourth coordinate, where the fourth coordinate is a coordinate in a camera coordinate system; The fourth coordinate is transformed according to a second projection transformation matrix corresponding to the virtual camera in the virtual reality scene to obtain the third coordinate, and the second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system.

4. The method according to claim 2, characterized in that: The obtaining of the second coordinate corresponding to the display screen includes: Identifying a preset marker in the target image; The second coordinate is determined according to the position of the preset mark in the target image.

5. The method according to claim 1, characterized in that The method further comprises: In response to the screen switching instruction, the next screen corresponding to the first screen is switched and displayed in the virtual display screen, the next screen belongs to a target screen set, and the target screen set is a set of screens displayable in the display screen.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a target three-dimensional model corresponding to the first picture; The target three-dimensional model is displayed in the virtual reality scene.

7. The method according to claim 6, characterized in that After displaying the target three-dimensional model in the virtual reality scene, the method further includes: In response to a state change instruction acting on the target three-dimensional model, the state of the target three-dimensional model is switched, and the target three-dimensional model after the state is switched is displayed in the virtual reality scene.

8. The method according to any one of claims 1 to 7, characterized in that: The first picture is obtained by capturing and encoding a picture currently displayed on the display screen by the display screen; The displaying the first picture on the virtual display screen includes: Decoding the first picture; The decoded picture is displayed on the virtual display screen.

9. The method according to any one of claims 1 to 7, characterized in that: Before receiving the first picture sent by the display screen, the method further includes: Acquire the communication address of the display screen, and establish a communication connection with the display screen based on the communication address of the display screen.

10. A screen display device, characterized in that: A virtual reality device used in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes the virtual reality device and a display screen, and the device includes: A picture receiving module, used for receiving a first picture sent by the display screen, where the first picture is the picture currently displayed on the display screen; A creation module, used to create a virtual display screen in a virtual reality scene corresponding to the virtual reality device, wherein the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world; A display module is used to display the first picture on the virtual display screen.

11. A virtual reality device, characterized in that: The virtual reality device comprises a memory, a processor and a display panel, wherein the memory and the display panel are connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the virtual reality device implements the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 9.