Display picture interaction method and device, equipment and storage medium

Abstract

The invention provides a display picture interaction method and device, equipment and a storage medium. The method comprises the steps that a display screen sends a first picture currently displayed by the display screen to virtual reality equipment; the virtual reality device obtains a first user operation acting on a first picture displayed in a virtual reality scene corresponding to the virtual reality device; the virtual reality equipment sends a first operation instruction corresponding to the first user operation to the display screen, the first operation instruction is used for indicating the display screen to respond to the first user operation, and the first picture is updated in the display screen; the display screen responds to the first user operation, the first picture is updated in the display screen, and a second picture is obtained; and the virtual reality equipment receives the second picture sent by the display screen, and updates the first picture into the second picture in the virtual reality scene. According to the technical scheme, the picture displayed in the display screen can be controlled and synchronized through the virtual reality equipment.

Description

Technical Field

[0001] The present application relates to the field of display, and in particular to a display screen interaction method, device, equipment and storage medium. Background Art

[0002] With the development of science and technology, teaching methods have become diversified. In addition to using the display screen in the classroom to demonstrate courseware and conduct traditional courseware teaching, teachers and students can also use virtual reality (VR) equipment to demonstrate immersive virtual reality scenes and conduct virtual reality teaching.

[0003] At present, courseware teaching and virtual reality teaching are two independent teaching methods. When conducting courseware teaching, the virtual reality device needs to be taken off, and when conducting virtual reality teaching, the virtual reality device needs to be worn. If courseware teaching and virtual reality teaching are to be overlapped in one class, the virtual reality device needs to be frequently taken off and put on, which is cumbersome to operate. Summary of the invention

[0004] The present application provides a display screen interaction method, device, equipment and storage medium to solve the technical problem of cumbersome operation caused by frequently taking off and putting on virtual reality devices when overlapping courseware teaching and virtual reality teaching.

[0005] In a first aspect, a display screen interaction method is provided, which is applied to a virtual reality device in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes the virtual reality device and a display screen, and the method includes:

[0006] Acquire a first user operation, where the first user operation is a user operation acting on a first picture displayed in a virtual reality scene corresponding to the virtual reality device, and the first picture is a picture currently displayed on the display screen;

[0007] Sending a first operation instruction corresponding to the first user operation to the display screen, wherein the first operation instruction is used to instruct the display screen to respond to the first user operation and update the first screen in the display screen;

[0008] Receive a second picture sent by the display screen, and update the first picture to the second picture in the virtual reality scene, where the second picture is a picture obtained by updating the first picture in the display screen in response to the first user operation.

[0009] In this technical solution, after obtaining a first user operation acting on a first picture displayed in a virtual reality scene, the virtual reality device sends a first operation instruction corresponding to the first user operation to a display screen, and the first operation instruction is used to instruct the display screen to respond to the first user operation, and update the first picture displayed on the display screen in the display screen, thereby realizing control of the picture displayed on the display screen through the virtual reality device; then, the second picture sent by the display screen is received, and the first picture is updated to the second picture in the virtual reality scene, thereby realizing real-time synchronization between the screen picture displayed in the virtual reality device and the picture displayed on the display screen; thereby, the user can realize synchronous display and control of the picture content displayed on the display screen through the virtual reality device, and after wearing the virtual reality device, the user can use the virtual reality device to overlap courseware teaching and virtual reality teaching, without repeatedly taking off and putting on the virtual reality device, which can improve the convenience of operation.

[0010] In combination with the first aspect, in a possible implementation, the method further includes: receiving a third picture sent by the display screen; updating the first picture to the third picture in the virtual reality scene, wherein the third picture is a picture obtained by updating the first picture in the display screen in response to a second user operation, wherein the second user operation is a user operation acting on the first picture displayed in the display screen. The picture in the virtual reality device is updated as the user operates in the display screen, and real-time synchronization of the screen picture in the virtual device and the picture in the display screen can be achieved.

[0011] In conjunction with the first aspect, in a possible implementation, the first picture is displayed on a virtual display screen in the virtual reality scene, and the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world. Displaying the picture displayed on the display screen on a virtual display screen having the same position and posture as the position and posture of the display screen can make the display of the picture on the display screen in the virtual reality device more reasonable and enhance the sense of reality.

[0012] In combination with the first aspect, in a possible implementation, the method further includes: obtaining a first coordinate and a second coordinate corresponding to the display screen, the first coordinate being used to indicate the position of the display screen in the real world, and the second coordinate being used to indicate the position of the display screen in a target image, the target image being an image obtained by the virtual reality device currently photographing the display screen; determining a first projection transformation matrix corresponding to the virtual reality device according to the first coordinate and the second coordinate, the first projection transformation matrix being used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system; performing coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a third coordinate, the third coordinate being used to indicate the position of the display screen in the virtual world; and creating the virtual display screen in the virtual reality scene according to the third coordinate. Determining the projection transformation matrix of the virtual reality device according to the coordinates reflecting the position of the display screen in the real world and the image, and transforming the coordinates reflecting the display screen in the real world according to the projection transformation matrix to obtain the coordinates reflecting the position of the display screen in the virtual world, and creating the virtual display screen, so that a virtual display screen having the same position and posture as the position and posture of the display screen can be created in the virtual reality scene.

[0013] In combination with the first aspect, in a possible implementation, the first coordinate is transformed according to the first projection transformation matrix to obtain a third coordinate, including: according to the first projection transformation matrix, the first coordinate is transformed to obtain a fourth coordinate, the fourth coordinate is a coordinate in a camera coordinate system; according to the second projection transformation matrix corresponding to the virtual camera in the virtual reality scene, the fourth coordinate is transformed to obtain the third coordinate, the second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system. By first converting the coordinates reflecting the real world position to the camera coordinate system, and then converting them to the virtual world coordinate system, the conversion from the real world to the virtual world coordinates can be achieved.

[0014] In combination with the first aspect, in a possible implementation, obtaining the second coordinates corresponding to the display screen includes: identifying a preset mark in the target image; and determining the second coordinates according to a position of the preset mark in the target image. Marking the display screen with the preset mark facilitates rapid determination of the coordinates of the display screen in the image.

[0015] In conjunction with the first aspect, in a possible implementation, the first picture is displayed on a virtual display screen in the virtual reality scene, and a virtual operation button corresponding to the virtual display screen is also displayed in the virtual reality scene, and the virtual operation button is used to control the picture displayed on the virtual display screen. Displaying a virtual operation button for controlling the picture displayed on the virtual display screen in the virtual reality scene can improve the convenience of operation.

[0016] In combination with the first aspect, in a possible implementation, the position of the virtual operation button in the virtual reality scene changes as the position of the virtual display screen in the virtual reality scene changes, or the position of the virtual operation button in the virtual reality scene changes as the position of the user in the virtual reality scene changes. The position of the virtual operation button in the virtual reality scene changes as the position of the virtual display screen in the virtual reality scene or the position of the user in the virtual reality scene changes, which can enhance the immersion of the virtual reality scene and improve the user experience.

[0017] In combination with the first aspect, in a possible implementation, the method further includes: when there is a three-dimensional object in the second picture, obtaining a three-dimensional model corresponding to the three-dimensional object, and displaying the three-dimensional model in the virtual reality device. When there is a three-dimensional object in the picture displayed on the display screen, displaying the three-dimensional model in the virtual reality device can enhance the sense of immersion.

[0018] In combination with the first aspect, in a possible implementation, before obtaining the first user operation, the method further includes: obtaining a communication address of the display screen, and establishing a communication connection with the display screen based on the communication address of the display screen. By obtaining the communication address of the display screen and establishing a communication connection with the display screen, subsequent data transmission between the virtual reality device and the display screen can be facilitated.

[0019] In a second aspect, another display screen interaction method is provided, which is applied to a display screen in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes a virtual reality device and the display screen, and the method includes:

[0020] Sending a first picture to the virtual reality device, where the first picture is a picture currently displayed on the display screen, and the first picture is used by the virtual reality device to be displayed in a virtual reality scene corresponding to the virtual reality device;

[0021] receiving a first operation instruction sent by the virtual reality device, where the first operation instruction is used to instruct the display screen to respond to a first user operation and update the first picture on the display screen, where the first user operation is a user operation acting on the first picture displayed in the virtual reality scene;

[0022] In response to the first user operation, updating the first screen on the display screen;

[0023] A second picture is sent to the virtual reality device, where the second picture is a picture obtained by updating the first picture, and the second picture is used by the virtual reality device to update the first picture in the virtual reality scene.

[0024] In this technical solution, the display screen sends the picture currently displayed on the display screen to the virtual reality device, so that the picture currently displayed on the display screen can be displayed in the virtual reality device; then, the first operation instruction sent by the virtual reality device after obtaining the first user operation acting on the first picture displayed in the virtual reality device is received, and the first operation instruction is used to instruct the display screen to respond to the first user operation and update the first picture displayed on the display screen; then the display screen responds to the first user operation, updates the first picture on the display screen, and sends the second picture obtained by updating the first picture to the virtual reality device, thereby realizing the control and synchronization between the pictures displayed on the display screen by the virtual reality device. After wearing the virtual reality device, the user can use the virtual reality device to overlap courseware teaching and virtual reality teaching, without repeatedly taking off and putting on the virtual reality device, which can improve the convenience of operation.

[0025] In conjunction with the second aspect, in a possible implementation, the method further includes: responding to a second user operation, updating the first screen in the display screen, the second user operation being a user operation acting on the first screen displayed in the display screen; and sending a third screen to the virtual reality device, the third screen being a screen obtained by updating the first screen, the third screen being used by the virtual reality device to update the first screen in the virtual reality scene. The screen in the virtual reality device is updated with the user's operation in the display screen, and real-time synchronization of the screen screen in the virtual device and the screen in the display screen can be achieved.

[0026] In conjunction with the second aspect, in a possible implementation, before sending the first picture to the virtual reality device, the method further includes: obtaining a communication address of the virtual reality device, and establishing a communication connection with the virtual reality device based on the communication address of the virtual reality device. By obtaining the communication address of the virtual reality device and establishing a communication connection with the virtual reality device, subsequent data transmission between the virtual reality device and the display screen can be facilitated.

[0027] In a third aspect, a display screen interaction device is provided, which is applied to a virtual reality device in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes the virtual reality device and a display screen, and the device includes:

[0028] an operation acquisition module, configured to acquire a first user operation, wherein the first user operation is a user operation acting on a first screen displayed in a virtual reality scene corresponding to the virtual reality device, and the first screen is a screen currently displayed on the display screen;

[0029] an instruction sending module, used for sending a first operation instruction corresponding to the first user operation to the display screen, wherein the first operation instruction is used for instructing the display screen to respond to the first user operation and update the first screen in the display screen;

[0030] The display module is used to receive a second picture sent by the display screen, and update the first picture to the second picture in the virtual reality scene, where the second picture is the picture obtained by updating the first picture in the display screen in response to the first user operation.

[0031] In a fourth aspect, another display screen interaction device is provided, which is applied to a display screen in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes a virtual reality device and the display screen, and the device includes:

[0032] A picture sending module, used for sending a first picture to the virtual reality device, where the first picture is the picture currently displayed on the display screen, and the first picture is used by the virtual reality device to display in the virtual reality scene corresponding to the virtual reality device;

[0033] an instruction receiving module, configured to receive a first operation instruction sent by the virtual reality device, wherein the first operation instruction is used to instruct the display screen to respond to a first user operation and update the first picture on the display screen, wherein the first user operation is a user operation acting on the first picture displayed in the virtual reality scene;

[0034] A picture processing module, configured to update the first picture on the display screen in response to the first user operation;

[0035] The picture sending module is also used to send a second picture to the virtual reality device, where the second picture is a picture obtained by updating the first picture, and the second picture is used by the virtual reality device to update the first picture in the virtual reality scene.

[0036] In a fifth aspect, a computer device is provided, comprising a memory, a display panel, and one or more processors, wherein the memory and the display panel are connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory. When the one or more processors execute the one or more computer programs, the computer device implements the display screen interaction method of the first aspect or the second aspect.

[0037] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the display screen interaction method of the first aspect or the second aspect.

[0038] The present application can achieve the following technical effects: users can use virtual reality devices to achieve synchronous display and control of the picture content displayed on the display screen. After putting on the virtual reality device, the user can use the virtual reality device to overlap courseware teaching and virtual reality teaching without repeatedly taking off and putting on the virtual reality device, which can improve the convenience of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the system architecture of a virtual-real interconnection system provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of a display screen interaction method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a screen interaction provided in an embodiment of the present application;

[0042] Figure 4 A flowchart of another display screen interaction method provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of another screen interaction provided in an embodiment of the present application;

[0044] Figure 6 is a schematic diagram of a virtual reality scene provided in an embodiment of the present application;

[0045] Figure 7 is a schematic diagram of a virtual operation button provided in an embodiment of the present application;

[0046] Figure 8 It is a structural schematic diagram of a display screen interaction device provided in an embodiment of the present application;

[0047] Fig. 9 is a structural schematic diagram of another display screen interaction device provided in an embodiment of the present application;

[0048] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0050] The technical solution of the present application is applicable to scenarios where display screens and virtual reality devices coexist. Scenarios where display screens and virtual devices coexist include but are not limited to teaching scenarios, game scenarios, meeting scenarios, and the like.

[0051] The display screen and the virtual reality device can establish a communication connection. In one technical solution, after the display screen and the virtual reality device establish a communication connection, the virtual reality device can project the picture displayed in the virtual reality device onto the display screen, so that users who are not wearing the virtual reality device can view the picture displayed in the virtual reality device through the display screen, so that both users wearing the virtual reality device and users not wearing the virtual reality device can see the picture displayed in the virtual reality device. This technical solution allows teachers to view the display screen in the virtual reality device on the student side through the display screen and then guide the students, thereby completing virtual reality teaching. If the teacher wants to conduct courseware teaching, although the teacher can directly use the display screen for courseware teaching, the students need to take off the virtual reality device and return to the courseware teaching. If the teacher overlaps courseware teaching and virtual reality teaching in one class, students need to frequently take off and put on the virtual reality device, which is cumbersome to operate.

[0052] In view of this, the present application proposes a new technical solution, in which the display screen can be synchronized in real time between the display screen and the virtual reality device, and the user can control the picture displayed on the display screen through the virtual reality device, that is, the user can use the virtual reality device to achieve synchronous display and control of the picture content displayed on the display screen. Therefore, after wearing the virtual reality device, teachers and students can use the virtual reality device to overlap courseware teaching and virtual reality teaching, without having to repeatedly take off and put on the virtual reality device, which can improve the convenience of operation.

[0053] The technical solution of the present application can be applied to a virtual reality interconnection system composed of a virtual reality device and a display screen. For ease of understanding, the virtual reality interconnection system of the present application is first introduced. Figure 1 , Figure 1 A schematic diagram of the system architecture of a virtual-real interconnection system provided in an embodiment of the present application. Figure 1 As shown, the virtual-reality interconnection system 10 includes a virtual reality device 101 and a display screen 102, wherein the display screen 102 is a display device for display, and the display screen 102 can specifically be a projection screen, an intelligent interactive display tablet, and the like. The virtual reality terminal 101 is a terminal with a virtual reality function, that is, the virtual reality terminal 101 has the ability to render and display a virtual reality scene. Specifically, the virtual reality terminal 101 can be a VR glasses, a VR helmet, and the like. The virtual reality terminal 101 can have a camera, and the camera is used to shoot the environment outside the virtual reality terminal 101 to obtain a picture of the external environment. In the present application, the camera is used to shoot the picture of the external environment where the display screen is located.

[0054] The virtual reality device 101 and the display screen 102 can establish a communication connection through device identification and device connection technology, thereby realizing the interconnection between the virtual device 101 and the display screen 102, building a virtual-reality interconnection system, and then various communication interactions can be performed between the virtual reality device 101 and the display screen 102. The purpose of device identification is to establish a temporary communication connection between the virtual reality device 101 and the display screen 102, and to complete the exchange and negotiation of various communication information (such as communication address, communication protocol, service type, port number, etc.) required for establishing a long-term communication connection based on the temporary communication connection; the purpose of device connection is to establish a communication connection required for communication interaction based on the communication information obtained through identification and negotiation.

[0055] Virtual-reality interconnection services may be respectively run in the virtual reality device 101 and the display screen 102. The virtual-reality interconnection service refers to a service that can be perceived and interacted with by the virtual reality device 101 and the display screen 102. The display screen 102 may serve as a service provider of the virtual-reality interconnection service, and the virtual reality device 101 may serve as a service user of the virtual-reality interconnection service.

[0056] In order to enable the virtual-reality interconnection service to operate normally between the virtual reality device 101 and the display screen 102, the virtual reality device 101 and the display screen 102 need to establish a communication connection to realize the interconnection between the virtual device 101 and the display screen 102, so as to maintain the virtual interconnection service based on the communication connection. The virtual reality device 101 and the display screen 102 can obtain the communication information (such as communication address, communication protocol, service type, port number, etc.) required by both parties for the virtual-reality interconnection service based on the device identification technology, and establish a communication connection capable of maintaining the virtual-reality interconnection service based on the acquired communication information.

[0057] The device identification between the virtual reality device 101 and the display screen 102 may include a device discovery process and a service discovery process. The device discovery process refers to the process of discovering a target device that can communicate with the surrounding area. The service discovery process refers to the process of requesting the target device to provide a service after discovering a target device that can communicate with the surrounding area. The service in this application refers to the service that the virtual reality device can perceive and interact with the interactive smart tablet (hereinafter referred to as the virtual-reality interconnection service). The device discovery process and service discovery process can be implemented between the virtual reality device 101 and the display screen 102 through one or more of the following device identification schemes:

[0058] (1) Based on wireless fidelity (WiFi) direct connection protocol and multicast domain name system (mDNS) protocol

[0059] (2) Based on the Bluetooth low energy (BLE) protocol and the generic attribute profile (GATT) protocol

[0060] (3) Based on near field communication (NFC) protocol and simple NDEF exchange protocol (SNEP)

[0061] (4) Based on universal plug and play (UPnP) protocol and simple service discovery protocol (SSDP)

[0062] (5) Based on multicast domain name service (Bonjour)

[0063] (6) Based on WiFi beacon frame

[0064] (7) The QR code carries the communication information of the display screen, and the virtual reality device scans the QR code.

[0065] After the device identification process, the virtual reality device 101 and the display screen 102 can establish a communication connection that can maintain the virtual-real interconnection service. The virtual reality device 101 and the display screen 102 can establish a communication connection based on WiFi communication technology or through Ethernet, which is not limited in this application.

[0066] based on Figure 1 The virtual reality network shown can implement the technical solution of the present application. The technical solution of the present application is described in detail below.

[0067] See also Figure 2 , Figure 2 A flowchart of a display screen interaction method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0068] S201, the virtual reality device and the display screen establish a communication connection.

[0069] Here, the virtual reality device and the display screen can obtain the communication addresses of both parties and establish a communication connection based on the communication addresses of both parties. Among them, the virtual reality device and the display screen can obtain the communication addresses of both parties through device identification technology. The virtual reality device and the display screen can complete the device discovery process and service discovery process through one or more device identification schemes introduced above, thereby obtaining the communication addresses of both parties and establishing a communication connection. For the specific method of establishing a communication connection between the virtual reality device and the display screen, please refer to the above introduction.

[0070] By obtaining the communication addresses of both parties and establishing a communication connection between the two parties, it is possible to facilitate subsequent data interactions between the virtual reality device and the display screen.

[0071] S202: The display screen sends a first picture to the virtual reality device, and the virtual reality device receives the first picture.

[0072] The first picture is the picture currently displayed on the display screen. For example, the first picture can be as follows: Figure 3 As shown in P1.

[0073] The display screen and virtual reality device in the virtual-reality interconnection system can transmit the picture in the display screen to the virtual reality device based on the video streaming method. The display screen can capture the picture currently displayed in the display screen based on the full-screen capture or window capture method to obtain a screenshot picture; then use any encoding method such as H264, H265, VP8 and VP9 to encode the screenshot picture to obtain a first picture; then send the first picture to the virtual reality device through the data stream transmission channel between the virtual reality device, and after the virtual reality device receives the first picture, it uses the decoding method corresponding to the encoding method used by the display screen to decode the received first picture to obtain a decoded picture to be displayed. Among them, the data stream transmission channel between the display screen and the virtual reality device can be established based on a preset stream transmission protocol, and the preset stream transmission protocol can be any one of the video streaming protocols including HTTP live streaming (HLS) protocol based on HTTP, dynamic adaptive streaming over HTTP (DASH) protocol based on HTTP, real time messaging protocol (RTMP) protocol, web real time communication (WebRTC) protocol, and HTTP smooth stream protocol based on HTTP.

[0074] S203: The virtual reality device displays a first picture in a virtual reality scene corresponding to the virtual reality device.

[0075] The virtual reality device may create a virtual display screen in the virtual reality scene corresponding to the virtual reality device, for displaying the first picture (i.e., the decoded picture), that is, the first picture is displayed in the virtual reality device on the virtual display screen in the virtual reality scene corresponding to the virtual reality device, and the virtual reality scene corresponding to the virtual reality device refers to the virtual three-dimensional scene currently displayed in the virtual reality device. The virtual display screen may be presented as a two-dimensional surface, a three-dimensional thin sheet, or a three-dimensional model in the virtual reality scene, and this application does not limit this.

[0076] For example, the virtual reality scene currently displayed in the virtual reality device is as follows: Figure 3 As shown in P2 in FIG. 1 , the virtual display screen created in the virtual display scene can be as follows: Figure 3 As shown in w in the figure, it exists in the virtual reality scene and is presented in the style of a display screen; suppose the picture displayed on the display screen is as follows Figure 3 As shown in P1 in FIG. 1 , the first picture can be displayed in the virtual reality scene as follows: Figure 3 As shown in P3 in FIG. 1 , the first picture is displayed on the virtual display screen w.

[0077] In some possible cases, the position and posture of the virtual display screen in the virtual reality scene can be the same as the position and posture of the real screen in the real world. In this way, the display of the picture on the display screen in the virtual reality device can be more reasonable and the realism of the picture can be enhanced.

[0078] Among them, the position and posture of the display screen in the real world refers to the position and posture of the display screen in the real world relative to the virtual reality device, that is, the position and posture of the display screen relative to the user wearing the virtual reality device. The position and posture of the virtual display screen in the virtual reality scene refers to the position and posture of the virtual display screen relative to the virtual camera in the virtual reality scene, that is, the position and posture of the virtual display screen relative to the user position in the virtual reality scene. For example, the display screen is set perpendicular to the ground in the real world, the display screen is located within the line of sight of the user wearing the virtual reality device, and the right front side of the user wearing the virtual reality device is in the direction of (0°, 0°, 60°) with the user's line of sight, then the virtual display screen is located in the right front side of the user position in the virtual reality scene in the direction of (0°, 0°, 60°). The shape of the virtual display screen can be the same as the display screen, or it can be different from the display screen.

[0079] In a feasible implementation, the relative position relationship between the display screen and the virtual reality device can be determined based on visual recognition, so as to create a virtual display screen in the virtual reality scene whose position and posture are the same as the position and posture of the display screen in the real world.

[0080] Specifically, a virtual display screen can be created through the following steps A1-A4:

[0081] A1. Obtain the first coordinate and the second coordinate corresponding to the display screen.

[0082] Here, the first coordinate is used to represent the position of the display screen in the real world. The display screen can be abstracted as a rectangle in the real world. The position coordinates of the four vertices of the rectangle can be used to represent the rectangle. Therefore, the position coordinates of the vertices corresponding to the four corners of the display screen can be used to represent the position of the display screen in the real world, that is, the first coordinate includes the position coordinates of the vertices corresponding to the four corners of the display screen. The first coordinate may include (x 11 ,y 11 ,0)、(x 12 ,y 12 ,0)、(x 13 , x 13 ,0)、(x 14 ,y 14 ,0),(x 1 ,y 1 , 0) represents the coordinates in the real-world coordinate system. The vertex corresponding to one of the corners of the display screen can be used as the coordinate origin of the real-world coordinate system, and the position coordinates of the vertices corresponding to the other three corners of the display screen in the real-world coordinate system can be determined according to the size of the display screen.

[0083] Specifically, the first coordinate of the display screen can be pre-calibrated and saved in the display screen. The virtual reality device can obtain the first coordinate corresponding to the display screen based on the communication connection with the display screen. Alternatively, the first coordinate of the display screen can also be pre-calibrated and saved in a QR code posted on the display screen, and the virtual reality device can obtain the first coordinate corresponding to the display screen when scanning the QR code. Alternatively, the first coordinate of the display screen can also be pre-calibrated and saved in a QR code displayed on the display screen, and the virtual reality device can obtain the first coordinate corresponding to the display screen when scanning the QR code to establish a communication connection. This application does not limit the implementation method of obtaining the first coordinate.

[0084] The second coordinate is used to indicate the position of the display screen in the target image. The target image is the image obtained by the virtual reality device currently shooting the display screen. The second coordinate is a pixel coordinate. The second coordinate changes with the relative position relationship between the virtual reality device and the display screen, that is, the second coordinate changes with the shooting angle when the virtual reality device shoots the display screen. Similar to the first coordinate, the second coordinate includes the position coordinates of the vertices corresponding to the four corners of the display screen in the image. The second coordinate may include (x 21 ,y 21 )、(x 22 ,y 22 )、(x 23 ,y 23 )、(x 24 ,y 24 ), where (x 2 ,y 2 ) represents the coordinates in the pixel coordinate system, (x 21 ,y 21 ) can be combined with the first coordinate (x 11 , x 11 , 0) corresponds to the same angle, (x 22 ,y 22 ) can be combined with the first coordinate (x 12 ,y 12 , 0) corresponds to the same angle, (x 23 ,y 23 ) can be combined with the first coordinate (x 13 ,y 13 , 0) corresponds to the same angle, (x 24 ,y 24 ) can be combined with the first coordinate (x 14 ,y 14 , 0) correspond to the same angle.

[0085] The virtual reality device can perform image recognition on the target image to identify the positions of the four corners of the display screen in the target image and obtain the second coordinates.

[0086] In some possible situations, clearly identifiable features can be set in advance for the four corners of the display screen, so that the positions of the four corners of the display screen in the image can be quickly and accurately identified. Among them, clearly identifiable features can be set in advance for the display screen or the four corners of the display screen in the following ways: a1, a special appearance can be set for the display screen, in which the four corners of the display screen are designed as clearly distinguishable preset patterns, and the preset patterns can be used as marks of the four corners of the display screen to be identified in the target image; a2, the display screen displays a preset picture on the display screen, and the four corners of the preset picture are clearly distinguishable preset patterns, and the preset image is used as a mark of the four corners of the display screen to be identified in the target image; a3, a QR code can also be pre-posted on the four corners of the display screen, and the QR code is used as a mark corresponding to the four corners of the display screen to be identified in the target image. It is not limited to setting clearly identifiable features for the four corners of the display screen in the above manner.

[0087] When clearly identifiable features are set in advance for the four corners of the display screen, the virtual reality device can identify the preset mark in the target image after capturing the target image, and determine the second coordinate according to the position of the preset mark in the target image.

[0088] Among them, when a clearly identifiable feature is set by the above-mentioned method a1 or a2, the preset mark is a preset pattern. After the virtual reality device captures the target image, it can detect the preset pattern in the target image, and determine the position coordinates of the preset image in the target image as the second coordinates; when a clearly identifiable feature is set by the above-mentioned method a3, the preset mark is a QR code. After the virtual reality device captures the target image, it can detect the QR code in the target image, and determine the position coordinates of the QR code in the target image as the second coordinates.

[0089] Optionally, in the case where no obvious identifiable features are set in advance for the four corners of the display screen, the positions of the four corners of the display screen in the target image can be determined by a corner point detection algorithm to determine the second coordinate. Alternatively, the position of the display screen in the target image can be detected by a target recognition model, and then the positions of the four corners of the display screen in the target image can be determined to determine the second coordinate. This application does not limit the implementation method of obtaining the second coordinate.

[0090] A2. Determine a first projection transformation matrix corresponding to the virtual reality device according to the first coordinates and the second coordinates corresponding to the display screen.

[0091] The first projection transformation matrix is ​​used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system. The first projection transformation matrix may include a rotation matrix R 1 and the translation matrix T 1 .

[0092] In a feasible implementation, the SolvePnP function may be called to solve the projection matrix according to the first coordinates and the second coordinates corresponding to the display screen to obtain the first projection transformation matrix corresponding to the virtual reality device.

[0093] Optionally, based on other algorithms for solving PnP, the first projection transformation matrix corresponding to the virtual reality device can be obtained according to the first coordinates and the second coordinates corresponding to the display screen.

[0094] A3. According to the first projection transformation matrix corresponding to the virtual reality device, coordinate transformation is performed on the first coordinate corresponding to the display screen to obtain a third coordinate.

[0095] The third coordinate is used to indicate the position of the display screen in the virtual world. Similar to the first coordinate and the second coordinate, the third coordinate includes the position coordinates of the vertices corresponding to the four corners of the display screen in the virtual world. The third coordinate may include (x 31 ,y 31 , z 1 )、(x 32 ,y 32 , z 2 )、(x 33 ,y 33 , z 3 )、(x 34 ,y 34 , z 4 ), (x 3 ,y 3 , z) represents the coordinates in the virtual world coordinate system, (x 31 ,y 31 , z 1 ) and the aforementioned (x 21 ,y 21 and (x 11 , x 11 , 0) corresponds to the same angle, (x 32 ,y 32 , z 2 ) and the aforementioned (x 22 ,y 22 ) and (x 12 ,y 12 , 0) corresponds to the same angle, (x 33 ,y 33 , z 3 ) and the aforementioned (x 23 ,y 23 ) and (x 13 , x 13 , 0) corresponds to the same angle, (x 34 ,y 34 , z 4) and the aforementioned (x 24 ,y 24 ) and (x 14 , x 14 , 0) correspond to the same angle.

[0096] Specifically, the first coordinate corresponding to the display screen can be transformed according to the first projection transformation matrix to obtain a fourth coordinate, which is a coordinate in the camera coordinate system, that is, P c =R 1 (P w1 -T 1 ), P c is the fourth coordinate, P w1 is the first coordinate, P w1 Specifically, it can be the aforementioned (x 11 ,y 11 ,0)、(x 12 ,y 12 ,0)、(x 13 , x 13 ,0)、(x 14 ,y 14 ,0),(x 1 ,y 1 , 0); then, according to the second projection transformation matrix corresponding to the virtual camera in the virtual reality scene, the fourth coordinate is transformed to obtain the third coordinate. The second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system, that is, P w2 =R 2 (P c -T 2 ), P w2 is the third coordinate, P w1 For the aforementioned (x 11 ,y 11 , 0), P w2 For the aforementioned (x 31 ,y 31 , z 1 );P w1 For the aforementioned (x 11 ,y 11 , 0), P w2 For the aforementioned (x 31 ,y 31 , z 1 );P w1 For the aforementioned (x 12 ,y 12 , 0), P w2 For the aforementioned (x 32 ,y 32 , z 2 );P w1 For the aforementioned (x 13,y 13 , 0), P w2 For the aforementioned (x 33 ,y 33 , z 3 );P w1 For the aforementioned (x 14 ,y 14 , 0), P w2 For the aforementioned (x 34 ,y 34 , z 4 );R 2 is the rotation matrix of the camera coordinate system relative to the virtual world coordinate system, T 2 is the translation matrix of the camera coordinate system relative to the virtual world coordinate system. The second projection transformation matrix can be obtained based on the position and posture of the virtual camera in the virtual reality scene. In this way, the conversion from the real world to the virtual world coordinates can be achieved.

[0097] A4. Create a virtual display screen in the virtual reality scene according to the third coordinate.

[0098] Specifically, the virtual reality device can create a virtual display screen in the virtual reality scene by creating a two-dimensional surface piece or a two-dimensional thin sheet in the virtual reality scene or placing a three-dimensional display screen model in the virtual reality scene. The virtual reality device can create a two-dimensional surface piece or a two-dimensional thin sheet or place a three-dimensional display screen model at a position corresponding to the third coordinate according to the third coordinate, thereby creating a virtual display screen having the same position and posture as the position and posture of the display screen.

[0099] The projection transformation matrix of the virtual reality device is determined according to the coordinates reflecting the position of the display screen in the real world and in the image, and the coordinates reflecting the display screen in the real world are transformed according to the projection transformation matrix to obtain the coordinates reflecting the position of the display screen in the virtual world, and a virtual display screen is created, so that a virtual display screen with the same position and posture as the position and posture of the display screen can be created in the virtual reality scene.

[0100] S204: The virtual reality device obtains a first user operation performed on the first screen.

[0101] Specifically, the virtual reality device can detect the virtual operation of the user on the virtual reality scene, and map the virtual operation of the user on the virtual reality scene to the first user operation on the first screen according to the preset operation mapping relationship. The preset operation mapping relationship refers to the mapping relationship between the virtual operation of the user on the virtual reality scene and the user operation on the screen. The virtual reality device can obtain the virtual operation of the user in the virtual reality scene by obtaining the user's hand movements or eye movement changes.

[0102] For example, the preset operation mapping relationship is shown in Table 1:

[0103] Virtual operations on virtual reality scenes User operations on the screen Ray click on the virtual left button in the VR scene Click the left mouse button Ray click virtual right button in VR scene Right click your mouse Ray click on a slide button in a VR scene Slide the mouse Ray movement Mouse movement … …

[0104] When the virtual reality device detects that the virtual operation performed by the user on the virtual reality scene is a ray click of a virtual left button in the virtual reality scene, it is determined that the first user operation performed on the first screen is a click of the left mouse button.

[0105] It should be noted that the above Table 1 is only an example of the present application and does not limit the present application.

[0106] S205: The virtual reality device sends a first operation instruction to the display screen, and the display screen receives the first operation instruction.

[0107] The first operation instruction is used to instruct the display screen to respond to the first user operation and update the first screen in the display screen. Updating the first screen in the display screen means switching the display screen in the display screen so that the screen currently displayed on the display screen is updated to the screen corresponding to the first user operation.

[0108] S206: The display screen responds to the first user operation and updates the first picture on the display screen to obtain a second picture.

[0109] Here, the second screen is the screen corresponding to the first user operation. Figure 3 As shown in P1 in FIG. 1 , the first user operation is to double-click the icon of "File 1" in P1, and the second screen is the screen after opening File 1. The display screen responds to the first user operation and updates the first screen in the display screen to obtain the second screen as shown in FIG. Figure 3 P4 in.

[0110] S207: The display screen sends the second picture to the virtual reality device, and the virtual reality device receives the second picture.

[0111] Here, the method of transmitting the second image between the display screen and the virtual reality device can refer to the method of transmitting the first image in the aforementioned step S202, and will not be repeated here.

[0112] S208: The virtual reality device displays a second picture in the virtual reality scene corresponding to the virtual reality device.

[0113] Here, the way in which the virtual reality device displays the second picture in the virtual reality device can refer to the implementation of the aforementioned step S203. Figure 3 As shown in P4 of FIG. 1 , the virtual reality device displays the second picture in the virtual reality device, which can be as follows Figure 3 As shown in P5.

[0114] exist Figure 2 In the corresponding technical solution, after obtaining the first user operation acting on the first screen displayed in the virtual reality device, the virtual reality device sends a first operation instruction corresponding to the first user operation to the display screen, and the first operation instruction is used to instruct the display screen to respond to the first user operation, and update the first screen displayed on the display screen in the display screen, so as to control the screen displayed on the display screen through the virtual reality device; then the second screen sent by the display screen is received, and the first screen is updated to the second screen in the virtual reality device, so as to realize real-time synchronization between the screen screen displayed in the virtual reality device and the screen displayed on the display screen; thereby, the user can realize control and synchronization of the screen content displayed on the display screen through the virtual reality device, and after wearing the virtual reality device, the user can use the virtual reality device to overlap the courseware teaching and virtual reality teaching, without repeatedly taking off and putting on the virtual reality device, which can improve the convenience of operation.

[0115] See also Figure 4 , Figure 4 A flowchart of another display screen interaction method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method comprises the following steps:

[0116] S301, the virtual reality device and the display screen establish a communication connection.

[0117] S302: The display screen sends a first picture to the virtual reality device, and the virtual reality device receives the first picture.

[0118] S303: The virtual reality device displays a first picture in a virtual reality scene corresponding to the virtual reality device.

[0119] Here, for the specific implementation of steps S301 to S303, reference may be made to the introduction of the aforementioned steps S201 to S203, which will not be repeated here.

[0120] S304: The display screen obtains a second user operation performed on the first screen.

[0121] The second user operation refers to a user operation performed by a user on the display screen to control the first screen. Specifically, the display screen can obtain the second user operation performed on the first screen through various operating peripherals (such as a touch pad, a touch screen, a mouse, a keyboard, a wireless presenter, etc.).

[0122] S305: The display screen responds to the second user operation and updates the first picture on the display screen to obtain a third picture.

[0123] Here, the third screen is the screen corresponding to the second user operation. Updating the first screen on the display screen to obtain the third screen means switching the first screen on the display screen so that the display screen displays the third screen. For example, the second user operation is double-clicking Figure 5 In the example, if the browser is opened in P1, the third screen is the screen after the browser is opened. The display screen responds to the second user operation and updates the first screen in the display screen. The third screen can be obtained as follows: Figure 5 P4 in.

[0124] S306: The display screen sends the third picture to the virtual reality device, and the virtual reality device receives the third picture.

[0125] Here, the method of transmitting the third picture between the display screen and the virtual reality device may refer to the method of transmitting the first picture in the aforementioned step S202.

[0126] S307: The virtual reality device displays a third picture in the virtual reality scene corresponding to the virtual reality device.

[0127] Here, the method of displaying the third picture in the virtual reality device can refer to the implementation method of the aforementioned step S203. Figure 5 As shown in P4 in FIG. 1 , the virtual reality device displays the third picture in the virtual reality device, which can be as shown in FIG. Figure 5 As shown in P5.

[0128] exist Figure 3 In the corresponding technical solution, after the virtual reality device establishes a communication connection with the display screen, the display screen sends the image of the virtual reality device itself to the virtual reality device, and the virtual reality device displays the image displayed on the display screen in the virtual reality scene. Moreover, the image in the virtual reality device is updated as the user operates on the display screen, thereby realizing real-time synchronization of the screen image in the virtual device and the image on the display screen.

[0129] In some possible situations, when there is a three-dimensional object in the picture displayed on the display screen (which may be the first picture, the second picture or the third picture mentioned above), the virtual reality device can also obtain a three-dimensional model corresponding to the three-dimensional object and display the three-dimensional model in the virtual reality device.

[0130] The virtual reality device can display the three-dimensional model while displaying the screen in the virtual reality device, and the three-dimensional model can be located in front of the display screen or on the side of the display screen in the virtual reality scene corresponding to the virtual reality device. The virtual reality device can obtain the three-dimensional model corresponding to the three-dimensional object from a preset three-dimensional model library or from a display screen.

[0131] For example, see Figure 6, assuming the screen displayed on the display is as follows Figure 6 As shown in P6 of FIG. 1 , since the picture shows a basketball, the virtual reality device can obtain a three-dimensional model corresponding to the basketball and display the three-dimensional model in the virtual reality device. The virtual reality device can display the three-dimensional model as follows: Figure 6 As shown in P7.

[0132] When there are three-dimensional objects in the picture displayed on the display screen, displaying the three-dimensional model in the virtual reality device can enhance the sense of immersion.

[0133] In some possible cases, a virtual operation button corresponding to the virtual display screen may also be displayed in the virtual reality scene, and the virtual operation button is used to control the picture displayed in the virtual display screen. Figure 7 Displaying virtual operation buttons for controlling the images displayed on the virtual display screen in the virtual reality scene can improve the convenience of operation.

[0134] The position of the virtual operation button in the virtual reality scene may change as the position of the virtual display screen in the virtual reality scene changes, or the position of the virtual operation button in the virtual reality scene changes as the position of the user in the virtual reality scene changes. For example, the virtual operation button may always be located on the side of the virtual display screen in the virtual reality scene, or located above the virtual display screen, etc.

[0135] The position of the virtual operation button in the virtual reality scene changes with the position of the virtual display screen in the virtual reality scene or the position of the user in the virtual reality scene, which can enhance the immersion of the virtual reality scene and improve the user experience.

[0136] The method of the present application is introduced above, and the device of the present application is introduced below.

[0137] See also Figure 8 , Figure 8 1 is a schematic diagram of a display screen interaction device provided in an embodiment of the present application. The display screen interaction device can be applied to a virtual reality device in a virtual-reality interconnection system. For example, the virtual reality device can be Figure 1 Virtual reality device 101. Figure 8 As shown, the display screen interaction device 40 includes:

[0138] An operation acquisition module 401 is used to acquire a first user operation, where the first user operation is a user operation acting on a first screen displayed in a virtual reality scene corresponding to the virtual reality device, where the first screen is a screen currently displayed on the display screen;

[0139] An instruction sending module 402 is used to send a first operation instruction corresponding to the first user operation to the display screen, wherein the first operation instruction is used to instruct the display screen to respond to the first user operation and update the first screen in the display screen;

[0140] The display module 403 is used to receive the second picture sent by the display screen, and update the first picture to the second picture in the virtual reality scene, where the second picture is the picture obtained by updating the first picture in the display screen in response to the first user operation.

[0141] In a possible design, the display module 403 is further used to: receive a third picture sent by the display screen; update the first picture to the third picture in the virtual reality scene, wherein the third picture is a picture obtained by updating the first picture in the display screen in response to a second user operation, and the second user operation is a user operation acting on the first picture displayed on the display screen.

[0142] In a possible design, the first picture is displayed on a virtual display screen in the virtual reality scene, and the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world.

[0143] In a possible design, the above-mentioned display screen interaction device 40 also includes a creation module 404, which is used to obtain a first coordinate and a second coordinate corresponding to the display screen, the first coordinate is used to indicate the position of the display screen in the real world, and the second coordinate is used to indicate the position of the display screen in the target image captured by the virtual reality device; according to the first coordinate and the second coordinate, a first projection transformation matrix corresponding to the virtual reality device is determined, and the first projection transformation matrix is ​​used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system; according to the first projection transformation matrix, the first coordinate is transformed to obtain a third coordinate, and the third coordinate is used to indicate the position of the display screen in the virtual world; according to the third coordinate, the virtual display screen is created in the virtual reality scene.

[0144] In a possible design, the creation module 404 is specifically used to: perform coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a fourth coordinate, where the fourth coordinate is the coordinate in the camera coordinate system; perform coordinate transformation on the fourth coordinate according to the second projection transformation matrix corresponding to the virtual camera in the virtual reality scene to obtain the third coordinate, where the second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system.

[0145] In a possible design, the creation module 404 is specifically used to: identify a preset mark in the target image; and determine the second coordinate according to the position of the preset mark in the target image.

[0146] In a possible design, the first picture is displayed on a virtual display screen in the virtual reality scene. A virtual operation button corresponding to the virtual display screen is also displayed in the virtual reality scene. The virtual operation button is used to control the picture displayed on the virtual display screen.

[0147] In one possible design, the position of the virtual operation button in the virtual reality scene changes as the position of the virtual display screen in the virtual reality scene changes, or the position of the virtual operation button in the virtual reality scene changes as the position of the user in the virtual reality scene changes.

[0148] In a possible design, the display module 403 is further used to: when there is a three-dimensional object in the second picture, obtain a three-dimensional model corresponding to the three-dimensional object, and display the three-dimensional model in the virtual reality device.

[0149] In a possible design, the above-mentioned display screen interaction device 40 also includes a connection establishment module 405, which is used to obtain the communication address of the display screen and establish a communication connection with the display screen based on the communication address of the display screen.

[0150] It should be noted that Figure 7 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0151] After acquiring a first user operation acting on a first picture displayed in a virtual reality scene, the above-mentioned device sends a first operation instruction corresponding to the first user operation to a display screen, wherein the first operation instruction is used to instruct the display screen to respond to the first user operation, update the first picture displayed in the display screen in the display screen, and control the picture displayed in the display screen through the virtual reality device; then, the second picture sent by the display screen is received, and the first picture is updated to the second picture in the virtual reality scene, thereby realizing real-time synchronization between the screen picture displayed in the virtual reality device and the picture displayed in the display screen; thereby, the user can realize synchronous display and control of the picture content displayed in the display screen through the virtual reality device, and after wearing the virtual reality device, the user can use the virtual reality device to overlap courseware teaching and virtual reality teaching, without repeatedly taking off and putting on the virtual reality device, which can improve the convenience of operation.

[0152] See also Fig. 9 , Fig. 91 is a schematic diagram of the structure of another display screen interaction device provided in an embodiment of the present application. The display screen interaction device can be applied to a display screen in a virtual-real interconnection system. The display screen can be, for example, the aforementioned Figure 1 The display screen 102 in FIG. Fig. 9 As shown, the display screen interaction device 50 includes:

[0153] A picture sending module 501 is used to send a first picture to the virtual reality device, where the first picture is the picture currently displayed on the display screen, and the first picture is used by the virtual reality device to display in the virtual reality scene corresponding to the virtual reality device;

[0154] An instruction receiving module 502 is used to receive a first operation instruction sent by the virtual reality device, where the first operation instruction is used to instruct the display screen to respond to a first user operation and update the first picture on the display screen, where the first user operation is a user operation acting on the first picture displayed in the virtual reality scene;

[0155] A picture processing module 503, configured to update the first picture on the display screen in response to the first user operation;

[0156] The picture sending module 501 is also used to send a second picture to the virtual reality device, where the second picture is a picture obtained by updating the first picture, and the second picture is used by the virtual reality device to update the first picture in the virtual reality scene.

[0157] In a possible design, the above-mentioned picture processing module 503 is also used to: respond to a second user operation to update the first picture in the display screen, and the second user operation is a user operation acting on the first picture displayed on the display screen; the above-mentioned picture sending module 501 is also used to: send a third picture to the virtual reality device, and the third picture is a picture obtained by updating the first picture, and the third picture is used by the virtual reality device to update the first picture in the virtual reality scene.

[0158] In a possible design, the above-mentioned display screen interaction device 50 also includes a connection establishment module 504, which is used to obtain the communication address of the virtual reality device and establish a communication connection with the virtual reality device based on the communication address of the virtual reality device.

[0159] It should be noted that Fig. 9 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0160] The above-mentioned device sends the picture currently displayed on the display screen to the virtual reality device, so that the picture currently displayed on the display screen can be displayed in the virtual reality device; then receives the first operation instruction sent by the virtual reality device after obtaining the first user operation acting on the first picture displayed on the virtual reality device, and the first operation instruction is used to instruct the display screen to respond to the first user operation and update the first picture displayed on the display screen; then the display screen responds to the first user operation, updates the first picture on the display screen, and sends the second picture obtained by updating the first picture to the virtual reality device, thereby realizing the control and synchronization between the pictures displayed on the display screen by the virtual reality device. After the user wears the virtual reality device, the virtual reality device can be used to overlap the courseware teaching and virtual reality teaching, without repeatedly taking off and putting on the virtual reality device, which can improve the convenience of operation.

[0161] See also Fig.10 , Fig.10 60 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, and the computer device may be a virtual reality device or a display screen in a virtual-real interconnection system. The computer device 60 includes a processor 601, a memory 602, and a display panel 603. The memory 602 and the display panel 603 are connected to the processor 601, for example, via a bus.

[0162] The processor 601 is configured to support the computer device 60 to perform the corresponding functions in the method in the above method embodiment. The processor 601 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0163] The memory 602 is used to store program codes, etc. The memory 602 may include a volatile memory (VM), such as a random access memory (RAM); the memory 602 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 602 may also include a combination of the above-mentioned types of memories.

[0164] The display panel 603 is used to display various screens.

[0165] When the computer device 60 is a virtual reality device in a virtual-reality interconnection system, the processor 601 may call the program code to perform the following operations:

[0166] Acquire a first user operation, where the first user operation is a user operation acting on a first picture displayed in a virtual reality scene corresponding to the virtual reality device, and the first picture is a picture currently displayed on the display screen;

[0167] Sending a first operation instruction corresponding to the first user operation to the display screen, wherein the first operation instruction is used to instruct the display screen to respond to the first user operation and update the first screen in the display screen;

[0168] Receive a second picture sent by the display screen, and update the first picture to the second picture in the virtual reality scene, where the second picture is a picture obtained by updating the first picture in the display screen in response to the first user operation.

[0169] When the computer device 60 is a display screen in a virtual-real interconnection system, the processor 601 may call the program code to perform the following operations:

[0170] Sending a first picture to the virtual reality device, where the first picture is a picture currently displayed on the display screen, and the first picture is used by the virtual reality device to be displayed in a virtual reality scene corresponding to the virtual reality device;

[0171] receiving a first operation instruction sent by the virtual reality device, where the first operation instruction is used to instruct the display screen to respond to a first user operation and update the first picture on the display screen, where the first user operation is a user operation acting on the first picture displayed in the virtual reality scene;

[0172] In response to the first user operation, updating the first screen on the display screen;

[0173] A second picture is sent to the virtual reality device, where the second picture is a picture obtained by updating the first picture, and the second picture is used by the virtual reality device to update the first picture in the virtual reality scene.

[0174] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0175] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0176] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A display screen interaction method, characterized in that: A virtual reality device applied to a virtual-reality interconnection system, wherein the virtual-reality interconnection system comprises the virtual reality device and a display screen, and the method comprises: Acquire a first user operation, where the first user operation is a user operation acting on a first picture displayed in a virtual reality scene corresponding to the virtual reality device, and the first picture is a picture currently displayed on the display screen; Sending a first operation instruction corresponding to the first user operation to the display screen, wherein the first operation instruction is used to instruct the display screen to respond to the first user operation and update the first screen in the display screen; Receive a second picture sent by the display screen, and update the first picture to the second picture in the virtual reality scene, where the second picture is a picture obtained by updating the first picture in the display screen in response to the first user operation.

2. The method according to claim 1, characterized in that: The method further comprises: Receiving a third picture sent by the display screen; In the virtual reality scene, the first picture is updated to the third picture, and the third picture is a picture obtained by updating the first picture in the display screen in response to the second user operation, and the second user operation is a user operation acting on the first picture displayed on the display screen.

3. The method according to claim 1, characterized in that The first picture is displayed on a virtual display screen in the virtual reality scene, and the position and posture of the virtual display screen in the virtual reality scene are the same as the position and posture of the display screen in the real world.

4. The method according to claim 3, characterized in that: The method further comprises: Acquire a first coordinate and a second coordinate corresponding to the display screen, wherein the first coordinate is used to indicate the position of the display screen in the real world, and the second coordinate is used to indicate the position of the display screen in a target image, wherein the target image is an image obtained by the virtual reality device currently photographing the display screen; Determine a first projection transformation matrix corresponding to the virtual reality device according to the first coordinates and the second coordinates, where the first projection transformation matrix is ​​used to reflect the conversion relationship between the real world coordinate system and the camera coordinate system; Performing coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a third coordinate, where the third coordinate is used to indicate a position of the display screen in the virtual world; The virtual display screen is created in the virtual reality scene according to the third coordinate.

5. The method according to claim 4, characterized in that The step of performing coordinate transformation on the first coordinate according to the first projection transformation matrix to obtain a third coordinate includes: According to the first projection transformation matrix, the first coordinate is transformed to obtain a fourth coordinate, where the fourth coordinate is a coordinate in a camera coordinate system; The fourth coordinate is transformed according to a second projection transformation matrix corresponding to the virtual camera in the virtual reality scene to obtain the third coordinate, and the second projection transformation matrix is ​​used to reflect the conversion relationship between the camera coordinate system and the virtual world coordinate system.

6. The method according to claim 4, characterized in that The obtaining of the second coordinate corresponding to the display screen includes: Identifying a preset marker in the target image; The second coordinate is determined according to the position of the preset mark in the target image.

7. The method according to any one of claims 1 to 6, characterized in that: The first picture is displayed on a virtual display screen in the virtual reality scene. A virtual operation button corresponding to the virtual display screen is also displayed in the virtual reality scene. The virtual operation button is used to control the picture displayed on the virtual display screen.

8. The method according to claim 7, characterized in that The position of the virtual operation button in the virtual reality scene changes as the position of the virtual display screen in the virtual reality scene changes, or the position of the virtual operation button in the virtual reality scene changes as the position of the user in the virtual reality scene changes.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When there is a three-dimensional object in the second picture, a three-dimensional model corresponding to the three-dimensional object is obtained, and the three-dimensional model is displayed in the virtual reality scene.

10. The method according to any one of claims 1 to 6, characterized in that: Before obtaining the first user operation, the method further includes: A communication address of the display screen is obtained, and a communication connection is established with the display screen based on the communication address of the display screen.

11. A display screen interaction method, characterized in that: A display screen applied to a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes a virtual reality device and the display screen, and the method includes: Sending a first picture to the virtual reality device, where the first picture is a picture currently displayed on the display screen, and the first picture is used by the virtual reality device to be displayed in a virtual reality scene corresponding to the virtual reality device; receiving a first operation instruction sent by the virtual reality device, where the first operation instruction is used to instruct the display screen to respond to a first user operation and update the first picture on the display screen, where the first user operation is a user operation acting on the first picture displayed in the virtual reality scene; In response to the first user operation, updating the first screen on the display screen; A second picture is sent to the virtual reality device, where the second picture is a picture obtained by updating the first picture, and the second picture is used by the virtual reality device to update the first picture in the virtual reality scene.

12. The method according to claim 11, characterized in that The method further comprises: In response to a second user operation, updating the first picture on the display screen, wherein the second user operation is a user operation acting on the first picture displayed on the display screen; A third picture is sent to the virtual reality device, where the third picture is a picture obtained by updating the first picture, and the third picture is used by the virtual reality device to update the first picture in the virtual reality scene.

13. The method according to claim 11 or 12, characterized in that: Before sending the first picture to the virtual reality device, the method further includes: Acquire the communication address of the virtual reality device, and establish a communication connection with the virtual reality device based on the communication address of the virtual reality device.

14. A display screen interaction device, characterized in that: A virtual reality device used in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes the virtual reality device and a display screen, and the device includes: an operation acquisition module, configured to acquire a first user operation, wherein the first user operation is a user operation acting on a first screen displayed in a virtual reality scene corresponding to the virtual reality device, and the first screen is a screen currently displayed on the display screen; an instruction sending module, used for sending a first operation instruction corresponding to the first user operation to the display screen, wherein the first operation instruction is used for instructing the display screen to respond to the first user operation and update the first screen in the display screen; The display module is used to receive a second picture sent by the display screen, and update the first picture to the second picture in the virtual reality scene, where the second picture is the picture obtained by updating the first picture in the display screen in response to the first user operation.

15. A display screen interaction device, characterized in that: A display screen used in a virtual-reality interconnection system, wherein the virtual-reality interconnection system includes a virtual reality device and the display screen, and the device includes: A picture sending module, used for sending a first picture to the virtual reality device, where the first picture is the picture currently displayed on the display screen, and the first picture is used by the virtual reality device to display in the virtual reality scene corresponding to the virtual reality device; an instruction receiving module, configured to receive a first operation instruction sent by the virtual reality device, wherein the first operation instruction is used to instruct the display screen to respond to a first user operation and update the first picture on the display screen, wherein the first user operation is a user operation acting on the first picture displayed in the virtual reality scene; A picture processing module, configured to update the first picture on the display screen in response to the first user operation; The picture sending module is also used to send a second picture to the virtual reality device, where the second picture is a picture obtained by updating the first picture, and the second picture is used by the virtual reality device to update the first picture in the virtual reality scene.

16. A computer device, characterized in that: The invention comprises a memory, a processor and a display panel, wherein the memory and the display panel are connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the computer device implements the method according to any one of claims 1 to 10 or any one of claims 11 to 13.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 10 or any one of claims 11 to 13.

Images