Operation data interaction method and system and storage medium

By calling callback functions and graphical rendering interfaces in the helmet display and PC display systems, data interaction and synchronization are achieved, screen synchronization problem is solved, VR gaming experience is improved, and the development process is simplified.

CN118981292AInactive Publication Date: 2024-11-19GUANGZHOU ZHUOYUAN VIRTUAL REALITY TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410849181.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, in the helmet display and PC display system, the picture of the PC display cannot be transmitted in real time and synchronized to the helmet display, resulting in players being unable to help players select content through the input device of the PC display in VR games.

Method used

By calling the callback function to copy the frame graphic data, read the interactive data in the preset data format, and display the data on the helmet display interface through the graphic rendering interface, realizing data interaction and synchronization between the PC display and the helmet display.

Benefits of technology

Real-time transmission of the PC monitor screen is realized to synchronize the helmet display, and supports players to trigger screen switching through VR controller operation, improving the gaming experience, saving development costs and simplifying the software architecture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981292A_ABST
    Figure CN118981292A_ABST
Patent Text Reader

Abstract

The invention relates to an operation data interaction method, which comprises the following steps of: calling a callback function, copying frame graphic data to obtain interaction data, reading the interaction data in a preset data format, rendering a page, and finally displaying the interaction data on a helmet-mounted display interface through a graphic rendering interface, so that the interaction data in the same system or the same windows platform can be displayed on the helmet-mounted display interface through the graphic rendering interface. And the data interaction between the PC display and the helmet-mounted display which are connected with the same host transmits and synchronizes the picture of the PC display to the helmet-mounted display in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of Web development technology, and in particular to an operation data interaction method, system and storage medium. Background Art

[0002] Helmet-mounted display, or HMD, is a 3DVR graphic display and observation device in virtual reality applications. It can be connected to a host separately to receive 3DVR graphic signals from the host. It is used in a head-mounted manner, and with the help of a three-degree-of-freedom spatial tracking locator, VR output effects can be observed. At the same time, the observer can move freely in space, such as walking and rotating freely, which gives a strong sense of immersion. At present, when players wear a head-mounted display, they can select content through the VR handle, and the selected content can be played and presented on the PC monitor for other people to see with the naked eye. However, other people cannot help players select content from the PC monitor through the input device of the PC monitor. In other words, the picture on the PC monitor cannot be transmitted and synchronized to the helmet display in real time. Summary of the invention

[0003] The present application provides an operation data interaction method, system and storage medium to solve the problem in the prior art that in a system composed of a helmet display, a PC display and a host, the image of the PC display cannot be transmitted and synchronized to the helmet display in real time.

[0004] In a first aspect, the present application provides an operation data interaction method, comprising:

[0005] By calling the callback function, the frame graphics data is copied to obtain the interactive data;

[0006] Read the interaction data in a preset data format and use it for page rendering;

[0007] The interactive data is displayed on a helmet display interface through a graphics rendering interface.

[0008] In a second aspect, the present application provides an operation data interaction system, comprising: a helmet display module, a main control module and a PC display module, wherein:

[0009] The PC display module is used to copy the frame graphic data to obtain the interactive data by calling the callback function;

[0010] The main control module is used to read the interactive data in a preset data format and use it for page rendering;

[0011] The helmet display module is used to display the interactive data on the helmet display interface through a graphics rendering interface.

[0012] In a third aspect, the present application also provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the operation data interaction method described in the present application.

[0013] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the present application calls the callback function, copies the frame graphic data to obtain the interactive data, then reads the interactive data in the preset data format and uses it for page rendering, and finally displays the interactive data on the helmet display interface through the graphics rendering interface, so as to realize the data interaction and data synchronization between the PC display and the helmet display located in the same system or the same Windows platform and connected to the same host, specifically: the screen of the PC display is transmitted and synchronized to the helmet display in real time, and further, based on the player's VR handle operation, the helmet display and the PC display are triggered to switch the screen to present the display screen corresponding to the VR handle operation. The present application adopts Web development technology, and only needs to develop a set of ordinary Web interface software at one time, and supports the display interface interaction between the helmet display and the PC display at the same time, which is different from the previous helmet display interface and PC display interface as two sets of software, and realizes the communication between the two sets of software, saving development costs and simplifying the software architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0017] Figure 1 A flowchart of an operation data interaction method provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a technical principle framework provided for an embodiment of the present application;

[0019] Figure 3 A flowchart of another operation data interaction method provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of an operation data interaction system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0023] In order to solve the technical problems in the prior art, the present application provides an operation data interaction method, which adopts Web development technology and only requires a set of ordinary Web interface software to be developed once, while supporting the display interface interaction between the helmet display and the PC display, and realizing the real-time transmission and synchronization of the PC display image to the helmet display.

[0024] Figure 1 An operation data interaction method provided in an embodiment of the present application includes:

[0025] S101, copying frame graphic data to obtain interactive data by calling a callback function;

[0026] Before understanding the implementation methods of the present application, the technology related to the present application is introduced first. Web technology refers to various tools and technologies used in the process of communication between different types of devices on the Internet. Web browsers are used to access web pages. Web browsers can be defined as programs that display text, data, pictures, animations, and videos on the Internet. Hyperlink resources on the World Wide Web can be accessed using software interfaces provided by Web browsers. Web development technology can generally be divided into two categories: client technology and server technology. The main task of a Web client is to display information content. The present application involves Web interface software development, which belongs to Web front-end technology, that is, client technology.

[0027] The implementation method of step S101 is:

[0028] First, the page drawing function calls the callback function, wherein the RGB pixel data of each frame is the input parameter of the callback function;

[0029] Secondly, the callback function copies the graphic data corresponding to each frame of RGB pixel data to the frame buffer, and the frame buffer stores the interaction data.

[0030] In one embodiment, the OnPaint function of the page drawing function of the PC display client calls the callback function, takes the RGB pixel data of each frame of the current display interface of the PC display as a parameter, and passes it into the callback function. The callback function copies the graphic data corresponding to the parameter to the frame buffer, that is, stores each frame of the PC display in the frame buffer. Through the page function call, the data to be interacted and displayed on another client is temporarily stored in the frame buffer.

[0031] The technical principle diagram of the embodiment of the present application is as follows Figure 2 As shown in the figure, CEF (Chromium Embedded Framework) is an embedded browser framework that supports HTML5 specifications. The upper layer uses js to implement interface interaction, and the lower layer uses C++ to implement the input devices of the PC display, such as keyboard and mouse event interaction, as well as device hardware communication, and communicates with js through websocket. CEF is also an open source Web browser control that supports Windows, Linux, and Mac platforms. In addition to providing C / C++ interfaces, there are also ported versions in other languages. The OPEN VR interface is a standard interface for PC VR that implements the content interface of VR (Virtual Reality, virtual reality technology), and the OVERLAY interface implements the display of the helmet display graphical interface. The DirectX library is a graphics library provided by the Windows platform, and screen rendering implements image output based on this library.

[0032] S102, reading the interaction data in a preset data format and using it for page rendering;

[0033] In one embodiment, the rendering thread reads the interactive data from the frame buffer to render the web page content. It can be understood that the rendering thread reads the two-dimensional image data in the D3D texture2d format from the frame buffer. The preset data format in this application is D3D texture2d.

[0034] S103: Displaying the interactive data on a helmet display interface through a graphics rendering interface.

[0035] In one embodiment, the helmet display client calls the OVERLAY sub-interface of the OPEN VR interface to display the screen content corresponding to the interactive data on the helmet display interface. That is, data interaction between the PC display client and the helmet display client is realized, and the display screen of the PC display client is displayed on the helmet display interface.

[0036] In the embodiment of the present application, by calling the callback function, the frame graphic data is copied to obtain the interactive data, and then the interactive data in the preset data format is read and used for page rendering. Finally, the interactive data is displayed on the helmet display interface through the graphics rendering interface, so as to realize the data interaction and data synchronization between the PC display and the helmet display located in the same system or the same Windows platform and connected to the same host, specifically: the picture of the PC display is transmitted and synchronized to the helmet display in real time. In this way, in the VR game scene, it is possible to support a third party to help the player select content to play from the PC display. Compared with the previous third party which can only provide language guidance and requires the player to operate the VR handle to select content to play through the description of the third party, the present application scheme is more convenient and brings a better gaming experience to the player.

[0037] Furthermore, in order to realize the operation interface through the VR controller, the PC display and the helmet display will switch the interface display and start the corresponding VR controller button content. It can be understood as: supporting players to watch content on demand through the helmet display and VR controller, such as: players can browse content, pay by themselves, play by themselves, etc., not limited to the PC display side. Figure 3 As shown, it also includes:

[0038] S104, calling the OPEN VR interface at a preset frequency to query whether there is a VR handle trigger event;

[0039] S105: If the query finds that there is, converting the VR handle trigger event into a corresponding PC display input device trigger event, wherein the PC display and the helmet display share a host and are located on the same platform or system;

[0040] In the embodiment of the present application, a timer is set with a preset frequency of 50fps. The timer calls the OPEN VR interface to query whether there is a VR handle trigger event. It can be understood that the timer detects whether a player triggers an operation button on the VR handle at a frequency of 50fps. Furthermore, if the input device of the PC monitor is a mouse, when it is detected that the player presses the operation button on the VR handle, a VR handle trigger event is generated. At this time, according to the different types of buttons on the VR handle, that is, different VR handle trigger events, they are converted into different mouse capture messages. The mouse capture message can be the mouse's WM_RBUTTONDOWN message, WM_RBUTTONUP message, and WM_MOUSEMOVE message.

[0041] It should be noted that the application scenario applicable to the embodiment of the present application is that the PC display and the helmet display share a host, and the Web development environment of the two display clients is the same platform or system, which is the Windows platform in the embodiment of the present application.

[0042] S106, obtaining VR handle position information, and converting the VR handle position information into corresponding interface position coordinate information of the PC display;

[0043] In one embodiment, S106 includes the following sub-steps:

[0044] S1061. Capturing movement information of the VR handle in real time, and calculating position information of the VR handle;

[0045] S1062, determining the intersection point between the front of the VR handle and the helmet display interface through the VR handle position information;

[0046] S1063: Map the position information of the intersection to X and Y coordinates of a preset window, wherein the preset window is an interface window of the PC display.

[0047] In an embodiment of the present application, the coordinates of the intersection of the ray formed by the orientation of the VR handle in the helmet display and the helmet display interface are first calculated. Specifically, the 3*4 rotation matrices of the VR handle and the helmet display are respectively obtained, and both are converted into 4*4 rotation matrices to obtain the rotation vector C. It should be noted that the 3*4 rotation matrix represents the posture and position of the VR handle and the helmet display, and the conversion into a 4*4 rotation matrix is ​​to facilitate subsequent calculations and processing. Exemplarily, A is the 4*4 rotation matrix of the VR handle, B is the 4*4 rotation matrix of the helmet display, and the rotation vector C = A*B. The VR handle position vector and the VR handle posture forward vector can be obtained through the rotation vector C.

[0048] For example, the VR handle position vector i = (C[3][0], C[3][1], C[3][2]), and the VR handle posture forward vector j = (C[2][0], C[2][1], C[2][2]).

[0049] Vector m is defined as the normal vector of the helmet display interface, which can be represented by (0, 0, -1). The intersection vector k can be obtained through vectors i, j and m. For example, the intersection vector k = i + (dot (ni, m) / dot (j, m)) * j, where n is a distance vector, which can be represented by (0, 0, h), h is the distance between the display interface of the helmet display and the person, and the preferred value is 4m, and dot is a dot product operation. According to the intersection vector, the intersection coordinates of the VR handle and the helmet display screen interface can be obtained. For example, the intersection coordinates l = (k [0], k [1]). It should be noted that the intersection coordinates are two-dimensional coordinate values, and the z-axis coordinate value can be the distance value h between the display interface of the helmet display and the person. It can also be understood here that the intersection coordinates are the intersection coordinates formed by the laser line emitted by the VR handle in the player's hand shooting at the display interface of the helmet display.

[0050] Afterwards, the intersection coordinates are directly mapped to the X and Y coordinates of the CEF window. Assuming that the size of the CEF window is (sx, sy), the X and Y coordinates of the CEF window are (l[0]*sx, l[1]*sy). Specifically, the preset window is the interface window of the PC display. Since this application is developed based on the CEF embedded software framework, the interface window of the PC display client, that is, the application window, is referred to as the CEF window here. It can be understood that the interface coordinate values ​​of the helmet display are converted into the interface coordinate values ​​of the PC display through the size of the CEF window, that is, the length and width values, to facilitate the subsequent function calls of the PC display client to trigger interface interaction operations.

[0051] S107, by calling a preset function, triggering the PC display interface and the helmet display interface to complete corresponding screen switching display based on the VR handle trigger event.

[0052] In one embodiment, a mouse message processing function is called, and the XY coordinate values ​​of the CEF window are used as input parameters of the mouse message processing function to trigger the PC display interface and the helmet display interface to complete the corresponding screen switching display.

[0053] In the embodiment of the present application, based on the player's VR handle operation, the helmet display and the PC display are triggered to switch the screen to present the display screen corresponding to the VR handle operation. In addition, players are supported to watch content on demand through the helmet display and VR handle, such as: players browse content, pay by themselves, play by themselves, etc., which improves the disadvantage of being limited to operating these services on the PC display side, and enhances the player's gaming experience.

[0054] like Figure 4 As shown, the embodiment of the present application provides an operation data interaction system, including: a helmet display module 41, a main control module 42 and a PC display module 43, wherein:

[0055] The PC display module 41 is used to copy the frame graphic data to obtain the interactive data by calling the callback function;

[0056] A main control module 42, used for reading the interactive data in a preset data format and using it for page rendering;

[0057] The helmet display module 43 is used to display the interactive data on the helmet display interface through a graphic rendering interface.

[0058] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the operation data interaction method provided in any of the aforementioned method embodiments are implemented.

[0059] The device / system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0061] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0062] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An operation data interaction method, characterized in that: The method comprises: By calling the callback function, the frame graphics data is copied to obtain the interactive data; Read the interaction data in a preset data format and use it for page rendering; The interactive data is displayed on a helmet display interface through a graphics rendering interface.

2. The method according to claim 1, characterized in that The method of copying the frame graphic data to obtain the interactive data by calling the callback function includes: The page drawing function calls the callback function, wherein the RGB pixel data of each frame is the input parameter of the callback function; The callback function copies the graphic data corresponding to each frame of RGB pixel data to a frame buffer, and the frame buffer stores the interaction data.

3. The method according to claim 1, characterized in that The step of reading the interaction data in a preset data format and using the data for page rendering includes: The rendering thread reads the interactive data from the frame buffer to render web page content, wherein the preset data format identifies two-dimensional image data of pixel points.

4. The method according to claim 1, characterized in that: Displaying the interactive data on a helmet display interface through a graphics rendering interface includes: The overlay sub-interface of the OPEN VR interface is called to display the screen content corresponding to the interactive data on the helmet display interface.

5. The method according to any one of claims 1 to 4, characterized in that: After displaying the interactive data on a helmet display interface through a graphics rendering interface, the method further includes: Call the OPEN VR interface at a preset frequency to check whether there is a VR controller trigger event; If the query finds that there is, converting the VR handle trigger event into a corresponding PC display input device trigger event, wherein the PC display and the helmet display share a host and are located on the same platform or system; Acquire VR handle position information, and convert the VR handle position information into corresponding interface position coordinate information of the PC display; By calling a preset function, the PC display interface and the helmet display interface are triggered to complete the corresponding screen switching display based on the VR handle trigger event.

6. The method according to claim 5, characterized in that The obtaining of VR handle position information and converting the VR handle position information into corresponding interface position coordinate information of the PC display includes: Capture the movement information of the VR handle in real time and calculate the position information of the VR handle; Determine the intersection point between the front of the VR handle and the helmet display interface through the VR handle position information; The position information of the intersection is mapped to the X and Y coordinates of a preset window, wherein the preset window is an interface window of the PC display.

7. The method according to claim 6, characterized in that The input device of the PC display is a mouse, and the converting of the VR handle trigger event into a corresponding PC display input device trigger event includes: According to different VR handle trigger events, convert them into different mouse capture messages; The method of calling a preset function to trigger the PC display interface and the helmet display interface to complete corresponding screen switching display based on the VR handle trigger event includes: The mouse message processing function is called to trigger the PC display interface and the helmet display interface to complete the corresponding screen switching display, wherein the X and Y coordinates of the preset window are the input parameters of the mouse message processing function.

8. An operation data interaction system, characterized in that: The system includes: a helmet display module, a main control module and a PC display module, wherein: The PC display module is used to copy the frame graphic data to obtain the interactive data by calling the callback function; The main control module is used to read the interactive data in a preset data format and use it for page rendering; The helmet display module is used to display the interactive data on the helmet display interface through a graphics rendering interface.

9. A storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Remote extension rendering system applied to mobile terminal virtual / augmented reality

    CN110751711A

  • Method and device for performing user interface display and interaction based on OGRE and medium

    CN114880056A

  • Pressure controlled kinetic feedback platform with modular attachments

    US20200097088A1

  • Method for estimating pose of handle, and virtual display device

    WO2024061238A1