Screen projection display method in virtual environment and VR equipment

Abstract

The invention relates to the technical field of VR, provides a projection screen display method in a virtual environment and VR equipment, and is used for improving the flexibility of multi-screen display. According to the method, on the basis of an SLAM space positioning technology and an Android system multi-window screen projection technology, a plurality of processes for controlling the number of screen projection windows are started by modifying kernel parameters of an Android system, and the screen projection window corresponding to each display device is positioned according to the position of the display device, so that dynamic creation and control of the screen projection windows are achieved, and in the screen projection process, the screen projection efficiency is improved. When the number of the display devices or the visual angle of the user changes, the position of the projection screen window corresponding to each display device can be repositioned, dynamic adjustment of the windows in the visual field range is achieved, and the optimal viewing effect of the user is guaranteed. And on the other hand, the at least one target window where the current user viewpoint is located is highlighted, so that the user can watch the screen projection content more conveniently, and the immersive experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and provides a screen projection display method and VR device in a virtual environment. Background Art

[0002] VR devices bring users an immersive experience and are now widely used in various industries, such as education and training, fire drills, virtual driving, real estate, etc. With the widespread application of VR devices, the immersive experience has been deeply loved by users. Users can use VR projection technology to project 2D images from traditional display devices (such as PCs, TVs, mobile phones, tablets, etc.) into the 3D virtual space of VR devices for playback.

[0003] Currently, based on the multi-window display function of the Android system, VR devices support the simultaneous projection of screen images of multiple display devices, allowing users to interact with multiple windows in a virtual environment and realize VR multi-screen interaction. Through VR screen projection, users can play games, watch movies, work, etc. in a fully immersive way.

[0004] However, the multi-window display of the Androdi system is implemented for installed applications, which is equivalent to that the multi-windows are defined in advance. In this way, during the screen projection process, when a user exits or a new user joins, the number of windows cannot be dynamically adjusted, and the user's perspective may change during the screen projection process. The defined multi-windows cannot meet the needs of dynamic adjustment in the virtual environment. Summary of the invention

[0005] The embodiments of the present application provide a screen projection display method and VR device in a virtual environment, which are used to improve the flexibility of screen projection from a 2D screen to a 3D space.

[0006] On the one hand, an embodiment of the present application provides a screen projection display method in a virtual environment, which is applied to a VR device, wherein the VR device establishes a screen projection connection with at least one display device, and the method includes:

[0007] Obtaining the initial number of display devices for screen projection, and modifying the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to a process, and the number of processes is used to control the number of screen projection windows;

[0008] Determine, according to the number of the projection windows and the position points of each display device, an initial position of the projection window corresponding to each display device in the display area of ​​the VR device;

[0009] When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is re-determined;

[0010] Calling at least one process to store the screen data streams sent by each display device in a memory, and performing texture drawing and display on the projection window corresponding to the target position of each display device according to each screen data stream in the memory;

[0011] The current user viewpoint is obtained, and at least one target window in each projection window where the current user viewpoint is located is enlarged and displayed.

[0012] On the other hand, an embodiment of the present application provides a VR device, including a processor, a memory, a display, and a communication interface, wherein the communication interface, the display, the memory, and the processor are connected via a bus;

[0013] The communication interface is used to establish a screen projection connection with at least one display device;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Obtaining the initial number of display devices for screen projection, and modifying the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to a process, and the number of processes is used to control the number of screen projection windows;

[0016] Determine, according to the number of the projection windows and the position points of each display device, an initial position of the projection window corresponding to each display device in the display area of ​​the VR device;

[0017] When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is re-determined;

[0018] Calling at least one process to store the screen data streams sent by each display device in a memory, and performing texture drawing on the projection window at the target position corresponding to each display device according to each screen data stream in the memory, and displaying it through the display;

[0019] The current user viewpoint is obtained, and at least one target window in each projection window where the current user viewpoint is located is enlarged and displayed.

[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a screen projection display method in a virtual environment provided in an embodiment of the present application.

[0021] The beneficial effects of a screen projection display method in a virtual environment provided by an embodiment of the present application are as follows:

[0022] Based on the Android system's support for multiple windows, when at least one display device simultaneously projects the screen to a VR device, the Android system's kernel parameters are modified according to the initial number of display devices, thereby starting multiple processes that control the number of projection windows, and positioning the projection window corresponding to each display device through SLAM spatial positioning technology, thereby achieving dynamic creation and control of the projection window and improving the flexibility of multi-screen display. In addition, during the projection process, when the number of display devices or the user's perspective changes, the target position of the projection window corresponding to each display device will be repositioned to achieve dynamic adjustment of the window within the field of view to ensure the user's best viewing effect. At the same time, at least one target window where the current user's viewpoint is located is highlighted so that the user can more conveniently view the projection content and enhance the user's immersive experience.

[0023] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of a traditional flat screen projection provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a VR device provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the corresponding relationship between the display device and the projection window provided in the embodiment of the present application;

[0028] Figure 4 A flow chart of a method for establishing a screen projection connection between a VR device and a display device provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of screen projection display logic of a VR device and a display device provided in an embodiment of the present application;

[0030] Figure 6 A flowchart of a screen projection display method in a virtual environment provided in an embodiment of the present application;

[0031] Figure 7 A flow chart of the projection window positioning method provided in an embodiment of the present application;

[0032] Fig. 8A and Figure 8B A schematic diagram of coordinate system conversion provided in an embodiment of the present application;

[0033] Fig. 9A A schematic diagram of the display area division of a VR device provided in an embodiment of the present application;

[0034] Fig. 9B and Fig. 9C A schematic diagram of a projection window corresponding to rays dividing a display area provided in an embodiment of the present application;

[0035] Fig.10 A flow chart of a method for determining a screen projection window corresponding to a display device provided in an embodiment of the present application;

[0036] FIG. 11A to FIG. 11D A schematic diagram of a process for determining a projection window corresponding to a display device provided in an embodiment of the present application;

[0037] Fig.12 A flowchart of rendering and displaying each projection window provided in an embodiment of the present application;

[0038] Fig.13 A system framework diagram for simultaneously rendering and displaying multiple projection windows provided in an embodiment of the present application;

[0039] Fig.14A and Fig. 14B A schematic diagram of highlighting the screen projection window provided in an embodiment of the present application;

[0040] Fig.15 A schematic diagram of one-to-one screen projection provided in an embodiment of the present application;

[0041] Fig.16 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0043] In real life, multiple users can interact in real time through screen projection on electronic devices. Taking the education and training industry as an example, teachers can use the screen projection content on students to timely understand the learning status and homework completion of students, and conduct comparative analysis on students' knowledge mastery.

[0044] Among them, the traditional flat screen projection is a projection method from 2D device to 2D device. Figure 1 As shown in the figure, multiple students project the contents on their tablets onto the teacher's smart TV screen, and each tablet Ci corresponds to a display window i on the smart TV screen. However, when multiple tablets are projected onto the smart TV at the same time, the more display windows there are, the smaller the display area occupied by each student's screen. The teacher cannot intuitively distinguish which display window corresponds to which student's screen content, and the position and screen of the smart TV are fixed, making it difficult to compare and control the students' projected screen content, resulting in a very unfriendly teaching experience.

[0045] Under limited conditions, when users want to get the experience of watching movies, reading, teaching, etc. on a large screen, traditional flat projection screens cannot meet user needs, and traditional flat projection screens lack immersion and interactivity.

[0046] With the rise of VR devices on the market, immersive interactive experience has gradually become known and recognized by people, and VR projection is being used more and more widely in education and training, games, entertainment and other fields.

[0047] VR devices usually include a head mounted display (HMD) and a handle. The HMD and the handle can be connected by wire or wirelessly, such as Figure 2 As shown, the HMD is equipped with a multi-camera, which can capture images of the environment and the light ring of the handle. Based on the images captured by the multi-camera, the HMD performs SLAM spatial positioning, calculates the 6DOf pose (i.e., position and posture) of the handle relative to the HMD, and the HMD can be used as a screen projection receiver to implement screen projection services. The handle has a small structural space and is mainly controlled by a microcontroller unit (MCU). Physical function keys are installed on the handle, which can control the screen displayed by the HMD according to the positioning results. In order to save power consumption, the handle controls the light ring to light up in a flashing manner through a reset circuit.

[0048] Based on VR devices, users can project 2D images from traditional display devices into 3D virtual space for playback, thereby obtaining an immersive interactive experience on a large screen. Starting from Android 7.0 (API level 24), the Android system introduced a multi-window mode that supports the simultaneous display of multiple application (Application, APP) windows. In multi-window mode, VR devices support the simultaneous projection of screen images from multiple display devices, and allow users to interact with multiple windows in a virtual environment, realizing VR multi-screen interaction.

[0049] Among them, the window modes of the Android system mainly include the following categories:

[0050] (1) Standard Mode

[0051] Standard mode is the default display mode of the Android system. Applications are displayed in full screen in this mode. That is, in standard mode, users can only see the content of one application, while other applications run in the background.

[0052] (2) Split-Screen Mode

[0053] Split-screen mode divides the screen into two parts, and two applications can be displayed at the same time. In split-screen mode, users can adjust the size of the application by dragging the edge of the application to achieve data interaction.

[0054] (3) Freeform Window Mode

[0055] Freeform window mode enables the simultaneous display of two or more applications, similar to the window mode in desktop operating systems. In freeform window mode, users can drag the application window to any position on the screen and adjust its size. This mode is suitable for tablets and large-screen devices.

[0056] However, the multi-window display of the Androdi system is implemented for installed applications, which is equivalent to that the multi-windows are defined in advance. For the windows displaying installed applications, users can freely drag and drop and interact with applications between two or more windows, but the number of windows cannot be adjusted in real time, and the user's perspective will change during the screen projection process. Therefore, the defined multi-windows cannot meet the needs of dynamic adjustment in the virtual environment.

[0057] For example, when the number of projection devices n is greater than 2, such as n=7, that is, 7 active windows can be displayed simultaneously in the virtual scene, the window size can be adjusted, and different windows can be dragged and dropped and interacted with. However, when two users leave or two new users join during the projection process, that is, n=5 or n=9, the VR device cannot reduce or increase windows to meet the multi-screen display requirements.

[0058] In view of this, an embodiment of the present application provides a screen projection display method in a virtual environment. The method is based on SLAM spatial positioning technology and Android system multi-window screen projection technology. When multiple display devices simultaneously project screens to a VR device, multiple processes for controlling the number of windows are started by modifying the kernel parameters in the VR device Android system, and the projection window corresponding to each display device is located, so that the display of the projection window can be dynamically controlled according to the number of display devices, and the interaction of multiple projection windows corresponding to multiple display devices in a virtual environment is realized, thereby enhancing the user's immersion and interactive experience. Moreover, during the projection process, when the number of display devices changes or the user's perspective changes, the position of the projection window corresponding to each display device will be relocated to ensure the best viewing effect for the user. On the other hand, the method highlights the window where the current user's viewpoint is located so that the user can more conveniently view the projection content.

[0059] During the screen projection process, the VR device can establish a screen projection connection with one or more display devices. Figure 3 As shown, the screen projection service (Server) in the HMD of the VR device will start multiple software clients (Client), each Client corresponds to a display device (Display Terminal, DT). When the DT starts the browser screen projection, the VR device uses SLAM spatial positioning technology to predict the position of each DT in order to display the projection window corresponding to each DT.

[0060] Among them, the VR device can display the device's connection method including wireless connection or wired connection.

[0061] Exemplarily, the VR device can establish a screen projection connection with the display device through cellular mobile communication technology, such as the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0062] Exemplarily, the VR device can also establish a screen projection connection with the display terminal through short-range wireless technology, such as Wireless Fidelity (Wi-Fi) technology.

[0063] Exemplarily, the VR device can also establish a screen projection connection by connecting to the same local area network as the display device.

[0064] Taking a display device as an example, the process of establishing a projection connection between a VR device and a display device can be found in Figure 4 , mainly includes the following steps:

[0065] S401: The VR device and the display device are connected to the same local area network.

[0066] S402: Enter the IP address and port number of the VR device on the display device.

[0067] S403: The display device opens a web page through a browser and executes the screen recording logic written in the web page.

[0068] S404: The display device sends a screen projection request to the VR device using the input IP address and port number according to the screen recording logic.

[0069] S405: The VR device applies for screen recording permission of the display device according to the screen projection request.

[0070] S406: After confirming that the VR device is granted the screen recording permission, the display device establishes a WebSocket connection with the VR device.

[0071] S407: After the WebSocket connection is established, the display device sends the recorded screen data stream to the VR device.

[0072] S408: The VR device displays the received screen data stream.

[0073] During the screen projection process, the HMD of the VR device acts as a server, running Http Server and WebSocket Server. Among them, Http Server is used to provide access to web page files, allowing browsers to access resources in the web page, and the screen recording logic is written in the web page; while WebSocket Server is used to receive the connection established by the browser and receive the recorded screen projection data stream. The web page runs in the browser and is used to record the screen projection data stream and send it to the HMD of the VR device. The HMD temporarily stores the received screen projection data stream in StreamBuffer. When the player plays the screen projection data, it reads the screen projection data stream from StreamBuffer for playback.

[0074] like Figure 5As shown, it is a schematic diagram of the screen projection display logic of the VR device and the display device. The user accesses the HTTP Server through the browser in the display device to open the web page, so that when operating the screen projection, the screen recording logic in the web page is executed to connect to the VR device through WebSocket and then send the recorded screen projection data stream to the WebSocket Server at the receiving end of the VR device, so that the VR device receives the screen projection data stream and temporarily stores the screen projection data stream in the StreamBuffer. When the VR device determines the screen projection window corresponding to the display device, it obtains the screen projection data stream from the StreamBuffer and plays the screen projection data stream on the corresponding window through the player of the VR device.

[0075] For example, during the screen projection process, the screen projection protocol used by the VR device and the display device can be browser screen projection or mirror screen projection based on a private protocol.

[0076] After the VR device establishes a screen projection connection with at least one display device, the VR device displays the screen projection content of at least one display device in the virtual space. Figure 6 , is a flow chart of a screen projection display method in a virtual environment provided in an embodiment of the present application. The process is executed by a VR device, specifically an HMD, and mainly includes the following steps:

[0077] S601: Obtain the initial number of display devices for screen projection, and modify the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to a process, and the number of processes is used to control the number of screen projection windows.

[0078] In actual applications, multiple display devices may be projected onto a VR device for display at the same time. In order to simultaneously display the projection contents of multiple display devices and realize interaction with different projection contents, the number of projection windows can be controlled according to the initial number of projection display devices.

[0079] At present, VR devices generally use the Android system. In one example, the VR device modifies the kernel parameters of the Android system to support multiple processes according to the initial number of display devices for projection. In this way, the VR device can call multiple processes to run multiple software clients at the same time, each software client is used to display the screen data stream sent by each display device, and the number of projection windows is controlled and created according to the number of processes, so that each display device corresponds to a projection window, so that the screen data streams sent by multiple devices can be displayed simultaneously.

[0080] For example, the modified kernel parameters include at least a min_free_kbytes parameter and an oom_adj parameter for controlling the number of processes.

[0081] S602: Determine the initial position of the projection window corresponding to each display device in the display area of ​​the VR device according to the number of projection windows and the position points of each display device.

[0082] In one example, after creating multiple projection windows, SLAM spatial positioning technology is used to determine the position of the projection window corresponding to each display device.

[0083] Considering that the optimal viewing distance from the screen window to the user is generally 3 to 5 meters, in order to ensure the user's viewing experience, an optimal viewing space area can be created in the virtual space according to the viewing position of the human eye as the display area of ​​each projection window, and then the position of each projection window is determined by the nearest grid area method according to the user's position (that is, the position of the HMD in the VR device). In specific implementation, the process of determining the position of the projection window can be found in Figure 7 , mainly includes the following steps:

[0084] S6021: Determine the position point of each display device according to the device images collected by multiple cameras on the VR device.

[0085] In actual applications, based on the device images captured by multiple cameras on the VR device, the SLAM spatial positioning technology is used to calculate the map position coordinates of each display device in the map coordinate system, and then each map position coordinate is converted into the projection position coordinates of the display device in the projection display coordinate system of the VR device to obtain the position point of each display device.

[0086] In one embodiment, the map position coordinates of the display device can be converted into the projection position coordinates in the following two ways:

[0087] Method 1: If the coordinate system parameters of the map coordinate system are the same as the coordinate system parameters of the projection display coordinate system, the map position coordinates are reduced by a specified multiple to obtain the projection position coordinates of the display device, where the coordinate system parameters include the origin position, the position and direction of the horizontal axis, the position and direction of the vertical axis, and the position and direction of the vertical axis.

[0088] In the embodiment of the present application, both the map coordinate system and the projection coordinate system are spatial rectangular coordinate systems established with the center point of the VR device (specifically HMD) as the origin, and the coordinate system parameters of the map coordinate system and the projection coordinate system are pre-set. Fig. 8A As shown, the map coordinate system and the projection display coordinate system both take the center point of the HMD as the origin, and the position and direction of each coordinate axis are the same.

[0089] It should be noted that the specified multiple in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specified multiple.

[0090] Method 2: If the coordinate system parameters of the map coordinate system are different from the coordinate system parameters of the projection display coordinate system, the map position coordinates will be converted using the pre-set rotation and translation matrix between the map coordinate system and the projection display coordinate system to obtain the projection position coordinates of the display device.

[0091] like Figure 8B As shown, the directions of the x-axis and y-axis of the map coordinate system and the projection display coordinate system are not the same. It can be seen from the figure that the position of p(x, y, z) in the map coordinate system converted to the projection display coordinate system is (x1, y1, z1).

[0092] After determining the position point of each display device in the projection display coordinate system, establish the optimal viewing space area in the projection display coordinate system.

[0093] S6022: Taking the current position of the VR device as the view origin, an initial ray is emitted directly in front of it.

[0094] In one example, the current position of the HMD of the VR device is taken as the perspective origin of the best viewing space area, a horizontal ray is emitted directly in front of it, and then the range of the best viewing space area is determined by the horizontal ray.

[0095] S6023: Taking the initial ray as a reference, determining a spatial region corresponding to a left and right viewing angle of the initial ray being a first specified angle and an upper and lower viewing angle being a second specified angle as a display region.

[0096] For example, the range of the first specified angle may be (0°, 160°), and the range of the second specified angle may be (0°, 90°). The specific angle values ​​of the first specified angle and the second specified angle may be set according to actual conditions, and the embodiments of the present application do not limit the specific angle values ​​of the first specified angle and the second specified angle.

[0097] S6024: Divide the display area equally according to the number of display devices.

[0098] In one example, taking the current position of the HMD as the origin, the same number of rays are emitted in the display area according to the number of display devices, wherein the rays are in the same horizontal plane and the angles between any two adjacent rays are the same.

[0099] For example, the angle between any two adjacent rays is obtained by dividing the first specified angle by the number of display devices. Fig. 9AAs shown, assuming that the number of display devices is 5 and the angle between any two adjacent rays is θ, the rays emitted in the display area.

[0100] S6025: For each display device, determine the position of the projection window corresponding to the display device based on a first distance from the display device to the current position of the VR device and a second distance from the position point of the display device to the ray dividing the display area.

[0101] In actual applications, when the nearest grid method is used to determine the position of the projection window corresponding to each display device, the display type and position of the projection window corresponding to each ray emitted in the display area can be determined based on the first distance from the display device to the current position of the VR device. Then, based on the second distance from the position point of each display device to the divided rays, the projection window corresponding to the ray closest to the display device can be used as the projection window corresponding to the display device.

[0102] In the embodiment of the present application, the display types of the projection window include a single-layer display type and a multi-layer display type, and the number of projection windows corresponding to any ray of the single-layer display type is one, and the number of projection windows corresponding to any ray of the display type is greater than one.

[0103] In one embodiment, the display type and position of the projection window corresponding to each ray may be determined in the following two ways:

[0104] Method 1: If the number of display devices is less than the specified number, the display type is determined to be a single-layer display type, and for any ray, the position of the projection window corresponding to the ray is obtained according to the first distance.

[0105] In one embodiment, the position of the projection window corresponding to any ray can be obtained in the following manner: for any ray, the position of the target point whose length from the endpoint of the ray is a first distance is determined as the position of the projection window corresponding to the ray.

[0106] like Fig. 9B As shown, taking the first distance r as an example, the position of the projection window corresponding to ray 1 is the position of point A, the position of the projection window corresponding to ray 2 is the position corresponding to point B, the position of the projection window corresponding to ray 3 is the position of point C, the position of the projection window corresponding to ray 4 is the position of point D, and the position of the projection window corresponding to ray 5 is the position of point E.

[0107] Method 2: If the number of display devices is not less than the specified number, the display type is determined to be a multi-layer display type, and for any ray, the respective positions of the multiple projection windows corresponding to the ray are determined according to the first distance.

[0108] In one embodiment, the positions of the multiple projection windows corresponding to each ray may be determined in the following manner:

[0109] For any ray, the target total number of projection windows corresponding to any ray is obtained according to the number of display devices and the number of each ray. After the position of the target point whose length from the endpoint of the ray is a first distance is determined as the position of a projection window corresponding to the ray, the current number of projection windows of the ray is updated. If the current number of projection windows of the ray is less than the target total number after the update, the horizontal position coordinate and the vertical position coordinate of a projection window are determined with the first distance from the endpoint of the ray being a target multiple, and the vertical position coordinate of the last determined projection window is expanded by a fixed multiple to obtain the position of a projection window, and then the step of updating the current number of projection windows corresponding to the ray is returned until the current number of projection windows of the ray is not less than the target total number, wherein the target multiple is obtained based on the current number of projection windows of the ray.

[0110] In the embodiment of the present application, the target multiple is obtained by adding 1 to the projection window of the ray, but the embodiment of the present application does not limit the target multiple here, and the target multiple in the embodiment of the present application can be set according to the actual situation. And in the embodiment of the present application, the number of display devices is divided by the number of each ray to obtain the target total number of projection windows corresponding to any ray. If the value obtained by dividing the number of display devices by the number of each ray is not an integer, the integer part of the value plus 1 is determined as the target total number of projection windows corresponding to any ray.

[0111] For example, if the total number of targets of the projection window corresponding to any ray is 2, Fig. 9C As shown, each ray corresponds to two projection windows.

[0112] In the above two methods, the first distance is obtained based on the projection position coordinates of each display device and the current position of the HMD of the VR device. Specifically, the first distance can be determined in the following manner: for any display device, the sub-distance between the display device and the VR device is obtained according to the map position coordinates of the display device and the current position of the HMD; the sub-distance is multiplied by a pre-set weight to obtain the corresponding middle length of the display device; the average value of the middle lengths of each display device is determined as the first distance.

[0113] After determining the position of the projection window corresponding to each ray, for any display device, the projection window corresponding to the display device is determined by using the second distance from the position point of the display device in the projection display coordinate system to each ray. Fig.10 , mainly includes the following steps:

[0114] S6025_1: For any display device, determine the second distance between the terminal device and each ray according to the position point of the display device.

[0115] In one embodiment, for any ray, the equation of the straight line corresponding to the ray is obtained based on the position and origin of the projection window corresponding to the ray, and the second distance from the terminal device to the ray is obtained by using the distance equation from the point to the straight line according to the projection position coordinates of the display device.

[0116] S6025_2: Determine the ray corresponding to the shortest second distance as the target ray corresponding to the display device.

[0117] like Fig.11A As shown, assuming that the distance between display device a and ray 1 is the shortest, ray 1 is used as the target ray corresponding to display device a.

[0118] S6025_3: According to the target rays corresponding to each display device, obtain the target display device corresponding to each ray.

[0119] In one embodiment, for any ray, a traversal of each display device is performed. For any traversed display device, if the ray is the target ray of the display device, the display terminal device is determined as the target terminal device corresponding to the ray, and the step of traversing each display device is returned until there is no untraversed display device.

[0120] S6025_4: For any ray, if the number of target display devices corresponding to the ray is one, and the number of projection windows corresponding to the ray is one, the projection window corresponding to the ray is determined as the projection window corresponding to the target display device.

[0121] like Fig. 11B As shown, the display device and the VR device are one-to-one screen projection. In the embodiment of the present application, the size of the projection window is pre-set, and the size of the projection window can be set according to the actual situation. The embodiment of the present application does not limit the size of the projection window.

[0122] S6025_5: For any ray, if the number of target display devices corresponding to the ray is one, and the number of projection windows corresponding to the ray is multiple, then the projection window that is closest to the target display device among the projection windows corresponding to the ray is determined as the projection window corresponding to the target display device.

[0123] like Fig. 11CAs shown, the target display device corresponding to ray 5 is display device m, and the projection windows corresponding to ray 5 are projection window P and projection window Q. As can be seen from the figure, projection window P is the projection window closest to display device m, so projection window P is determined to be the projection window corresponding to display device m.

[0124] S6025_6: For any ray, if the number of target display devices corresponding to the ray is multiple, traverse the projection windows corresponding to the ray in ascending order of distance from the HMD.

[0125] S6025_7: For any traversed projection window, determine the projection window as the projection window of the target display device that is closest to the projection window.

[0126] S6025_8: Determine whether the target display devices corresponding to the ray have all determined the corresponding projection windows. If so, end; if not, return to execute S6025_6.

[0127] like Fig.11D As shown, the target display devices corresponding to ray 4 are display device a and display device b, and the projection windows corresponding to ray 4 are projection window R and projection window H. As can be seen from the figure, projection window R is the projection window closest to display device a, so projection window R is determined to be the projection window corresponding to display device a, and projection window H is the projection window closest to terminal device b, so projection window H is determined to be the projection window corresponding to display device b.

[0128] In one example, after determining the initial position of the projection window corresponding to each display device in the display area of ​​the VR device in the above manner, since old users may leave or new users may join during the projection process, the number of display devices connected to the VR device will change, and the viewing angle of the user wearing the HMD may change during the projection process. Therefore, each projection window needs to be dynamically adjusted.

[0129] S603: When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is re-determined.

[0130] In one example, when the number of display devices changes, the projection windows displayed in the virtual environment are adjusted in real time and dynamically by controlling the number of processes. In specific implementation, kernel parameters are modified to update the number of processes according to the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes, and the display area is redivided according to the target number, and the target position of the projection window corresponding to each display device is determined according to the result of the redivision.

[0131] For example, when performing multi-screen projection display (i.e., the number of display devices n is greater than 2), assuming that the initial number of display devices is 7, during the projection process, (1) if two new users join, at this time, the target number of display devices is 9, then the parameters such as min_free_kbytes and oom_adj in the Android system are modified to add two processes, thereby adding a projection window for each of the two newly added display devices, that is, the number of projection windows is 9; (2) if two old users exit, at this time, the target number of display devices is 5, then the parameters such as min_free_kbytes and oom_adj in the Android system are modified to reduce two processes, thereby reducing two projection windows for displaying the screen data stream sent by the exiting display devices, that is, the number of projection windows is 5.

[0132] When the projection windows are increased or decreased according to the target number of display devices, in order to coordinate the display of each projection window and avoid disordered arrangement of the projection windows, the position of each window in the display area can be updated to determine the target position of each window.

[0133] In one example, when the positions of the windows in the display area are re-determined according to the number of the adjusted projection windows, it can be regarded as a rearrangement of the projection windows. Therefore, the process of determining the target position of each projection window is similar to the process of determining the initial position. Figures 7 to 11D The process will not be repeated here.

[0134] In one example, when the user's viewing angle changes, the original optimal viewing space area is no longer applicable, so the new display area of ​​each projection window needs to be re-determined. In specific implementation, the posture difference of the viewing angle origin before and after the change is obtained, and based on the posture difference, the target position of the projection window corresponding to each display device in the new display area is determined.

[0135] For example, the pose difference only calculates the change in the horizontal direction and ignores the change in the vertical direction.

[0136] For example, when only the user's viewing angle changes and the number of display devices does not change, the number of projection windows remains unchanged. At this time, the position of each projection window can be displaced only according to the difference in posture.

[0137] In an embodiment of the present application, during the screen projection process, multiple screen projection windows are calibrated once within the field of view according to the number of display devices and changes in the user's viewing angle, thereby realizing changes in the windows in the field of view and ensuring that what the user sees is what he gets.

[0138] After calibrating each projection window, the projection contents of multiple display devices are rendered and displayed in a virtual environment according to the target position of each projection window.

[0139] S604: Call at least one process to store the screen data streams sent by each display device in the memory, and perform texture drawing and display on the projection window of the corresponding target position of each display device according to each screen data stream in the memory.

[0140] In one example, each screen data stream recorded by a display device corresponds to a SurfaceFlinger, each SurfaceFlinger contains a Surface, and the display of each screen data stream serves as a software client in the VR device screen projection service, and corresponds to a Texture framework for texture drawing in the application layer.

[0141] See also Fig.12 , which is the rendering and display process of each projection window, mainly includes the following steps:

[0142] S6041: For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger.

[0143] S6042: Store the screen data streams in each cache queue into the memory.

[0144] S6043: Each software client communicates with the screen projection service across processes. Through the Texture framework, each screen data stream in the memory is textured on the projection window corresponding to the corresponding display device, and each projection window after texture drawing is displayed at the same time.

[0145] like Fig.13As shown in the figure, it is a system framework for rendering and displaying multiple screen projection windows at the same time, including the operating system kernel (OSKernel) layer, the service (Framework) layer, the application (APP) layer and the rendering display (Unity) layer. Among them, the OS Kernel layer is used to modify the kernel parameters to control the process of running multiple APPs, as well as to perform SLAM positioning and receive the handle posture. In the Framework layer, there is a Surface in the SurfaceFlinger corresponding to each app. The data of the App is stored in the corresponding buffer queue (Bufferqueue). The data stored in each Bufferqueue is aggregated into the memory (HardwareBuffer), and each frame of data will be updated. The screen projection service (Server) reads data from the Hardware Buffer through inter-process communication (Inter-Process Communication, IPC). In the APP layer, each APP is a Client of the Server, which is used to display the screen data stream sent by each display device. Therefore, a corresponding Texture will be created for each APP, and each screen data stream obtained by the Server will be updated on the Texture of the corresponding APP. In the Unity layer, the screen projection window is displayed at the target position of the screen projection window corresponding to each display device.

[0146] S605: Acquire the current user's viewpoint, and enlarge and display at least one target window in each projection window where the current user's viewpoint is located.

[0147] In one example, in order to facilitate user viewing, at least one target window in each projection window where the current user's viewpoint is located can be enlarged and displayed, so that the user can clearly read the projection content.

[0148] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the number of target windows. For example, the number may be 1, 2, 3, etc.

[0149] like Fig.14A As shown in FIG. 1 , the effect of enlarging and displaying a target window is shown in FIG. Fig. 14B As shown, the effect of enlarging the display of two target windows is shown.

[0150] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the magnification factor of the target window, such as 1.5 times that of other projection windows.

[0151] In one example, while displaying each projection window, the user can use gesture recognition technology to perform interactive operations such as zooming in, zooming out, dragging and dropping the selected projection window.

[0152] In one example, an embodiment of the present application can also use one-to-one screen projection between a display device and a VR device. At this time, the number of processes opened is 1, and the number of screen projection windows created is 1. Through the screen projection window, the VR device displays the screen data stream sent by the display device.

[0153] For example, Fig.15 As shown, in the movie watching scene, the user projects the movie screen on the PC to the HMD of the VR device for display, thereby obtaining the atmosphere of watching movies on a large screen and improving the user's viewing experience.

[0154] In the screen projection method in the virtual environment provided in the embodiment of the present application, not only one-to-one screen projection can be achieved, but also multiple processes can be started by modifying the kernel parameters, thereby creating and controlling multiple screen projection windows to achieve many-to-one screen projection. By projecting the 2D screen of the traditional display device into the 3D space of the VR device for process display, an immersive experience that cannot be provided by traditional flat screen projection is provided, and a large-screen display function is provided. In addition, during the screen projection process, the screen projection window can also be dynamically adjusted according to the number of display devices connected to the VR device and the changes in the user's perspective, thereby improving the flexibility of multi-screen display.

[0155] Based on the same technical concept, an embodiment of the present application provides a VR device, specifically an HMD of a VR device. The HMD of the VR device can implement the steps of the above-mentioned screen projection display method in a virtual environment and can achieve the same technical effect.

[0156] See also Fig.16 , the HMD of the VR device includes a processor 1601, a memory 1602, a display 1603 and a communication interface 1604, and the communication interface 1604, the display 1603, the memory 1602 and the processor 1601 are connected via a bus 1605;

[0157] The communication interface 1604 is used to establish a screen projection connection with at least one display device;

[0158] The memory 1602 stores a computer program, and the processor 1601 performs the following operations according to the computer program:

[0159] Get the initial number of display devices for screen projection, and modify the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to a process, and the number of processes is used to control the number of screen projection windows;

[0160] According to the number of projection windows and the position points of each display device, determine the initial position of the projection window corresponding to each display device in the display area of ​​the VR device;

[0161] When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is re-determined;

[0162] Calling at least one process to store the screen data streams sent by each display device in a memory, and performing texture drawing on the projection window at the target position corresponding to each display device according to each screen data stream in the memory, and displaying it through the display 1603;

[0163] The current user viewpoint is obtained, and at least one target window in each projection window where the current user viewpoint is located is enlarged and displayed.

[0164] Optionally, when the number of display devices changes or the user's viewing angle changes, the processor 1601 re-determines the target position of the screen projection window corresponding to each display device, and the specific operation is:

[0165] When the number of display devices changes, the kernel parameters are modified to update the number of processes according to the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes, and the display area is redivided according to the target number, and the target position of the projection window corresponding to each display device is determined according to the result of the redivision;

[0166] When a change in the user's viewing angle is detected, the position difference between the viewing angle origin before and after the change is obtained, and based on the position difference, the target position of the corresponding projection window of each display device in the new display area is determined.

[0167] Optionally, each screen data stream recorded by the display device corresponds to a SurfaceFlinger, each SurfaceFlinger contains a Surface, and the display of each screen data stream serves as a software client in the VR device screen projection service, and corresponds to a Texture framework for texture drawing in the application layer;

[0168] The processor 1601 calls at least one process to store the screen data streams sent by each display device in the memory, and performs texture drawing on the projection window of the target position corresponding to each display device according to each screen data stream in the memory, and displays it through the display 1603. The specific operations are:

[0169] For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger;

[0170] The screen data streams in each cache queue are uniformly stored in the memory;

[0171] Each software client communicates with the screen projection service across processes. Through the Texture framework, each screen data stream in the memory is textured on the projection window corresponding to the corresponding display device, and each projection window after texture drawing is simultaneously displayed through display 1603.

[0172] Optionally, the kernel parameters include at least the min_free_kbytes parameter and the oom_adj parameter.

[0173] Optionally, the processor 1601 determines the position of each screen projection window in the following manner:

[0174] Determine the location of each display device based on device images captured by multiple cameras on the VR device;

[0175] Take the current position of the VR device as the view origin and emit a horizontal ray directly in front of it;

[0176] Taking the horizontal ray as a reference, a spatial region corresponding to a left and right viewing angle of the initial ray being a first specified angle and a top and bottom viewing angle being a second specified angle is determined as a display region;

[0177] Divide the display area equally according to the number of display devices;

[0178] For each display device, the position of the projection window corresponding to the display device is determined based on a first distance from the display device to the current position of the VR device and a second distance from the position point of the display device to the ray that divides the display area.

[0179] in, Fig.16 The memory 1602 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 1602 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1602 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 1601 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0180] It should be noted that Fig.16This is just an example, and provides the necessary hardware for a VR device to execute the steps of a method for projecting and displaying a screen in a virtual environment provided in an embodiment of the present application. If not shown, the virtual reality device may also include hardware of a conventional VR device such as a speaker, a pickup, function buttons, a power supply, an IMU, and a handle.

[0181] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a screen projection display method in a virtual environment in the aforementioned embodiment.

[0182] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a screen projection display method in a virtual environment in the aforementioned embodiment.

[0183] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0184] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A screen projection display method in a virtual environment, characterized in that: Applied to a VR device, the VR device establishes a screen projection connection with at least one display device, and the method includes: Obtaining the initial number of display devices for screen projection, and modifying the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to a process, and the number of processes is used to control the number of screen projection windows; Determine, according to the number of the projection windows and the position points of each display device, an initial position of the projection window corresponding to each display device in the display area of ​​the VR device; When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is re-determined; Calling at least one process to store the screen data streams sent by each display device in a memory, and performing texture drawing and display on the projection window corresponding to the target position of each display device according to each screen data stream in the memory; The current user viewpoint is obtained, and at least one target window in each projection window where the current user viewpoint is located is enlarged and displayed.

2. The method according to claim 1, characterized in that When the number of display devices changes or the user's viewing angle changes, re-determining the target position of the projection window corresponding to each display device includes: When the number of display devices changes, the kernel parameters are modified again to update the number of processes according to the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes, and the display area is re-divided according to the target number, and the target position of the projection window corresponding to each display device is determined according to the result of the re-division; When a change in the user's viewing angle is detected, the position difference between the viewing angle origin before and after the change is obtained, and based on the position difference, the target position of the corresponding projection window of each display device in the new display area is determined.

3. The method according to claim 1, characterized in that The screen data stream recorded by each display device corresponds to a SurfaceFlinger, each SurfaceFlinger contains a Surface, and the display of each screen data stream serves as a software client in the VR device screen projection service, and corresponds to a Texture framework for texture drawing in the application layer; Calling at least one process to store the screen data streams sent by each display device in a memory, and performing texture drawing and display on the projection window corresponding to the target position of each display device according to each screen data stream in the memory, including: For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger; The screen data streams in each cache queue are uniformly stored in the memory; Each software client communicates with the screen projection service across processes. Through the Texture framework, each screen data stream in the memory is textured on the projection window corresponding to the corresponding display device, and each projection window after texture drawing is displayed at the same time.

4. The method according to any one of claims 1 to 3, characterized in that The kernel parameters include at least a min_free_kbytes parameter and an oom_adj parameter.

5. The method according to any one of claims 1 to 3, characterized in that The position of each projection window is determined by: Determine the location of each display device based on device images captured by multiple cameras on the VR device; Taking the current position of the VR device as the view origin, a horizontal ray is emitted directly in front of it; Taking the horizontal ray as a reference, determining a spatial region corresponding to a left-right viewing angle of the initial ray being a first specified angle and a top-bottom viewing angle being a second specified angle as a display region; Equally dividing the display area according to the number of display devices; For each display device, the position of the projection window corresponding to the display device is determined based on a first distance from the display device to the current position of the VR device and a second distance from the position point of the display device to the ray that divides the display area.

6. A VR device, characterized in that: It includes a processor, a memory, a display and a communication interface, wherein the communication interface, the display, the memory and the processor are connected via a bus; The communication interface is used to establish a screen projection connection with at least one display device; The memory stores a computer program, and the processor performs the following operations according to the computer program: Obtaining the initial number of display devices for screen projection, and modifying the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to a process, and the number of processes is used to control the number of screen projection windows; Determine, according to the number of the projection windows and the position points of each display device, an initial position of the projection window corresponding to each display device in the display area of ​​the VR device; When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is re-determined; Calling at least one process to store the screen data streams sent by each display device in a memory, and performing texture drawing on the projection window at the target position corresponding to each display device according to each screen data stream in the memory, and displaying it through the display; The current user viewpoint is obtained, and at least one target window in each projection window where the current user viewpoint is located is enlarged and displayed.

7. The VR device according to claim 6, wherein: When the number of display devices changes or the user's viewing angle changes, the processor re-determines the target position of the projection window corresponding to each display device, and the specific operation is: When the number of display devices changes, the kernel parameters are modified again to update the number of processes according to the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes, and the display area is re-divided according to the target number, and the target position of the projection window corresponding to each display device is determined according to the result of the re-division; When a change in the user's viewing angle is detected, the position difference between the viewing angle origin before and after the change is obtained, and based on the position difference, the target position of the corresponding projection window of each display device in the new display area is determined.

8. The VR device according to claim 6, wherein: The screen data stream recorded by each display device corresponds to a SurfaceFlinger, each SurfaceFlinger contains a Surface, and the display of each screen data stream corresponds to a Texture framework for texture drawing in the application layer as a software client in the VR device screen projection service; the processor calls at least one process to store the screen data streams sent by each display device in the memory respectively, and performs texture drawing on the projection window of the target position corresponding to each display device according to each screen data stream in the memory, and displays it through the display, and the specific operations are: For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger; The screen data streams in each cache queue are uniformly stored in the memory; Each software client communicates with the screen projection service across processes. Through the Texture framework, each screen data stream in the memory is textured on the projection window corresponding to the corresponding display device, and each projection window after texture drawing is simultaneously displayed through the display.

9. The VR device according to any one of claims 6 to 8, characterized in that: The kernel parameters include at least a min_free_kbytes parameter and an oom_adj parameter.

10. The VR device according to any one of claims 6 to 8, characterized in that: The processor determines the position of each projection window in the following manner: Determine the location of each display device based on device images captured by multiple cameras on the VR device; Taking the current position of the VR device as the view origin, a horizontal ray is emitted directly in front of it; Taking the horizontal ray as a reference, determining a spatial region corresponding to a left-right viewing angle of the initial ray being a first specified angle and a top-bottom viewing angle being a second specified angle as a display region; Equally dividing the display area according to the number of display devices; For each display device, the position of the projection window corresponding to the display device is determined based on a first distance from the display device to the current position of the VR device and a second distance from the position point of the display device to the ray that divides the display area.

Images