A method for screen projection display in a virtual environment and a VR device

Abstract

This application relates to the field of VR technology, providing a screen projection display method and VR device in a virtual environment to improve the flexibility of multi-screen display. The method is based on SLAM spatial positioning technology and Android system multi-window projection technology. By modifying the kernel parameters of the Android system, multiple processes controlling the number of projection windows are started. The projection window is located according to the position of each display device, thereby achieving dynamic creation and control of the projection windows. Furthermore, during the projection process, when the number of display devices or the user's viewing angle changes, the position of the projection window corresponding to each display device is repositioned, achieving dynamic adjustment of windows within the field of view to ensure the best viewing experience for the user. On the other hand, the method highlights at least one target window where the current user's viewpoint is located, making it easier for the user to view the projection content and enhancing the user's immersive experience.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and provides a method for screen projection display in a virtual environment and a VR device. Background Technology

[0002] VR devices, by providing users with an immersive experience, are now widely used in various industries, such as education and training, fire drills, virtual driving, and real estate. With the widespread application of VR devices, immersive experiences have become very popular among users. Users can use VR projection technology to project 2D images from traditional display devices (such as PCs, TVs, mobile phones, tablets, etc.) into the 3D virtual space of VR devices for playback.

[0003] Currently, based on the multi-window display function of the Android system, VR devices can simultaneously project the screen images of multiple display devices, allowing users to interact with multiple windows in a virtual environment and achieve VR multi-screen interaction. Through VR screen projection, users can enjoy fully immersive gaming, movie watching, and office work.

[0004] However, Androdi's multi-window display is implemented for pre-installed applications. This means that the number of windows is predefined. As a result, during screen mirroring, the number of windows cannot be dynamically adjusted when a user leaves or a new user joins. Furthermore, the user's perspective may change during screen mirroring, and the predefined multi-window feature cannot meet the needs of dynamic adjustment in a virtual environment. Summary of the Invention

[0005] This application provides a method for screen projection display in a virtual environment and a VR device to improve the flexibility of projecting from a 2D screen to a 3D space.

[0006] On one hand, embodiments of this application provide a screen projection display method in a virtual environment, applied to a VR device, wherein the VR device establishes a screen projection connection with at least one display device, and the method includes:

[0007] The initial number of display devices to be projected is obtained, and the kernel parameters of the Android system are modified according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to one process, and the number of processes is used to control the number of projection windows;

[0008] Based on the number of projection windows and the location of each display device, determine the initial position of the projection window corresponding to each display device within the display area of ​​the VR device;

[0009] When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is redefined.

[0010] At least one process is invoked to store the screen data streams sent by each display device in memory. Based on the screen data streams in memory, textures are drawn and displayed on the projection window at the corresponding target position of each display device.

[0011] Obtain the current user viewpoint and enlarge at least one target window in each projection window where the current user viewpoint is located.

[0012] On the other hand, this application provides a VR device including a processor, a memory, a display, and a communication interface, wherein the communication interface, the display, the memory, and the processor are connected via a bus;

[0013] The communication interface is used to establish a projection connection with at least one display device;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] The initial number of display devices to be projected is obtained, and the kernel parameters of the Android system are modified according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to one process, and the number of processes is used to control the number of projection windows;

[0016] Based on the number of projection windows and the location of each display device, determine the initial position of the projection window corresponding to each display device within the display area of ​​the VR device;

[0017] When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is redefined.

[0018] At least one process is invoked to store the screen data streams sent by each display device in memory. Based on the screen data streams in memory, textures are drawn on the projection window at the corresponding target position of each display device and displayed through the display.

[0019] Obtain the current user viewpoint and enlarge at least one target window in each projection window where the current user viewpoint is located.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a screen projection display method in a virtual environment provided in embodiments of this application.

[0021] The beneficial effects of the screen projection display method in a virtual environment provided in this application embodiment are as follows:

[0022] Leveraging Android's support for multi-window functionality, when at least one display device simultaneously projects its screen to a VR device, the Android system's kernel parameters are modified based on the initial number of display devices. This initiates multiple processes controlling the number of projection windows. SLAM spatial positioning technology is used to locate the projection window corresponding to each display device, enabling dynamic creation and control of projection windows and enhancing the flexibility of multi-screen displays. Furthermore, during projection, when the number of display devices or the user's viewing angle changes, the target position of the projection window corresponding to each display device is repositioned, dynamically adjusting the windows within the field of view to ensure optimal viewing for the user. Simultaneously, at least one target window at the current user's viewpoint is highlighted for easier viewing of the projection content, enhancing the user's immersive experience.

[0023] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of a conventional planar projection provided for an embodiment of this application;

[0026] Figure 2 A schematic diagram of a VR device provided in an embodiment of this application;

[0027] Figure 3 A schematic diagram illustrating the correspondence between the display device and the projection window provided in an embodiment of this application;

[0028] Figure 4 A flowchart illustrating the method for establishing a screen projection connection between a VR device and a display device as provided in this application embodiment;

[0029] Figure 5 This is a schematic diagram illustrating the screen projection display logic of the VR device and the display device provided in the embodiments of this application;

[0030] Figure 6 A flowchart illustrating the screen projection method in a virtual environment provided in this application embodiment;

[0031] Figure 7 A flowchart of the screen projection window positioning method provided in the embodiments of this application;

[0032] Figure 8A and Figure 8B This is a schematic diagram of coordinate system transformation provided in an embodiment of this application;

[0033] Figure 9A This is a schematic diagram illustrating the display area division of a VR device provided in an embodiment of this application;

[0034] Figure 9B and Figure 9C A schematic diagram of the projection window corresponding to the ray that divides the display area provided in this application embodiment;

[0035] Figure 10 This is a flowchart illustrating a method for determining the projection window corresponding to a display device, as provided in an embodiment of this application.

[0036] Figures 11A to 11D This is a schematic diagram illustrating the process of determining the projection window corresponding to the display device, as provided in an embodiment of this application.

[0037] Figure 12 A flowchart illustrating the rendering and display process of each projection window provided in the embodiments of this application;

[0038] Figure 13 This application provides a system framework diagram for the simultaneous rendering and display of multiple projection windows in an embodiment.

[0039] Figure 14A and Figure 14B This is a schematic diagram illustrating the highlighted projection window provided in an embodiment of this application;

[0040] Figure 15 This is a schematic diagram of one-to-one screen projection provided in an embodiment of this application;

[0041] Figure 16 A structural diagram of a virtual reality device provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0043] In real life, multiple users can interact in real time by projecting their screens onto electronic devices. Taking the education and training industry as an example, teachers can use the content projected from students' screens to promptly understand students' learning status and homework completion, and conduct comparative analysis of students' knowledge mastery.

[0044] Traditional flat-screen projection is a 2D device-to-2D device projection method. For example... Figure 1 As shown, multiple students project content from their tablets onto the teacher's smart TV screen, with each tablet Ci corresponding to a display window i on the smart TV screen. However, when multiple tablets are projected onto the smart TV simultaneously, the more display windows there are, the smaller the display area occupied by each student's screen becomes. It is not easy for the teacher to intuitively distinguish which display window corresponds to which student's screen content. Furthermore, the position and screen of the smart TV are fixed, making it difficult to compare and control the students' projected content, resulting in a rather unfriendly teaching experience.

[0045] Under limited conditions, when users want to enjoy large-screen viewing, reading, and teaching experiences, traditional flat-screen projection cannot meet their needs, and it lacks immersion and interactivity.

[0046] With the rise of VR devices on the market, immersive interactive experiences have gradually become known and recognized, and VR projection is being used more and more widely in education, training, games, entertainment and other fields.

[0047] VR devices typically consist of a head-mounted display (HMD) and controllers, which can be connected via wired or wireless means, such as... Figure 2 As shown, the HMD is equipped with a multi-view camera that can capture environmental images and images of the controller's light ring. Based on the images captured by the multi-view camera, the HMD performs SLAM spatial localization, calculating the controller's 6DOf pose (i.e., position and orientation) relative to the HMD. The HMD can also act as a screen mirroring receiver to provide screen mirroring services. The controller has a small structural space and is mainly controlled by a microcontroller unit (MCU). The controller has physical function keys that can control the screen displayed on the HMD based on the localization results. To save power, the controller uses a reset circuit to control the light ring to flash.

[0048] VR devices allow users to project 2D images from traditional displays into a 3D virtual space for playback, creating an immersive interactive experience on a large screen. Starting with Android 7.0 (API level 24), the Android system introduced a multi-window mode, supporting the simultaneous display of multiple application (app) windows. In multi-window mode, VR devices can simultaneously project screen images from multiple display devices, allowing users to interact with multiple windows within a virtual environment, achieving VR multi-screen interaction.

[0049] The window modes in the Android system mainly include the following categories:

[0050] (1) Standard Mode

[0051] Standard mode is the default display mode of the Android system. In this mode, applications are displayed in full screen. That is, in standard mode, users can only see the content of one application, while other applications run in the background.

[0052] (2) Split-Screen Mode

[0053] Split-screen mode divides the screen into two parts, allowing two applications to be displayed simultaneously. In split-screen mode, users can adjust the size of applications by dragging their edges, enabling data interaction.

[0054] (3) Freeform Window Mode

[0055] Freeform window mode allows two or more applications to be displayed simultaneously, similar to window modes in desktop operating systems. In freeform window mode, users can drag application windows to any position on the screen and resize them. This mode is suitable for tablets and large-screen devices.

[0056] However, Androdi's multi-window display is implemented for pre-installed applications. In other words, the multi-window is predefined. For windows displaying installed applications, users can freely drag and drop and interact with applications between two or more windows, but the number of windows cannot be adjusted in real time. Furthermore, the user's perspective changes during the projection process. Therefore, the predefined multi-window cannot meet the needs of dynamic adjustment in a virtual environment.

[0057] For example, when the number of projection devices n is greater than 2, such as n=7, that is, 7 active windows can be displayed simultaneously in the virtual scene. The window size can be adjusted, and different windows can be dragged and dropped and interacted with. However, when two users leave or two new users join during the projection process, that is, n=5 or n=9, the VR device cannot reduce or increase the number of windows to meet the multi-screen display requirements.

[0058] In view of this, this application provides a screen projection display method in a virtual environment. This method is based on SLAM spatial positioning technology and Android system multi-window projection technology. When multiple display devices simultaneously project onto a VR device, it modifies the kernel parameters in the VR device's Android system to start multiple processes controlling the number of windows and locates the projection window corresponding to each display device. This allows for dynamic control of the projection window display based on the number of display devices, enabling interaction between multiple projection windows on multiple display devices in the virtual environment, enhancing the user's immersion and interactive experience. Furthermore, during the projection process, when the number of display devices changes or the user's viewpoint changes, the position of the projection window corresponding to each display device is repositioned to ensure the best viewing effect for the user. On the other hand, this method highlights the window where the user's current viewpoint is located, making it easier for the user to view the projected content.

[0059] During screen mirroring, a VR device can establish a mirroring connection with one or more display devices. When a VR device establishes a mirroring connection with multiple display devices, such as... Figure 3 As shown, the screen mirroring service (Server) in the HMD of the VR device will launch multiple software clients (Client), each client corresponding to a display terminal (DT). When the DT starts the browser screen mirroring, the VR device uses SLAM spatial positioning technology to predict the position of each DT in order to display the screen mirroring window corresponding to each DT.

[0060] VR devices can display whether they are connected wirelessly or via wired connection.

[0061] For example, VR devices can establish a projection connection with display devices through cellular mobile communication technologies, such as 5G technology.

[0062] For example, VR devices can also establish a projection connection with a display terminal through short-range wireless technologies, such as Wireless Fidelity (Wi-Fi) technology.

[0063] For example, VR devices can also establish a screen mirroring connection by connecting to the same local area network as the display device.

[0064] Taking a display device as an example, the process of establishing a screen mirroring connection between a VR device and a display device can be found in [link to documentation]. Figure 4 It mainly includes the following steps:

[0065] S401: The VR device and the display device are connected to the same local area network.

[0066] S402: Enter the IP address and port number of the VR device on the display device.

[0067] S403: The display device opens a webpage through a browser and executes the screen recording logic written in the webpage.

[0068] S404: The display device sends a screen projection request to the VR device using the input IP address and port number, based on the screen recording logic.

[0069] S405: The VR device requests screen recording permission from the display device based on the screen projection request.

[0070] S406: After confirming that the VR device has granted screen recording permission, the display device establishes a WebSocket connection with the VR device.

[0071] S407: After the WebSocket connection is established, the display device sends a recorded screen data stream to the VR device.

[0072] S408: The VR device displays the received screen data stream.

[0073] During screen mirroring, the VR device's HMD acts as the server, running both an HTTP server and a WebSocket server. The HTTP server provides access to web page files, allowing the browser to access resources within the webpage, and the screen recording logic is written within the webpage. The WebSocket server receives connections established by the browser and receives the recorded screen mirroring data stream. The webpage, running in the browser, records the screen mirroring data stream and sends it to the VR device's HMD. The HMD temporarily stores the received screen mirroring data stream in a StreamBuffer. When the player plays the screen mirroring data, it reads the screen mirroring data stream from the StreamBuffer for playback.

[0074] like Figure 5The diagram illustrates the screen mirroring logic between a VR device and a display device. The user accesses an HTTP server through a browser on the display device to open a webpage. During screen mirroring, the webpage's screen recording logic is executed to establish a WebSocket connection with the VR device. The recorded screen mirroring data stream is then sent to the WebSocket server on the VR device's receiving end, allowing the VR device to receive the data stream and temporarily store it in a StreamBuffer. Once the VR device identifies the corresponding screen mirroring window on the display device, it retrieves the screen mirroring data stream from the StreamBuffer and plays it on the corresponding window using the VR device's player.

[0075] For example, during the screen mirroring process, the screen mirroring protocol used by the VR device and the display device can be browser screen mirroring or mirroring based on a proprietary protocol.

[0076] After a VR device establishes a projection connection with at least one display device, the VR device displays the projected content from the at least one display device in virtual space. (See below) Figure 6 This is a flowchart of a screen projection display method in a virtual environment provided in this application embodiment. The process is executed by a VR device, specifically by an HMD, and mainly includes the following steps:

[0077] S601: Obtain the initial number of display devices to be projected, and modify the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to one process. The number of processes is used to control the number of projection windows.

[0078] In practical applications, there may be situations where multiple display devices simultaneously project content onto a VR device. In order to display the content projected from multiple display devices at the same time and enable interaction with different projected content, the number of projection windows can be controlled according to the initial number of display devices projecting content.

[0079] Currently, VR devices generally use the Android system. In one example, the VR device modifies the Android system kernel parameters based on the initial number of display devices to be projected, in order to support multiple processes. In this way, the VR device can call multiple processes to run multiple software clients simultaneously. Each software client is used to display the screen data stream sent by each display device, and controls the number of projection windows and creates projection windows according to the number of processes, so that each display device corresponds to one projection window, thereby enabling the simultaneous display of screen data streams sent by multiple devices.

[0080] For example, the modified kernel parameters include at least the min_free_kbytes and oom_adj parameters used to control the number of processes.

[0081] S602: Based on the number of projection windows and the location of each display device, determine the initial position of the projection window corresponding to each display device within the display area of ​​the VR device.

[0082] In one example, after creating multiple projection windows, SLAM spatial positioning technology is used to determine the position of the projection window corresponding to each display device.

[0083] Considering that the optimal viewing distance from the screen window to the user is generally 3-5 meters, to ensure a good viewing experience, an optimal viewing area can be created in the virtual space based on the user's eye position as the display area for each projection window. Then, based on the user's location (i.e., the position of the HMD in the VR device), the position of each projection window is determined using a proximity grid method. For specific implementation details on determining the projection window position, please refer to [link to documentation]. Figure 7 It mainly includes the following steps:

[0084] S6021: Determine the position points of each display device based on the device images captured by multiple cameras on the VR device.

[0085] In practical applications, based on the device images captured by multiple cameras on the VR device, SLAM spatial positioning technology is used to calculate the map position coordinates of each display device in the map coordinate system. Then, each map position coordinate is converted into the projection position coordinates of the display device in the projection display coordinate system of the VR device to obtain the position point of each display device.

[0086] In one embodiment, the map location coordinates of the display device can be converted into screen projection location coordinates in the following two ways:

[0087] Method 1: If the coordinate system parameters of the map coordinate system are the same as those of the projection display coordinate system, then the map position coordinates are scaled down by a specified factor to obtain the projection position coordinates of the display device. The coordinate system parameters include the origin position, the position and direction of the horizontal axis, the position and direction of the vertical axis, and the position and direction of the vertical coordinate axis.

[0088] In this embodiment, both the map coordinate system and the projection coordinate system are spatial rectangular coordinate systems established with the center point of the VR device (specifically, HMD) as the origin, and the coordinate system parameters of both the map coordinate system and the projection coordinate system are preset. For example, ... Figure 8A As shown, both the map coordinate system and the projection display coordinate system have the center point of the HMD as the origin, and the position and direction of each coordinate axis are the same.

[0089] It should be noted that the specified multiple in the embodiments of this application can be set according to the actual situation, and the embodiments of this application do not limit the specified multiple.

[0090] Method 2: If the coordinate system parameters of the map coordinate system are different from those of the projection display coordinate system, the map position coordinates will be transformed using a pre-set rotation and translation matrix between the map coordinate system and the projection display coordinate system to obtain the projection position coordinates of the display device.

[0091] like Figure 8B As shown, the x and y axes of the map coordinate system and the projection display coordinate system are not in the same direction. As can be seen from the figure, the position of p(x, y, z) in the map coordinate system is (x1, y1, z1) when transformed into the projection display coordinate system.

[0092] After determining the position of each display device in the projection display coordinate system, the optimal viewing space area is established under the projection display coordinate system.

[0093] S6022: Using the current position of the VR device as the origin of the viewpoint, emit an initial ray directly in front of it.

[0094] In one example, the origin of the viewpoint for the optimal viewing space area is set at the current position of the VR device's HMD. A horizontal ray is then emitted directly in front of it, and the range of the optimal viewing space area is determined by this horizontal ray.

[0095] S6023: Based on the initial ray, the spatial area corresponding to the left and right field of view angles of the initial ray as the first specified angle and the up and down field of view angles as the second specified angle is determined as the display area.

[0096] For example, the range of the first specified angle can be (0°, 160°), and the range of the second specified angle can be (0°, 90°). The specific angle values ​​of the first and second specified angles can be set according to the actual situation, and this application embodiment does not limit the specific angle values ​​of the first and second specified angles.

[0097] S6024: Divide the display area equally according to the number of display devices.

[0098] In one example, with the current position of the HMD as the origin, the same number of rays are emitted within the display area according to the number of display devices, wherein each ray is in the same horizontal plane and the included angle between any two adjacent rays is the same.

[0099] For example, the angle between any two adjacent rays is obtained by dividing a first specified angle by the number of display devices. Figure 9AAs shown, the number of display devices is 5, and the angle between any two adjacent rays is θ, which are emitted within the display area.

[0100] S6025: For each display device, determine the position of the projection window corresponding to the display device based on the first distance from the current position of the display device to the VR device, and the second distance from the position point of the display device to the ray dividing the display area.

[0101] In practical applications, when using the nearest grid method to determine the position of the projection window corresponding to each display device, we can first determine the display type and position of the projection window corresponding to each ray emitted in the display area based on the first distance from the display device to the current position of the VR device. Then, based on the second distance from the position point of each display device to the division of each ray, the projection window corresponding to the nearest ray is taken as the projection window corresponding to the display device.

[0102] In this embodiment, the display type of the projection window includes a single-layer display type and a multi-layer display type. In the single-layer display type, the number of projection windows corresponding to any ray is one, and in the display type, the number of projection windows corresponding to any ray is greater than one.

[0103] In one embodiment, the display type and position of the projection window corresponding to each ray can be determined in the following two ways:

[0104] Method 1: If the number of display devices is less than the specified number, the display type is determined to be a single-layer display type, and for any ray, the position of the projection window corresponding to the ray is obtained according to the first distance.

[0105] In one embodiment, the position of the projection window corresponding to any ray can be obtained by: for any ray, determining the position of the target point with a length of a first distance between the endpoint of the ray and the target point as the position of the projection window corresponding to the ray.

[0106] like Figure 9B As shown, taking the first distance as r as an example, the position of the projection window corresponding to ray 1 is the position of point A, the position of the projection window corresponding to ray 2 is the position of point B, the position of the projection window corresponding to ray 3 is the position of point C, the position of the projection window corresponding to ray 4 is the position of point D, and the position of the projection window corresponding to ray 5 is the position of point E.

[0107] Method 2: If the number of display devices is not less than the specified number, the display type is determined to be a multi-layer display type, and for any ray, the positions of the multiple projection windows corresponding to the ray are determined according to the first distance.

[0108] In one embodiment, the positions of the multiple projection windows corresponding to each ray can be determined in the following way:

[0109] For any given ray, based on the number of display devices and the number of each ray, the target total number of projection windows corresponding to that ray is obtained. After determining the position of a target point with a length of a first distance from the endpoint of the ray as the position of a projection window corresponding to that ray, the current number of projection windows for that ray is updated. If the current number of projection windows for that ray after the update is less than the target total number, then the first distance with a length of a multiple of the target distance from the endpoint of the ray is used to determine the horizontal and vertical coordinates of a projection window. The vertical coordinate of the previously determined projection window is then increased by a fixed multiple to obtain the position of a projection window. The process then returns to updating the current number of projection windows corresponding to that ray until the current number of projection windows for that ray is not less than the target total number. Here, the target multiple is obtained based on the current number of projection windows for that ray.

[0110] In this embodiment, the target multiplier is obtained by adding 1 to the projection window of the ray. However, this embodiment does not limit the target multiplier, and it can be set according to actual conditions. Furthermore, in this embodiment, the target total number of projection windows corresponding to any ray is obtained by dividing the number of display devices by the number of each ray. If the value obtained by dividing the number of display devices by the number of each ray is not an integer, the integer part of that value is added by 1 to determine the target total number of projection windows corresponding to any ray.

[0111] Taking an example where the total number of targets in the projection window corresponding to any ray is 2, such as Figure 9C As shown, each ray corresponds to two projection windows.

[0112] In both of the above methods, the first distance is obtained based on the projection position coordinates of each display device and the current position of the HMD of the VR device. Specifically, the first distance can be determined as follows: For any display device, obtain the sub-distance between the display device and the VR device based on the map position coordinates of the display device and the current position of the HMD; multiply the sub-distance by a pre-set weight to obtain the intermediate length corresponding to the display device; and determine the average of the intermediate lengths of each display device as the first distance.

[0113] After determining the position of the projection window corresponding to each ray, for any display device, the second distance from the position point of that display device in the projection coordinate system to each ray is used to determine the projection window corresponding to that display device. See [link to detailed process] for more information. Figure 10 It mainly includes the following steps:

[0114] S6025_1: For any display device, determine the second distance between the terminal device and each ray based on the location of the display device.

[0115] In one embodiment, for any ray, the equation of the line corresponding to the ray is obtained based on the position of the projection window corresponding to the ray and the origin. Based on the projection position coordinates of the display device, the second distance from the terminal device to the ray is obtained using the distance equation from the point to the line.

[0116] S6025_2: The ray corresponding to the shortest second distance is determined as the target ray corresponding to the display device.

[0117] like Figure 11A As shown, assuming that the distance between display device a and ray 1 is the shortest, then ray 1 is taken as the target ray corresponding to display device a.

[0118] S6025_3: Based on the target rays corresponding to each display device, obtain the target display device corresponding to each ray.

[0119] In one embodiment, for any ray, the process of traversing each display device is performed. For any traversed display device, if the ray is the target ray of that display device, then the display terminal device is determined as the target terminal device corresponding to the ray, and the process of traversing each display device is returned until there are no untraversed display devices.

[0120] S6025_4: For any ray, if the number of target display devices corresponding to the ray is one and the number of projection windows corresponding to the ray is one, then the projection window corresponding to the ray is determined as the projection window corresponding to the target display device.

[0121] like Figure 11B The image shows a one-to-one screen projection between a display device and a VR device. In this embodiment, the size of the projection window is pre-set, but can be adjusted according to actual conditions. This embodiment does not limit the size of the projection window.

[0122] S6025_5: For any ray, if the number of target display devices corresponding to the ray is one and the number of projection windows corresponding to the ray is multiple, then the projection window that is closest to the target display device among the projection windows corresponding to the ray is determined as the projection window corresponding to the target display device.

[0123] like Figure 11CAs shown in the figure, the target display device corresponding to ray 5 is display device m, and the projection windows corresponding to ray 5 are projection window P and projection window Q. It can be seen from the figure that projection window P is the projection window closest to display device m; therefore, projection window P is determined to be the projection window corresponding to display device m.

[0124] S6025_6: For any ray, if there are multiple target display devices corresponding to the ray, then traverse each projection window corresponding to the ray according to the distance from the HMD from smallest to largest.

[0125] S6025_7: For any screen projection window that has been traversed, determine that screen projection window as the target display device screen projection window that is closest to that screen projection window.

[0126] S6025_8: Determine whether the target display devices corresponding to the ray have all determined the corresponding projection window. If yes, end; otherwise, return to execute S6025_6.

[0127] like Figure 11D As shown in the figure, the target display devices corresponding to ray 4 are display device a and display device b, and the projection windows corresponding to ray 4 are projection window R and projection window H, respectively. It can be seen from the figure that projection window R is the projection window closest to display device a, therefore projection window R is determined to be the projection window corresponding to display device a. Similarly, projection window H is the projection window closest to terminal device b, therefore projection window H is determined to be the projection window corresponding to display device b.

[0128] In one example, after determining the initial position of the projection window corresponding to each display device within the display area of ​​the VR device using the above method, the number of display devices connected to the VR device may change due to the departure of old users or the addition of new users during the projection process. Furthermore, the viewing angle of the user wearing the HMD may change during the projection process. Therefore, it is necessary to dynamically adjust each projection window.

[0129] S603: When the number of display devices changes or the user's viewing angle changes, redetermine the target position of the projection window corresponding to each display device.

[0130] In one example, when the number of display devices changes, the number of projection windows displayed in the virtual environment is dynamically adjusted in real time by controlling the number of processes. Specifically, the kernel parameters are modified to update the number of processes based on the target number of display devices after the change, the number of projection windows is adjusted according to the number of processes, and the display area is re-divided according to the target number. The target position of the projection window for each display device is then determined based on the re-division result.

[0131] For example, when performing multi-screen projection (i.e., the number of display devices n is greater than 2), assuming the initial number of display devices is 7, during the projection process, (1) if two new users join, the target number of display devices is 9, then the parameters such as min_free_kbytes and oom_adj in the Android system are modified to add two processes, thereby adding one projection window for each of the two newly joined display devices, that is, the number of projection windows is 9; (2) if two old users leave, the target number of display devices is 5, then the parameters such as min_free_kbytes and oom_adj in the Android system are modified to reduce two processes, thereby reducing the number of projection windows for the screen data stream sent by the exiting display devices, that is, the number of projection windows is 5.

[0132] When the number of projection windows is increased or decreased according to the target number of display devices, in order to ensure the coordination of the display of each projection window and avoid the situation of disordered arrangement of the projection windows, the position of each window in the display area can be updated to determine the target position of each window.

[0133] In one example, when redetermining the position of each window within the display area based on the adjusted number of projection windows, it can be viewed as rearranging the projection windows. Therefore, the process of determining the target position of each projection window is similar to the process of determining the initial position described above. See [link to documentation] for details. Figures 7 to 11D The process will not be repeated here.

[0134] In one example, when the user's viewing angle changes, the original optimal viewing area is no longer applicable, so it is necessary to redetermine the new display area for each projection window. In practice, the pose difference of the origin of the viewing angle before and after the change is obtained, and based on the pose difference, the target position of the projection window for each display device within the new display area is determined.

[0135] For example, pose differences only calculate changes in the horizontal direction and ignore changes in the vertical direction.

[0136] For example, when only the user's perspective changes and the number of display devices remains the same, the number of projection windows remains unchanged. In this case, the position of each projection window can be shifted only according to the pose difference.

[0137] In the embodiments of this application, during the screen projection process, multiple projection windows are calibrated once within the field of view according to the number of display devices and the changes in the user's viewing angle, thereby realizing the change of windows in the field of view and ensuring that what the user sees is what they get.

[0138] After calibrating each projection window, the projection content from multiple display devices is rendered and displayed in a virtual environment based on the target position of each projection window.

[0139] S604: Call at least one process to store the screen data streams sent by each display device in memory, and draw and display textures on the projection window at the corresponding target position of each display device according to the screen data streams in memory.

[0140] In one example, each screen data stream recorded by a display device corresponds to a SurfaceFlinger, and each SurfaceFlinger contains a Surface. The display of each screen data stream serves as a software client in the VR device projection service, and at the application layer, it corresponds to a Texture framework for texture drawing.

[0141] See Figure 12 The rendering and display process for each projection window mainly includes the following steps:

[0142] S6041: For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger.

[0143] S6042: Stores the screen data streams in each buffer queue into memory.

[0144] S6043: Cross-process communication is performed between each software client and the screen mirroring service. Through the Texture framework, the screen data streams in memory are used to draw textures on the corresponding screen mirroring windows of the corresponding display devices, and the screen mirroring windows after texture drawing are displayed at the same time.

[0145] like Figure 13The diagram illustrates the system framework for simultaneously rendering and displaying multiple projection windows, comprising an operating system kernel (OSKernel) layer, a service (Framework) layer, an application (APP) layer, and a rendering and display (Unity) layer. The OSKernel layer modifies kernel parameters to control the processes running multiple apps, and performs SLAM localization and receives controller poses. In the Framework layer, each app has a corresponding SurfaceFlinger with a Surface. App data is stored in a corresponding buffer queue, and the data stored in each buffer queue is aggregated into memory (HardwareBuffer), updated with each frame of data. The projection service (Server) reads data from the Hardware Buffer via Inter-Process Communication (IPC). In the APP layer, each app is a client of the Server, used to display the screen data stream sent by each display device. Therefore, a corresponding Texture is created for each app, and each screen data stream received by the Server is updated on the corresponding app's Texture. In the Unity layer, the projection window is displayed at the target position for each display device.

[0146] S605: Obtain the current user viewpoint and enlarge at least one target window in each projection window where the current user viewpoint is located.

[0147] In one example, to facilitate user viewing, at least one target window in each projection window where the current user's viewpoint is located can be enlarged for display, allowing the user to clearly read the projection content.

[0148] It should be noted that the embodiments of this application do not impose a limiting requirement on the number of target windows. For example, there can be one, two, or three, etc.

[0149] like Figure 14A The image shows the effect of zooming in on a target window. Figure 14B The image shows the effect of magnifying the two target windows.

[0150] It should be noted that this application embodiment does not impose a limiting requirement on the magnification of the target window, such as 1.5 times that of other projection windows.

[0151] In one example, during the display of each projection window, the user can use gesture recognition technology to perform interactive operations such as zooming in, zooming out, and dragging the selected projection window.

[0152] In one example, this application embodiment can also use one-to-one screen projection between the display device and the VR device. In this case, the number of processes opened is 1, and the number of projection windows created is 1. Through the projection window, the VR device displays the screen data stream sent by the display device.

[0153] For example, such as Figure 15 As shown, in the movie-watching scenario, users project the movie screen from their PC onto the HMD of the VR device for display, thereby creating a large-screen viewing atmosphere and enhancing the user's viewing experience.

[0154] The screen projection method in the virtual environment provided in this application not only enables one-to-one projection but also allows for the creation and control of multiple projection windows by modifying kernel parameters to start multiple processes, achieving many-to-one projection. By projecting the 2D image from a traditional display device into the 3D space of a VR device, it provides an immersive experience that traditional planar projection cannot offer, and also provides large-screen display functionality. Furthermore, during the projection process, the projection window can be dynamically adjusted according to the number of display devices connected to the VR device and changes in the user's viewing angle, improving the flexibility of multi-screen display.

[0155] Based on the same technical concept, this application provides a VR device, specifically an HMD for a VR device. The HMD of the VR device can implement the steps of the above-mentioned screen projection display method in a virtual environment and achieve the same technical effect.

[0156] See Figure 16 The HMD of the VR device includes a processor 1601, a memory 1602, a display 1603 and a communication interface 1604, which are connected via a bus 1605.

[0157] Communication interface 1604 is used to establish a projection connection with at least one display device;

[0158] The memory 1602 stores a computer program, and the processor 1601 performs the following operations according to the computer program:

[0159] Obtain the initial number of display devices to be projected, and modify the kernel parameters of the Android system according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to one process. The number of processes is used to control the number of projection windows.

[0160] Based on the number of projection windows and the location of each display device, determine the initial position of the projection window corresponding to each display device within the display area of ​​the VR device;

[0161] When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is redefined.

[0162] At least one process is invoked to store the screen data streams sent by each display device in memory. Based on the screen data streams in memory, textures are drawn on the projection window at the corresponding target position of each display device and displayed through the monitor 1603.

[0163] Get the current user's viewpoint and enlarge at least one target window in each projection window that is the current user's viewpoint.

[0164] Optionally, when the number of display devices changes or the user's viewing angle changes, the processor 1601 redetermines the target position of the projection window corresponding to each display device. Specifically, the operation is as follows:

[0165] When the number of display devices changes, the kernel parameters are modified to update the number of processes based on the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes. In addition, the display area is re-divided according to the target number, and the target position of the projection window corresponding to each display device is determined based on the result of the re-division.

[0166] When a change in the user's viewing angle is detected, the pose difference of the origin of the viewing angle before and after the change is obtained, and based on the pose difference, the target position of the projection window of each display device in the new display area is determined.

[0167] Optionally, each screen data stream recorded by a display device corresponds to a SurfaceFlinger, each SurfaceFlinger contains a Surface, and the display of each screen data stream serves as a software client in the VR device projection service, corresponding to a Texture framework for texture drawing at the application layer.

[0168] The processor 1601 calls at least one process to store the screen data streams sent by each display device in memory. Based on the screen data streams in memory, it performs texture drawing on the projection window at the corresponding target position on each display device and displays it through the monitor 1603. The specific operation is as follows:

[0169] For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger;

[0170] The screen data streams in each cache queue are stored uniformly in memory;

[0171] Each software client communicates across processes with the screen mirroring service. Using the Texture framework, the screen data streams in memory are used to draw textures on the corresponding screen mirroring windows of the respective display devices. The textured screen mirroring windows are then displayed simultaneously on the monitor 1603.

[0172] Optionally, kernel parameters include at least the min_free_kbytes parameter and the oom_adj parameter.

[0173] Optionally, the processor 1601 determines the position of each projection window in the following way:

[0174] Based on the device images captured by multiple cameras on the VR device, determine the position points of each display device;

[0175] A horizontal ray is emitted directly in front of the VR device, using its current position as the origin of the viewpoint.

[0176] Using the horizontal ray as a reference, the spatial area corresponding to the left and right field of view of the initial ray as the first specified angle and the up and down field of view as the second specified angle is determined as the display area;

[0177] The display area is divided equally according to the number of display devices;

[0178] For each display device, the position of the projection window corresponding to the display device is determined based on the first distance from the current position of the display device to the VR device, and the second distance from the position point of the display device to the ray dividing the display area.

[0179] in, Figure 16 The memory 1602 can be volatile memory, such as random-access memory (RAM); the memory 1602 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory 1602 can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory can be a combination of the above-described memories; the processor 1601 can include one or more central processing units (CPUs) or digital processing units, etc.

[0180] It should be noted that, Figure 16This is merely an example illustrating the hardware necessary for a VR device to perform the screen projection display method steps provided in the embodiments of this application in a virtual environment. Not shown, the virtual reality device may also include hardware from conventional VR devices such as speakers, microphones, function buttons, a power supply, an IMU, and controllers.

[0181] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a screen projection display method in a virtual environment as described in the foregoing embodiments.

[0182] This application also provides a computer program product for storing a computer program that executes a screen projection display method in a virtual environment as described in the foregoing embodiments.

[0183] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for projecting display in a virtual environment, characterized in that, Applied to VR devices, wherein the VR device establishes a projection connection with at least one display device, the method includes: The initial number of display devices to be projected is obtained, and the kernel parameters of the Android system are modified according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to one process. The number of processes is used to control the number of projection windows. Based on the number of projection windows and the location of each display device, determine the initial position of the projection window corresponding to each display device within the display area of ​​the VR device; When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is redefined. At least one process is invoked to store the screen data streams sent by each display device in memory. Based on the screen data streams in memory, textures are drawn and displayed on the projection window at the corresponding target position of each display device. Obtain the current user viewpoint and enlarge at least one target window in each projection window where the current user viewpoint is located.

2. The method as described in claim 1, characterized in that, When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is redefined, including: When the number of display devices changes, the kernel parameters are modified to update the number of processes based on the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes. In addition, the display area is re-divided according to the target number, and the target position of the projection window corresponding to each display device is determined according to the result of the re-division. When a change in the user's viewing angle is detected, the pose difference of the origin of the viewing angle before and after the change is obtained, and the target position of the projection window of each display device in the new display area is determined based on the pose difference.

3. The method as described in claim 1, characterized in that, Each screen data stream recorded by a display device corresponds to a SurfaceFlinger, and each SurfaceFlinger contains a Surface. The display of each screen data stream serves as a software client in the VR device projection service, and at the application layer, it corresponds to a Texture framework for texture drawing. The process involves calling at least one process to store the screen data streams sent by each display device in memory, and then, based on the screen data streams in memory, performing texture drawing and display on the projection window at the corresponding target position on each display device, including: For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger; The screen data streams in each cache queue are stored uniformly in memory; Each software client communicates across processes with the screen mirroring service. Through the Texture framework, the screen data streams in memory are rendered into textures on the corresponding screen mirroring windows of the respective display devices, and the rendered textures are displayed in each screen mirroring window at the same time.

4. The method according to any one of claims 1-3, characterized in that, The kernel parameters include at least the min_free_kbytes parameter and the oom_adj parameter.

5. The method according to any one of claims 1-3, characterized in that, The positions of each projection window are determined in the following way: Based on the device images captured by multiple cameras on the VR device, determine the position points of each display device; A horizontal ray is emitted directly in front of the VR device, with the current position of the VR device as the origin of the viewpoint. Using the horizontal ray as a reference, the spatial area corresponding to the left and right field of view of the initial ray as the first specified angle and the up and down field of view as the second specified angle is determined as the display area; The display area is divided equally according to the number of display devices; For each display device, the position of the projection window corresponding to the display device is determined based on a first distance from the current position of the display device to the VR device, and a second distance from the position point of the display device to the ray dividing the display area.

6. A VR device, characterized in that, It includes a processor, a memory, a display, and a communication interface, wherein the communication interface, the display, the memory, and the processor are connected via a bus; The communication interface is used to establish a projection connection with at least one display device; The memory stores a computer program, and the processor performs the following operations according to the computer program: The initial number of display devices to be projected is obtained, and the kernel parameters of the Android system are modified according to the initial number to set the number of processes used to display the screen data stream sent by each display device, so that each screen data stream corresponds to one process. The number of processes is used to control the number of projection windows. Based on the number of projection windows and the location of each display device, determine the initial position of the projection window corresponding to each display device within the display area of ​​the VR device; When the number of display devices changes or the user's viewing angle changes, the target position of the projection window corresponding to each display device is redefined. At least one process is invoked to store the screen data streams sent by each display device in memory. Based on the screen data streams in memory, textures are drawn on the projection window at the corresponding target position of each display device and displayed through the display. Obtain the current user viewpoint and enlarge at least one target window in each projection window where the current user viewpoint is located.

7. The VR device as described in claim 6, characterized in that, When the number of display devices changes or the user's viewing angle changes, the processor redetermines the target position of the projection window corresponding to each display device. The specific operation is as follows: When the number of display devices changes, the kernel parameters are modified to update the number of processes based on the target number of display devices after the change, and the number of projection windows is adjusted according to the number of processes. In addition, the display area is re-divided according to the target number, and the target position of the projection window corresponding to each display device is determined according to the result of the re-division. When a change in the user's viewing angle is detected, the pose difference of the origin of the viewing angle before and after the change is obtained, and the target position of the projection window of each display device in the new display area is determined based on the pose difference.

8. The VR device as described in claim 6, characterized in that, Each screen data stream recorded by a display device corresponds to a SurfaceFlinger, and each SurfaceFlinger contains a Surface. The display of each screen data stream, as a software client in the VR device projection service, corresponds to a Texture framework for texture drawing at the application layer. The processor calls at least one process to store the screen data streams sent by each display device in memory. Based on each screen data stream in memory, texture drawing is performed on the projection window at the corresponding target position on each display device, and displayed through the monitor. The specific operation is as follows: For each process, the corresponding screen data stream is stored in a cache queue corresponding to the Surface in SurfaceFlinger; The screen data streams in each cache queue are stored uniformly in memory; Each software client communicates across processes with the screen mirroring service. Through the Texture framework, the screen data streams in memory are used to draw textures on the corresponding screen mirroring windows of the respective display devices, and the textured screen mirroring windows are displayed simultaneously on the monitor.

9. The VR device as described in any one of claims 6-8, characterized in that, The kernel parameters include at least the min_free_kbytes parameter and the oom_adj parameter.

10. The VR device as described in any one of claims 6-8, characterized in that, The processor determines the position of each projection window in the following way: Based on the device images captured by multiple cameras on the VR device, determine the position points of each display device; A horizontal ray is emitted directly in front of the VR device, with the current position of the VR device as the origin of the viewpoint. Using the horizontal ray as a reference, the spatial area corresponding to the left and right field of view of the initial ray as the first specified angle and the up and down field of view as the second specified angle is determined as the display area; The display area is divided equally according to the number of display devices; For each display device, the position of the projection window corresponding to the display device is determined based on a first distance from the current position of the display device to the VR device, and a second distance from the position point of the display device to the ray dividing the display area.

Images