A method, device and system for processing multiple windows in three-dimensional space

Abstract

The present invention relates to the field of computer systems, and provides a method, device and system for processing multiple windows in a three-dimensional space, by receiving attribute information of one or more windows, the windows including two-dimensional windows and / or three-dimensional windows; generating corresponding transformation matrices according to the attribute information of one or more windows; processing one or more windows based on the transformation matrix, wherein processing one or more windows based on the transformation matrix includes: displaying one or more windows based on the transformation matrix, and converting events in the three-dimensional space to a target window based on the transformation matrix, and sending them to the target window. The present invention can realize the simultaneous display of one or more two-dimensional windows and three-dimensional windows in a three-dimensional environment, thereby enabling users to interact with these windows at the same time, improving the application scenarios of extended reality devices, and enhancing user experience.

Description

Technical Field

[0001] The present invention relates to the fields of computer systems and human-computer interaction, and in particular to a multi-window processing method, device and system in a three-dimensional space. Background Art

[0002] Currently, the mainstream usage scenarios of virtual reality (VR) devices are gaming and watching movies, which mainly provide an immersive experience, that is, a three-dimensional (3D) application occupies the entire space, and users can only interact with one 3D application at a time. However, in many cases, users have multi-tasking needs, that is, users want to use an application on a VR device while also interacting with other applications. For example, when a user is playing a game on a VR device and receives a message, text message or phone call, the user hopes to have a pop-up message prompt and can easily reply to the message without having to exit or terminate the current application.

[0003] On augmented reality or mixed reality devices, users can see the real environment. At this time, the user experience is mainly non-immersive. Users hope to run multiple three-dimensional contents at the same time in a 360-degree space. For example, in an office scene, users hope to place a three-dimensional alarm clock application, a three-dimensional calendar application, and a three-dimensional notebook application on the desktop at the same time. However, based on current technology, users can only open one three-dimensional application in the space at the same time, and other applications will be suspended or hidden, resulting in a poor user experience.

[0004] Traditional PC operating systems such as Windows are two-dimensional (2D) multi-window systems, and users interact with multiple applications through multiple 2D windows on a 2D screen. Extended reality devices (including AR, VR, MR, etc.) provide users with display and interaction capabilities in 3D space. Therefore, there is an urgent need for a method to achieve multi-window effects in 3D space, further realize 3D multi-window capabilities, improve user experience with extended reality devices, and facilitate the development of 3D content in non-immersive mode. Summary of the invention

[0005] The present invention is used to solve the problem in the prior art that an extended reality device can only display the window of one application, so that a user cannot interact with multiple applications at the same time. Therefore, the window display method of the existing extended reality device has limited user interaction methods and poor user experience.

[0006] In order to solve the above technical problems, the present invention provides a multi-window processing method in a three-dimensional space, comprising:

[0007] Step S101: receiving attribute information of one or more windows, wherein the windows include two-dimensional windows and / or three-dimensional windows;

[0008] Step S103: Generate a corresponding transformation matrix according to the attribute information of one or more windows;

[0009] Step S105: Process one or more windows based on the transformation matrix.

[0010] As a further embodiment of the present invention, the receiving of the property information of one or more windows includes: receiving the property information of the window generated by the event from the one or more windows.

[0011] As a further embodiment of the present invention, step S105 includes:

[0012] According to the transformation matrix, events from the three-dimensional space are transformed into one or more windows and sent to the corresponding windows.

[0013] As a further embodiment of the present invention, processing one or more windows based on the transformation matrix further includes:

[0014] One or more windows generate corresponding window attribute information according to their own logical processing events;

[0015] Synthesizes the attribute information of windows generated by one or more windows.

[0016] As a further embodiment of the present invention, the attribute information of the window includes a coordinate system, and / or a position, and / or a rotation, and / or a size.

[0017] As a further embodiment of the present invention, the window attribute information includes window decoration information, which is used to control the window.

[0018] As a further embodiment of the present invention, the attribute information of the window includes display information, and / or interaction information, and / or self-logic triggering information.

[0019] As a further embodiment of the present invention, the display information includes information generated when a window is created or a window attribute is changed, or information generated when triggered by a change in user viewing angle.

[0020] As a further embodiment of the present invention, the processing of one or more windows based on the transformation matrix includes:

[0021] Determine the projection transformation matrix based on the window attribute information and the user's viewing angle posture information;

[0022] One or more windows are processed based on the projection transformation matrix.

[0023] As a further embodiment of the present invention, step S105 includes:

[0024] Sending the projection transformation matrix to the corresponding window;

[0025] The corresponding window uses a projection transformation matrix to render the scene inside the window to obtain a binocular two-dimensional image;

[0026] According to the latest window attribute information of the corresponding window and the latest binocular two-dimensional image corresponding to the corresponding window, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

[0027] As a further embodiment of the present invention, the step S105 further includes:

[0028] The window generates rendering instructions according to application logic;

[0029] According to the attribute information of the latest window of the corresponding window and the rendering instruction, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

[0030] As a further embodiment of the present invention, the step S105 further includes:

[0031] Sending the projection transformation matrix to the corresponding window;

[0032] The corresponding window generates rendering instructions using a projection transformation matrix;

[0033] According to the attribute information of the latest window of the corresponding window and the rendering instruction, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

[0034] As a further embodiment of the present invention, before performing synthetic rendering, decorative content is added based on the window decoration information, and synthetic rendering is completed to obtain and display the binocular image in the three-dimensional space.

[0035] As a further embodiment of the present invention, the interaction information is generated when a window interaction event is generated by user input.

[0036] As a further embodiment of the present invention, step S105 includes:

[0037] Generate the interaction information based on the window interaction event and the attribute information of the window;

[0038] A target window coordinate system is generated based on the interaction information and the window interaction event, and sent to the target window.

[0039] As a further embodiment of the present invention, the target window includes an immersive mode and a window mode.

[0040] As a further embodiment of the present invention, self-logic trigger information is generated when the application's own logic changes.

[0041] As a further embodiment of the present invention, the self-logic trigger information is used to modify the transformation matrix, indicate switching between immersive and window modes, or trigger window decoration information.

[0042] As a further embodiment of the present invention, when the target window enters the immersive mode, the subspace corresponding to the target window is converted into the three-dimensional space.

[0043] As a further embodiment of the present invention, other windows remain unchanged, so that the content of the target window is superimposed on the content of other windows.

[0044] As a further embodiment of the present invention, when the attribute information of the window exceeding the range of the target window is received, the interaction information is converted to the target window coordinate system, and the converted interaction information and interaction events are sent to the target window for processing.

[0045] As a further embodiment of the present invention, the attribute information of the window is updated based on user operation or application's own logic trigger.

[0046] A second aspect of the present invention provides a multi-window processing system in a three-dimensional space, comprising:

[0047] A receiving unit, configured to receive attribute information of one or more windows, wherein the windows include two-dimensional windows and / or three-dimensional windows;

[0048] A transformation matrix generating unit, which is used to generate a corresponding transformation matrix according to the attribute information of one or more windows;

[0049] The processing unit is configured to process one or more windows based on the transformation matrix.

[0050] A third aspect of the present invention provides an extended reality device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements instructions of the method described in any of the foregoing embodiments when executing the computer program.

[0051] A fourth aspect of the present invention provides a multi-window processing system in a three-dimensional space, such as the extended reality device and server as claimed in claim 24, wherein the server includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the instructions of the method described in any of the foregoing embodiments when executing the computer program.

[0052] A fifth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein when the computer program is run by a processor of an extended reality device, the computer program executes instructions of the method described in any of the aforementioned embodiments.

[0053] The multi-window processing method, device and system in a three-dimensional space provided by the present invention receive attribute information of one or more windows, wherein the windows include two-dimensional windows and / or three-dimensional windows; generate corresponding transformation matrices according to the attribute information of one or more windows; and process one or more windows based on the transformation matrix, wherein the processing of one or more windows based on the transformation matrix includes: displaying one or more windows based on the transformation matrix, and converting events in the three-dimensional space to a target window based on the transformation matrix, and sending them to the target window. Based on the transformation matrix and the user's observation angle, the synthesized display comes from different applications. The present invention can realize the simultaneous display of one or more two-dimensional windows and three-dimensional windows in a three-dimensional environment, thereby allowing users to interact with these windows at the same time, improving the application scenarios of extended reality devices, and enhancing user experience.

[0054] In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1A A flowchart of a multi-window processing method in a three-dimensional space according to an embodiment of the present invention is shown;

[0057] Figure 1B A first specific flow chart of a multi-window processing method in a three-dimensional space according to an embodiment of the present invention is shown;

[0058] Figure 2 A flowchart of a process for determining a projection transformation matrix according to an embodiment of the present invention is shown;

[0059] Figure 3 A second specific flow chart of the multi-window processing method in three-dimensional space according to an embodiment of the present invention is shown;

[0060] Figure 4A A schematic diagram of a three-dimensional window and a two-dimensional window coordinate system according to an embodiment of the present invention is shown;

[0061] Figure 4BA schematic diagram showing a multi-window display in a three-dimensional space in an extended reality device according to an embodiment of the present invention is shown;

[0062] Figure 5 A third specific flow chart of the multi-window processing method in three-dimensional space according to an embodiment of the present invention is shown;

[0063] Figure 6 A fourth specific flow chart of the multi-window processing method in three-dimensional space according to an embodiment of the present invention is shown;

[0064] Figure 7 A fifth specific flow chart of the multi-window processing method in three-dimensional space according to an embodiment of the present invention is shown;

[0065] Figure 8 A sixth specific flow chart of the multi-window processing method in a three-dimensional space according to an embodiment of the present invention is shown;

[0066] Fig. 9 A flowchart of a multi-window processing method on a three-dimensional multi-window system side according to an embodiment of the present invention is shown;

[0067] Fig.10 A flowchart of a multi-window processing method on the application side according to an embodiment of the present invention is shown;

[0068] Fig.11 A structural diagram showing a multi-window processing device in a three-dimensional space on a three-dimensional multi-window system side according to an embodiment of the present invention;

[0069] Fig.12 A schematic diagram showing the structure of a multi-window processing device in a three-dimensional space on the application side of an embodiment of the present invention is shown;

[0070] Fig.13 A structural diagram of an extended reality device according to an embodiment of the present invention is shown.

[0071] Description of the accompanying symbols:

[0072] 401. Window decoration information

[0073] 1101, first interaction unit;

[0074] 1102, projection transformation matrix calculation unit;

[0075] 1103, second interaction unit;

[0076] 1104, third interaction unit;

[0077] 1105. Synthesis unit;

[0078] 1201, window attribute information sending unit;

[0079] 1202, projection transformation matrix receiving unit;

[0080] 1203, interactive event processing unit;

[0081] 1204, rendering unit;

[0082] 1205, rendering related information sending unit;

[0083] 1302. Extended reality devices;

[0084] 1304. Processor;

[0085] 1306. Memory;

[0086] 1308, driving mechanism;

[0087] 1310, input / output module;

[0088] 1312. Input devices;

[0089] 1314. Output device;

[0090] 1316. Presentation equipment;

[0091] 1318. Graphical user interface;

[0092] 1320, network interface;

[0093] 1322, communication link;

[0094] 1324. Communication bus. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0096] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0097] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0098] In existing extended reality devices, one three-dimensional application occupies the entire space, and users can only interact with one three-dimensional application at a time. In fact, users have the need to use multiple three-dimensional applications at the same time in the extended reality device, or to use a mixture of two-dimensional and three-dimensional applications, or to use multiple three-dimensional windows created by one or more applications at the same time, or to use a mixture of two-dimensional and three-dimensional windows (for example, when watching a three-dimensional video or playing a three-dimensional game, the user needs to reply to an instant message; the user needs to view multiple windows at the same time to select a target window, etc.). The system of the existing extended reality device cannot interact with multiple three-dimensional applications at the same time or with a three-dimensional application and a two-dimensional application at the same time. Therefore, the existing extended reality device has limitations in usage scenarios when in use, cannot meet the user's needs in usage scenarios, and has a poor user experience for the user.

[0099] In order to solve the above problems existing in the prior art, in one embodiment of the present invention, a multi-window processing method in a three-dimensional space is provided, such as Figure 1A As shown, including:

[0100] Step S101: receiving attribute information of one or more windows, wherein the windows include two-dimensional windows and / or three-dimensional windows;

[0101] Step S103: Generate a corresponding transformation matrix according to the attribute information of one or more windows;

[0102] Step S105: Process one or more windows based on the transformation matrix.

[0103] Specifically, a 3D window is a subspace in a 3D space that can be displayed and interacted with. Specifically, the subspace (i.e., the 3D window) has the following characteristics:

[0104] (1) The subspace is a 360-degree space or a three-dimensional space with its own independent coordinate system;

[0105] (2) A subspace can be embedded into a space through a transformation matrix. For example, a subspace defined by a window is embedded into the entire three-dimensional space through a transformation matrix.

[0106] (3) The boundary of the subspace display / interaction can be set manually, for example, as a rectangle (two-dimensional window), a cuboid (three-dimensional window), or determined according to window attribute information, such as the projection transformation matrix (three-dimensional window) determined by the window size to determine the subspace display / interaction boundary; the boundary of the subspace display / interaction can also be determined by the object content contained in the subspace, for example, the place where there is a window corresponding to the object content is the area where the subspace can display / interact.

[0107] The above-mentioned two-dimensional window and three-dimensional window can be displayed on the extended reality device, and the two-dimensional window and the three-dimensional window can come from the same application or different applications, that is, different processes. These applications can be installed locally on the extended reality device, or installed on the access device of the extended reality device, or installed on the server, or installed on a cloud virtual machine.

[0108] The two-dimensional window described in the present invention is the same as the window of the existing intelligent terminal and computer, but it also has the position, rotation and scaling properties in the three-dimensional space. The application (window) and the three-dimensional multi-window system in the extended reality device can communicate using a message transmission interface, such as IPC communication.

[0109] The transformation matrix generated in the above step S103 can be generated by the three-dimensional multi-window system, and can also be provided by the window. When the transformation matrix is ​​provided by the window, it can also be used as a parameter in the attribute information of the window.

[0110] The transformation matrix generated in step S103 can realize the conversion of data between the world coordinate system and the window coordinate system. The transformation matrix is ​​updated based on user operations, system events or application logic triggers.

[0111] The above step S105 can realize the combined display of multiple two-dimensional windows and / or three-dimensional windows, and realize the interaction with each window based on the combined display of multiple windows. The present invention can improve the application scenarios of the extended reality device and enhance the user experience.

[0112] In one embodiment of the present invention, in the above step S101, receiving the property information of one or more windows includes: receiving the property information of the window generated by the event from one or more windows. Specifically, the event of the window can be triggered when the logic of the window application itself changes, and can also be triggered by the control device.

[0113] In one embodiment of the present invention, step S105 includes:

[0114] According to the transformation matrix, events from the three-dimensional space are transformed into one or more windows and sent to the corresponding windows.

[0115] In one embodiment of the present invention, the step S105 further includes:

[0116] One or more windows generate corresponding window attribute information according to their own logical processing events;

[0117] Synthesizes the attribute information of windows generated by one or more windows.

[0118] In this embodiment, the attribute information of the window includes at least display information, wherein the display information includes: image information (a 2D image of a 2D window and a 3D scene of a 3D window) or rendering instructions (for guiding the generation of a rendered image).

[0119] In a specific implementation, synthesizing the attribute information of windows generated by one or more windows includes: determining a projection transformation matrix based on a transformation matrix and user binocular perspective posture information; and synthesizing and rendering display information in the attribute information of windows generated by one or more windows based on the projection transformation matrix to obtain and display a binocular picture.

[0120] In one embodiment of the present invention, the attribute information of the window includes at least one of a coordinate system, a position, a rotation and a size.

[0121] In one embodiment of the present invention, the window attribute information further includes: window decoration information, wherein the window decoration information includes a UI control for controlling the window. The specific window decoration information is added to the corresponding position of the composite picture after the above-mentioned synthesis step.

[0122] In one embodiment of the present invention, the attribute information of the window also includes: display information, and / or interaction information, and / or self-logic trigger information. Among them, the display information includes information generated when the window is created or the window attributes are transformed, or when the user's perspective changes. Specifically, the display information is a two-dimensional picture to be displayed. For a three-dimensional window, its two-dimensional picture is determined based on the projection transformation matrix. The display information can also be a rendering instruction. The interaction information is generated when the user input generates a window interaction event. The self-logic trigger information is generated when the application's own logic changes. The self-logic trigger information is used to modify the transformation matrix, indicate switching between immersive and window modes, or trigger window decoration information.

[0123] In one embodiment of the present invention, processing one or more windows based on the transformation matrix includes:

[0124] Determine the projection transformation matrix based on the window attribute information and the user's binocular view posture information;

[0125] Based on the projection transformation matrix, one or more windows are processed, that is, one or more windows are merged and rendered to obtain a binocular picture, and the binocular picture is displayed.

[0126] In a specific implementation of the present invention, processing one or more windows based on the projection transformation matrix includes:

[0127] Sending the projection transformation matrix to the corresponding window;

[0128] The corresponding window uses a projection transformation matrix to render the scene inside the window to obtain a binocular two-dimensional image;

[0129] According to the attribute information of the latest window of the corresponding window and the binocular two-dimensional image corresponding to the corresponding window, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

[0130] In a specific implementation of the present invention, processing one or more windows based on the projection transformation matrix includes:

[0131] The window generates rendering instructions based on the application logic;

[0132] According to the attribute information of the latest window of the corresponding window and the rendering instruction, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

[0133] In a specific implementation of the present invention, processing one or more windows based on the projection transformation matrix includes:

[0134] Sending the projection transformation matrix to the corresponding window;

[0135] The corresponding window generates rendering instructions using a projection transformation matrix;

[0136] According to the attribute information of the latest window of the corresponding window and the rendering instruction, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

[0137] Furthermore, before synthetic rendering, decorative content is added based on the window decoration information, and synthetic rendering is completed to obtain and display the binocular image in the three-dimensional space.

[0138] In one embodiment of the present invention, the above-mentioned step S105 also includes: generating intersection information based on the window interaction event and the attribute information of the window; generating a target window coordinate system based on the intersection information and the window interaction event, and sending it to the target window, and the application of the target window responds to the window interaction event.

[0139] The process of determining the intersection information may be referred to in subsequent embodiments and will not be described in detail here.

[0140] For 3D windows, there are immersive mode and window mode. In immersive mode, users can view content from all angles under the window. In window mode, content within the window range can be displayed, or it is not limited to the size of the window. In order to increase interest or completeness, objects beyond the window range can be displayed, and users can interact with the content displayed in each mode.

[0141] In an embodiment of the present invention, when the target window enters the immersive mode, the subspace corresponding to the target window is converted into the three-dimensional space.

[0142] Furthermore, in some implementations, other windows remain unchanged, so that the content of the target window is superimposed on the content of other windows.

[0143] In one embodiment of the present invention, based on the immersive mode superimposed multi-window display, it also includes:

[0144] receiving at least one interaction event input by a user;

[0145] For each interaction event, if the interaction event includes pointing information, the window pointed to by the pointing information and the intersection information are determined, and the window to which the pixels within the pointing range of the pointing information belong is taken as the target window;

[0146] The intersection information is converted to the target window coordinate system, and the converted intersection information and interaction events are sent to the target window application for processing.

[0147] This embodiment can implement interaction between the content displayed in the immersive mode and the content displayed in the superimposed windows on the basis of superimposing multiple windows in the immersive mode, thereby improving the user experience.

[0148] In one embodiment of the present invention, when the attribute information of the window beyond the range of the target window is received, the interaction information is converted to the target window coordinate system, and the converted interaction information and interaction events are sent to the target window for processing.

[0149] In one embodiment of the present invention, a multi-window processing system in a three-dimensional space is further provided, comprising:

[0150] A receiving unit, configured to receive attribute information of one or more windows, wherein the windows include two-dimensional windows and / or three-dimensional windows;

[0151] A transformation matrix generating unit, which is used to generate a corresponding transformation matrix according to the attribute information of one or more windows;

[0152] The processing unit is configured to process one or more windows based on the transformation matrix.

[0153] In one embodiment of the present invention, a multi-window processing system in a three-dimensional space is also provided, including the extended reality device and a server described in the aforementioned embodiment, the server including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the instructions of the method described in any of the aforementioned embodiments when executing the computer program.

[0154] In one embodiment of the present invention, a multi-window processing method in a three-dimensional space is further provided, which is applied to a device supporting display of three-dimensional content or a server supporting a device supporting display of three-dimensional content, wherein an application capable of creating three-dimensional windows is connected to the three-dimensional task system, specifically, Figure 1B As shown, including:

[0155] Step 11, after the application creates a three-dimensional window or detects a change in window attributes, it sends the window attribute information to the three-dimensional multi-window system;

[0156] Step 12, the 3D multi-window system receives the window attribute information sent when the 3D window is created or when the window attribute changes, calculates the projection transformation matrix of the 3D window according to the user's binocular view posture information and the attribute information of each window, and sends it to the application of the corresponding 3D window;

[0157] Step 13, applying the projection transformation matrix of the received three-dimensional window, and rendering the three-dimensional scene inside the window into a binocular two-dimensional image or generating a rendering instruction according to the projection transformation matrix of the three-dimensional window;

[0158] Step 14, the 3D multi-window system receives a binocular 2D image or a rendering instruction; performs synthetic rendering according to the latest window attribute information of each window and the binocular 2D image or rendering instruction corresponding to each window to obtain and display the binocular picture.

[0159] This embodiment can realize the simultaneous display of multiple 3D windows provided by the same application or multiple applications in a 3D environment, thereby allowing users to interact with multiple 3D windows at the same time, improving the application scenarios of the extended reality device and enhancing the user experience. At the same time, this embodiment can also make the picture in the window change according to the user's perspective posture in the window mode, thereby improving the user experience.

[0160] In detail, step 11 is executed when the three-dimensional window is created and when the three-dimensional window attributes change. Specifically, the window creation instruction can be input by a control device connected to the extended reality device, and the control device includes but is not limited to a smart terminal, a handle, etc. The window attribute change is controlled by the application according to a preset logic or by an interactive event input by the user through the control device. Exemplarily, the window creation method is, for example, a user triggering an application in the installed application list, and the application list can be arranged vertically, horizontally, or in a circle, etc. The three-dimensional window attribute change triggering method is, for example, the user dragging the border and corners of the window. The present invention does not limit the window creation method and the three-dimensional window attribute change triggering method.

[0161] In some implementations, the attribute information of the window includes basic information such as window position, window size, window rotation, window dimension type, and window canvas information. Among them, the window position is represented by the three-dimensional coordinates of the center position of the window, and the window size is used to limit the three-dimensional size of the window. Window rotation refers to the rotation angle of the window, which is a basic attribute of the window in three-dimensional space and is used to guide the synthetic rendering of two-dimensional binocular images. The window dimension type is used to distinguish whether the window is a two-dimensional window or a three-dimensional window. The window canvas information is used to carry the result of the window rendering the three-dimensional scene into a two-dimensional binocular image, which is used for the synthesis of the final system.

[0162] Step 12 is performed when the user's binocular posture information changes and when new attribute information of the window is received. The user's binocular perspective posture information includes binocular position and binocular rotation data. In some implementations, the user's binocular perspective posture information can be measured by a user posture detection device on an extended reality device. Taking an XR device as an example, the IMU data in the XR device can provide head rotation data, and the binocular position data can be calculated from the binocular information relative to the head. In other implementations, the user's binocular perspective information can be calculated based on the user's mobile position information using a SLAM algorithm.

[0163] The projection transformation matrix refers to one projection matrix for each eye, such as Figure 2 As shown, the specific implementation steps of step 12 include:

[0164] Step 201, using the user's binocular view posture information as the starting point of the visual frustum represented by the projection transformation matrix;

[0165] Step 202, determining the orientation of the viewing frustum according to the window position in the window attribute information and the starting position of the viewing frustum;

[0166] Step 203, determining the near plane and far plane information of the viewing frustum according to the window size in the window attribute information;

[0167] Step 204, determining the projection transformation matrix of the three-dimensional window according to the starting point position, orientation, near plane and far plane of the viewing frustum.

[0168] Among them, the near plane and far plane of the viewing frustum are used to determine the visible range of the three-dimensional scene inside the window. The content beyond the window range is outside the viewing frustum and will not be rendered into the window screen.

[0169] The binocular two-dimensional image in step 13 is generated when the picture in the three-dimensional window changes or the projection transformation matrix changes, that is, as long as the picture in the three-dimensional window changes or the projection transformation matrix changes, the three-dimensional window will generate a binocular two-dimensional image or a rendering instruction and send it to the three-dimensional multi-window system. Specifically, the binocular two-dimensional image refers to an image for each of the left and right eyes. The specific implementation steps of rendering the three-dimensional scene inside the window based on the projection transformation matrix can refer to the existing process of rendering the three-dimensional scene into a two-dimensional image, which will not be described in detail here.

[0170] In step 13, generating rendering instructions according to the projection transformation matrix of the three-dimensional window includes: determining the three-dimensional content in the three-dimensional window according to the projection transformation matrix, that is, determining the three-dimensional object within the viewing frustum represented by the projection transformation matrix; generating rendering instructions according to the three-dimensional content in the three-dimensional window.

[0171] In detail, the 3D content in the 3D window can be directly generated by the application or obtained by reading and parsing the file. The 3D content can be described by traditional 3D model methods such as geometry and material, read from 3D model file formats such as fbx and obj, or described by other methods such as NeRF. The 3D content file can also be other custom file formats. The application parses the 3D content and converts it into metadata of the rendering object, where the metadata includes model geometry information, text content, material and other description information of the rendering object, as well as attribute information such as position, rotation, and scale.

[0172] In specific implementation, in order to provide users with a more complete and richer experience, the application can choose to specify that the display range is not determined based on the window size, but the range of the subspace represented by the window is determined based on the content of the rendered object to be displayed in the window. For example, the space occupied by all objects in the window belongs to the range of the subspace, and the current interaction event is determined based on the interaction with the objects in the window. Which window is interacting with. In this case, the system will send the user's projection transformation matrix (regardless of the window size) to the application, and the application can display content outside the window size range to achieve a more flexible application effect. For example, if the application contains a tree and a bird's nest, the bird can fly out of the nest and fly freely in a larger range in the space. To achieve this effect, the application can specify the use of the rendering object content to determine the range of the subspace represented by the window, thereby allowing the bird to fly freely in a larger range.

[0173] Step 14 can unify the image of the window coordinate system to the world coordinate system. The binocular picture refers to an image for each left eye, and each window refers to a created window. During specific implementation, the picture update frequency of each window is different. In the window currently displayed by the extended reality device, if the window attributes change, steps 11 to 13 will be executed. If the user's binocular perspective changes, steps 12 to 13 will be executed. Then, according to the attribute information of the current latest window of the changed window and the attribute information of the historical latest windows of other unchanged windows, the binocular two-dimensional image or rendering instruction corresponding to the changed window, and the binocular two-dimensional image or rendering instruction corresponding to other unchanged windows, synthetic rendering is performed to obtain the binocular picture.

[0174] This embodiment can ensure the correctness of projecting the three-dimensional scene in each window onto the two-dimensional plane by calculating the projection transformation matrix of each three-dimensional window, so that the user can see the correct display effect.

[0175] In a further embodiment of the present invention, the 3D multi-window system is also connected to an application that can create a 2D window, wherein the application that creates the 2D window and the application that creates the 3D window can be the same application, that is, an application can create both a 2D window and a 3D window. Figure 3 As shown, it also includes:

[0176] Step 301, the application sends the attribute information of the window to the 3D multi-window system when the 2D window is created or the window attribute changes; the 2D image is sent when the picture changes in the 2D window;

[0177] Step 302, the 3D multi-window system receives the attribute information and 2D picture of the 2D window; sets the 2D image sent by the 2D window as a binocular 2D image; performs synthetic rendering to obtain and display the binocular picture according to the attribute information of the latest window of each window and the binocular 2D image or rendering instruction corresponding to each window.

[0178] During implementation, step 301 and step 11 may be performed simultaneously. For two-dimensional applications, the left and right eye images are not distinguished. When performing synthetic rendering, for example Figure 4A As shown, the z-direction value of the two-dimensional window of the application is set to 0, and the two-dimensional image of the window in the application is set to a binocular two-dimensional image, so that the two-dimensional image can be synthetically rendered using step 14. Figure 4A In the figure, the left side is the coordinate system of the 3D window, and the right side is the coordinate system of the 2D window.

[0179] When the window attributes change, the image in the window will also change, that is, when receiving the window attribute information, the two-dimensional image will also be received.

[0180] When there is a change in the screen in the window of the extended reality device, such as a change in the three-dimensional window, execute steps 12 to 14 to update the window interface; if the two-dimensional window changes, execute step 14 to update the window interface.

[0181] For example, if Figure 4B As shown, Figure 4B A 2D window A is displayed for video communication, and two 3D windows B and C are displayed. 3D window B plays 3D video (the 3D effect is not shown in the figure, and the specific experience is similar to the effect of wearing glasses to watch 3D movies), and 3D window C displays a 3D model, supporting 360° observation and interaction. In specific implementation, users can also customize the background of the window interface.

[0182] This embodiment can realize the simultaneous display of a two-dimensional window and a three-dimensional window, thereby improving the user experience.

[0183] In a further embodiment of the present invention, on the basis of displaying multiple three-dimensional windows and / or two-dimensional windows, each three-dimensional window can also be controlled by the user, specifically, Figure 5 As shown, the three-dimensional multi-window system also performs the following steps:

[0184] Step 501, receiving at least one interaction event input by a user;

[0185] Step 502, for each interaction event, if the interaction event includes pointing information, determining a target window and intersection information between the direction in the pointing information and the target window according to the pointing information;

[0186] Step 503: convert the intersection information between the direction in the pointing information and the target window from the world coordinate system to the target window coordinate system, and send the converted intersection information and interaction event to the application of the target window for processing.

[0187] During implementation, for each interaction event, if the interaction event does not include pointing information, the currently activated window is used as the target window, and the interaction event is sent to the target window.

[0188] In step 501, the user may input window interaction events by operating a control device such as a handle, gesture, voice, eye movement, head control, mouse and keyboard.

[0189] Some interactive events input by control devices contain pointing information, such as handles, gestures, eye movements, head control, mice, etc. The pointing information can be regarded as a ray, which uses the 3Dof or 6Dof information (including position and rotation) of the control device to start from the position of the control device and point to an infinite distance. The direction of the ray is determined according to the posture of the control device (rotation). Alternatively, the pointing information is also used to determine the interactive object through direct touch, such as using a finger to touch a button. Some interactive events input by control devices do not contain pointing information, such as voice, keyboard, etc. The interactive events will be sent to the currently active window, where the active window is determined by: a newly created window or a window that the user has recently actively interacted with.

[0190] Interaction events also include command events, which include but are not limited to zooming in and out of windows, moving window positions, rotating windows, entering windows, closing windows, creating new windows, etc.

[0191] When implementing step 502, the target window and intersection information can be determined using a three-dimensional ray detection mechanism or collision detection method. Specifically, it includes: first, based on the pointing information and the position information of the displayed windows in the three-dimensional multi-window system, an intersection calculation is performed to determine the window that the pointing passes through and the intersection information between the direction in the pointing information and the target window; when the pointing passes through multiple windows, the window that first passes through is used as the target window. When the pointing does not pass through the window, the target window cannot be determined, and the user interaction event will not be sent to any application to which the window belongs, but some global operations can be performed by the system logic, such as calling out the system menu. The specific implementation process of the intersection calculation can refer to the prior art and will not be described in detail here.

[0192] When one or more users use multiple directional control devices at the same time, such as two controllers, multiple rays will point to one or more windows. In specific implementation, the ray-related interaction events will be sent to the ray-related target windows. For example, if the left hand's ray points to window A, the left hand's key events will be sent to window A. If the right hand's ray points to window B, the right hand's key events will be sent to window B.

[0193] Through step 503, the target window can directly respond when receiving the interaction event. Specifically, the scene content is modified according to the application logic triggered by the interaction event.

[0194] During implementation, the user can switch between the window state and the three-dimensional scene inside the window by moving the position (for example, moving the head, hand, body, etc.). When the interactive event is entering the window, the immersive state is entered, that is, the window size can be considered infinite. At this time, the user can view all the contents of 360° inside the target window. When the user moves into a certain window, that is, the binocular position is within the window range, the near plane and far plane of the window projection transformation matrix are set to the same value as the global projection transformation matrix, that is, when the user moves into a certain window, the complete scene in this window can be seen, which is equivalent to the effect that this window is in immersive mode, and the scene in this window occupies the entire field of view of the user.

[0195] like Figure 6 As shown, when the target window is a three-dimensional window and the interaction event is to enter the immersive mode of the target window, the multi-window processing process in the three-dimensional space includes:

[0196] Step 601, the 3D multi-window system calculates the projection transformation matrix of the target window (i.e., the global projection transformation matrix) according to the user's binocular view posture information, and sends it to the application of the target window;

[0197] Step 602: The target window application renders the three-dimensional scene according to the received projection transformation matrix to obtain a binocular two-dimensional image or generates a rendering instruction, and sends the binocular two-dimensional image or the rendering instruction to the three-dimensional multi-window system;

[0198] Step 603: the 3D multi-window system renders and displays a binocular picture according to the attribute information of the target window and the binocular 2D image or rendering instruction corresponding to the target window.

[0199] This embodiment can realize entering the immersive state from the window state.

[0200] In a further embodiment, after the target window enters the immersive state, the above steps 11 to 14 or steps 301 to 302 are further executed.

[0201] This embodiment can ensure that after the user is in an immersed state, the user can trigger the display of multiple windows, thereby meeting the user's need to operate multiple windows in an immersive state. For example, when the user is watching a three-dimensional movie, he needs to reply to a message. At this time, the user can create a communication window, thereby realizing the realization of watching the movie and replying to the message at the same time.

[0202] In one embodiment of the present invention, Figure 7 As shown, based on the immersive mode superimposed multi-window display, the multi-window processing method in the three-dimensional space also includes:

[0203] Step 701: The three-dimensional multi-window system receives at least one interaction event input by a user;

[0204] For each interaction event, if the interaction event includes pointing information, the window to which the pixels within the pointing range of the pointing information belong and the intersection information are determined. The intersection information can also be used to determine the interaction object by direct touch, such as using a finger to touch a button, and the window to which the pixels within the pointing range of the pointing information belong is used as the target window; the intersection information is converted to the target window coordinate system according to the transformation matrix, and the converted intersection information and interaction event are sent to the target window application for processing;

[0205] For each interaction event, if the interaction event does not include pointing information, the currently activated window is used as the target window; and the interaction event is sent to the target window.

[0206] Step 702: The application executes corresponding logic according to the intersection information and the interaction event. When the display screen changes, the above steps 12 to 14, or steps 301 to 302 are executed.

[0207] When exiting the immersive state, entering the multi-window display mode, the above steps 11 to 14 can be used to realize window merging display, and the user can also control and view the internal image of the 3D window.

[0208] In one embodiment of the present invention, in the above step 13, when the 3D multi-window system receives a rendering instruction generated by a 3D window using a projection transformation matrix, and the rendering instruction contains object information that exceeds the range specified by the window size attribute, such as Figure 8 As shown, after merging and displaying, it also includes:

[0209] Step 801, the three-dimensional multi-window system receives at least one interaction event input by the user; for each interaction event, if the interaction event contains pointing information, the window to which the first object pointed to by the pointing information belongs and the intersection information are determined, and the intersection information can also be used to determine the interaction object by direct touch, such as using a finger to touch a button and using the window to which the pointing object belongs as the target window; the intersection information is converted to the target window coordinate system according to the transformation matrix, and the converted intersection information and interaction event are sent to the target window for processing.

[0210] Step 802: The application executes corresponding logic according to the intersection information and the interaction event. When the display screen changes, the above steps 12 to 14, or steps 301 to 302 are executed.

[0211] This embodiment enables objects displayed in each window to interact with each other, further improving the user's interactive experience.

[0212] In one embodiment of the present invention, in order to facilitate the user to control the window, decoration information is also set on the window. Specifically, when the above step 11 is implemented, the window decoration information is also received. The window decoration information includes UI controls for realizing the control of the window. The UI controls include but are not limited to zoom-in controls, zoom-out controls, and close controls. In specific implementation, the decoration information may also include a title bar, etc.

[0213] After step 14 renders the binocular image, it also includes: adding the window decoration information to the binocular image according to the attribute information of the window, and the adding result is as follows: Figure 4B As shown in window decoration information 401.

[0214] During implementation, the window decoration information may include not only UI controls but also animation display controls, so that the decoration displays animation effects and blends in with the background to enhance interest.

[0215] In one embodiment of the present invention, in the above step 12, for the projection transformation matrix of each three-dimensional window, before the three-dimensional multi-window system sends the projection transformation matrix of the three-dimensional window to the three-dimensional window, the following steps are further performed: judging whether the three-dimensional window is within the user's field of view according to the attribute information of the three-dimensional window and the user's binocular perspective posture information, if not, not sending the projection transformation matrix to the three-dimensional window, if yes, sending the projection transformation matrix to the three-dimensional window; or

[0216] In the above step 12, before the three-dimensional multi-window system sends the projection transformation matrix of the three-dimensional window based on the user's binocular perspective posture information and the attribute information of each window, it also performs the following steps: for the attribute information of each window, it is determined whether the three-dimensional window is within the user's field of view based on the attribute information of the window and the user's binocular perspective posture information; if so, the projection transformation matrix of the three-dimensional window is calculated.

[0217] Among them, the visible range of the window interface can be considered as the viewing frustum range represented by the global projection transformation matrix. The global projection transformation matrix and the user's binocular perspective posture information are combined to determine the user's field of view, that is, which windows can be seen by the user.

[0218] This embodiment can save the performance of application window rendering and system rendering and improve the efficiency of picture synthesis.

[0219] In one embodiment of the present invention, after receiving a rendering instruction sent by a three-dimensional window application, the three-dimensional multi-window system first converts the position information relative to the window coordinate system in the rendering instruction into the world coordinate system, and then performs synthetic rendering on the windows based on the latest converted rendering instructions of all windows and the latest window attribute information of each window to obtain a binocular image.

[0220] During implementation, due to the different update frequencies of the window images, when one of the window images within the user's viewing angle changes, the synthetic rendering process is performed.

[0221] Window creation and window screen changes can be triggered when the window responds to user interaction events.

[0222] In some implementations, the system rendering and application window rendering methods can also be used in combination. Specifically, the above step 14 is to perform synthetic rendering to obtain and display the binocular picture based on the window's latest binocular two-dimensional image and its latest window attribute information, as well as the window's latest rendering instructions and its latest window attribute information.

[0223] In some implementations, the rendering instructions of the window include: metadata of the rendering object. The metadata includes description information and attribute information of the rendering object, such as geometric information (shape, size, etc.), text content, and material information, and the attribute information includes position, rotation, scaling, etc. The metadata can also be a rendering object model, and the system can generate a rendering object image based on the rendering object model. During implementation, the attribute information can be specified by the user to achieve interaction with the window and enrich the user experience.

[0224] In some implementations, metadata may also include a motion model, which specifies the motion law of the object. For example, if the object is a butterfly, the butterfly motion model defines the motion law of the butterfly flying. The 3D multi-window system can calculate the incremental metadata and incremental posture information of the object in real time according to the motion model, and after completing the coordinate system conversion, merge it with the content in other windows to render a dynamic binocular picture.

[0225] This method can reduce the amount of data transmission between the application and the system and improve the efficiency of screen generation.

[0226] In some embodiments, the user interaction event corresponds to a series of adjustment operations. After the application window performs the adjustment operation, a rendering instruction including incremental information and / or posture incremental information is generated to the system based on the change information before and after the adjustment. The system performs merged rendering based on the received rendering instruction and the user's binocular perspective posture information.

[0227] In one embodiment of the present invention, a multi-window processing method applied to a three-dimensional multi-window system is also provided. Fig. 9 As shown, including:

[0228] Step 901, receiving window attribute information sent when a 3D window is created or when window attributes are changed;

[0229] Step 902, calculating the projection transformation matrix of the three-dimensional window according to the user's binocular view posture information and the attribute information of each window, and sending it to the application of the corresponding three-dimensional window;

[0230] Step 903, receiving a binocular two-dimensional image obtained by rendering the three-dimensional scene inside the window using the projection transformation matrix; or receiving a rendering instruction generated by the application using the projection transformation matrix;

[0231] Step 904, based on the latest window attribute information of each window and the binocular two-dimensional image or rendering instruction corresponding to each window, perform synthesis rendering to obtain and display the binocular picture. In one embodiment of the present invention, a multi-window processing method applied to the application side is also provided, such as Fig.10 As shown, including:

[0232] Step 1001, after creating a 3D window or detecting a change in window attributes, sending the window attribute information to a 3D multi-window system;

[0233] Step 1002, receiving a projection transformation matrix of a three-dimensional window sent by a three-dimensional multi-window system, wherein the projection transformation matrix of the three-dimensional window is calculated by the three-dimensional multi-window system according to binocular perspective posture information of the user and attribute information of the window;

[0234] Step 1003, according to the projection transformation matrix of the three-dimensional window, the three-dimensional scene inside the window is rendered into a binocular two-dimensional image or a rendering instruction is generated;

[0235] Step 1004, sending the binocular two-dimensional image or rendering instruction image of the window to the three-dimensional multi-window system, so that the three-dimensional multi-window system performs synthesis rendering according to the latest attribute information of each window and the latest binocular two-dimensional image or rendering instruction of each window to obtain and display the binocular picture.

[0236] The window attribute information in step 1001 includes: window position, window size, rotation, window dimension type and window canvas information. The initial value of the window attribute information is specified by the application and can be set arbitrarily. It can be adjusted by the system or the user later.

[0237] The specific implementation process of step 1003 of rendering the three-dimensional scene into a two-dimensional image may refer to the prior art, and the present invention is not limited to this.

[0238] The present invention determines the projection transformation matrix by the system according to the attribute information of the window provided by the application and the user's binocular perspective posture information. The application renders the three-dimensional system in the window into a binocular two-dimensional image or generates a rendering instruction according to the projection transformation matrix. The system synthesizes and renders the binocular two-dimensional images or rendering instructions of all windows to obtain a binocular picture, so that multiple three-dimensional windows can be displayed in an extended reality device, providing users with more interactive scenes and improving user experience.

[0239] Based on the same inventive concept, the present invention also provides a multi-window processing device in a three-dimensional space, as described in the following embodiments. Since the principle of solving the problem by the multi-window processing device in a three-dimensional space is similar to that of the multi-window processing method in a three-dimensional space, the implementation of the multi-window processing device in a three-dimensional space can refer to the multi-window processing method in a three-dimensional space, and the repeated parts will not be repeated.

[0240] Specifically, Fig.11 As shown, a multi-window processing device in a three-dimensional space applied to a three-dimensional multi-window system includes:

[0241] The first interaction unit 1101 is used to receive the property information of the window sent when the 3D window is created or when the window property changes;

[0242] The projection transformation matrix calculation unit 1102 is used to calculate the projection transformation matrix of the three-dimensional window according to the user's binocular perspective posture information and the attribute information of each window, and send it to the application of the three-dimensional window;

[0243] The second interaction unit 1103 is used to receive a binocular two-dimensional image obtained by rendering the three-dimensional scene inside the window by the application using the projection transformation matrix; or receive a rendering instruction generated by the application using the projection transformation matrix;

[0244] The third interaction unit 1104 receives the user's interaction event, determines the interaction target application of the interaction event, and sends the interaction event to the target application.

[0245] The synthesis unit 1105 is used to perform synthesis rendering to obtain and display a binocular picture according to the latest window attribute information of each window and the binocular two-dimensional image or rendering instruction corresponding to each window.

[0246] like Fig.12 As shown, a multi-window processing device in a three-dimensional space applied to a three-dimensional window application includes:

[0247] The window attribute information sending unit 1201 is used to send the window attribute information to the 3D multi-window system after creating the 3D window or detecting the change of the window attribute;

[0248] A projection transformation matrix receiving unit 1202 is used to receive a projection transformation matrix of a three-dimensional window sent by the three-dimensional multi-window system, wherein the projection transformation matrix of the three-dimensional window is calculated by the three-dimensional multi-window system according to the user's binocular perspective posture information and the attribute information of the window;

[0249] The interactive event processing unit 1203 is used to receive interactive events for the application distributed by the system and execute corresponding application logic for processing.

[0250] A rendering unit 1204 is used to render the three-dimensional scene inside the window into a binocular two-dimensional image or generate a rendering instruction according to the projection transformation matrix of the three-dimensional window;

[0251] The rendering related information sending unit 1205 is used to send the binocular two-dimensional image or rendering instruction of the window to the three-dimensional multi-window system, so that the three-dimensional multi-window system performs synthetic rendering according to the latest attribute information of each window and the binocular two-dimensional image or rendering instruction corresponding to each window to obtain and display the binocular picture.

[0252] Through the implementation of the technical solution of the present invention, the three-dimensional space displayed by the extended reality device can be divided into multiple subspaces (each window), and the subspaces can be independent of each other or interact with each other. For example, by triggering a button in window A, the change of the scene in window B can be controlled, and the linkage effect between windows can be determined by the internal logic of the application. Users can interact with the scenes in each subspace respectively, thereby improving the flexibility of using the extended reality device and enhancing the user experience.

[0253] In one embodiment of the present invention, an extended reality device is also provided. Fig.13 As shown, the extended reality device 902 may include one or more processors 1304, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The extended reality device 1302 may also include any memory 1306, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 1306 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the extended reality device 1302. In one case, when the processor 1304 executes an associated instruction stored in any memory or a combination of memories, the extended reality device 1302 may perform any operation of the associated instruction. The extended reality device 1302 also includes one or more drive mechanisms 1308 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0254] The extended reality device 1302 may also include an input / output module 1310 (I / O) for receiving various inputs (via input device 1312) and for providing various outputs (via output device 1314). A specific output mechanism may include a presentation device 1316 and an associated graphical user interface (GUI) 1318. In other embodiments, the input / output module 1310 (I / O), input device 1312, and output device 1314 may not be included, and the device may be used as only an extended reality device in a network. The extended reality device 1302 may also include one or more network interfaces 1320 for exchanging data with other devices via one or more communication links 1322. One or more communication buses 1324 couple the components described above together.

[0255] The communication link 1322 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1322 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0256] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0257] An embodiment of the present invention further provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the method shown in any of the aforementioned embodiments.

[0258] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0259] It should also be understood that in the embodiments of the present invention, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0260] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0261] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0262] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0263] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0264] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0265] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0266] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-window processing method in a three-dimensional space, characterized in that: include: Step S101: receiving attribute information of one or more windows, wherein the windows include two-dimensional windows and three-dimensional windows, or the windows include three-dimensional windows, and the windows are from the same application or different applications; the three-dimensional window is a subspace in a three-dimensional space capable of display and interaction, and the subspace has an independent coordinate system; Step S103: Generate a corresponding transformation matrix according to the attribute information of one or more windows; The subspace can be embedded into a three-dimensional space through the transformation matrix; Step S105: Process one or more windows based on the transformation matrix.

2. The multi-window processing method in three-dimensional space as claimed in claim 1, characterized in that: The receiving the property information of one or more windows includes: receiving the property information of the window generated by the event from the one or more windows.

3. The multi-window processing method in three-dimensional space as claimed in claim 1, characterized in that: The step S105 comprises: According to the transformation matrix, events from the three-dimensional space are transformed into one or more windows and sent to the corresponding windows.

4. The multi-window processing method in three-dimensional space as claimed in claim 1 or 3, characterized in that: Based on the transformation matrix, one or more windows are processed, and the process also includes: One or more windows generate corresponding window attribute information according to their own logical processing events; Synthesizes the attribute information of windows generated by one or more windows.

5. The multi-window processing method in three-dimensional space as claimed in claim 1, characterized in that: The property information of the window includes a coordinate system, and / or a position, and / or a rotation, and / or a size.

6. The multi-window processing method in three-dimensional space as claimed in claim 5, characterized in that: The window attribute information includes window decoration information, which is used to control the window.

7. The multi-window processing method in three-dimensional space as claimed in claim 6, characterized in that: The attribute information of the window includes display information, and / or interaction information, and / or self-logic trigger information.

8. The multi-window processing method in three-dimensional space as claimed in claim 7, characterized in that: The display information includes information generated when a window is created or a window attribute is changed, or information generated when triggered by a change in user viewing angle.

9. The multi-window processing method in three-dimensional space as claimed in claim 1, characterized in that: The processing of one or more windows based on the transformation matrix includes: Determine the projection transformation matrix based on the window attribute information and the user's viewing angle posture information; One or more windows are processed based on the projection transformation matrix.

10. The multi-window processing method in three-dimensional space as claimed in claim 9, characterized in that: The step S105 comprises: Sending the projection transformation matrix to the corresponding window; The corresponding window uses a projection transformation matrix to render the scene inside the window to obtain a binocular two-dimensional image; According to the latest window attribute information of the corresponding window and the latest binocular two-dimensional image corresponding to the corresponding window, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

11. The multi-window processing method in three-dimensional space as claimed in claim 1, characterized in that: The step S105 further includes: The window generates rendering instructions according to application logic; According to the attribute information of the latest window of the corresponding window and the rendering instruction, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

12. The multi-window processing method in three-dimensional space as claimed in claim 9 or 11, characterized in that: The step S105 further includes: Send the projection transformation matrix to the corresponding window; The corresponding window generates rendering instructions using a projection transformation matrix; According to the attribute information of the latest window of the corresponding window and the rendering instruction, synthetic rendering is performed to obtain and display the binocular picture of the three-dimensional space.

13. The multi-window processing method in three-dimensional space according to any one of claims 10 to 12, characterized in that: Before performing synthetic rendering, decorative content is added based on the window decoration information, and synthetic rendering is completed to obtain and display the binocular image in the three-dimensional space.

14. The multi-window processing method in three-dimensional space as claimed in claim 7, characterized in that: The interaction information is generated when a window interaction event is generated by user input.

15. The multi-window processing method in three-dimensional space as claimed in claim 14, characterized in that: The step S105 comprises: Generate the interaction information based on the window interaction event and the attribute information of the window; A target window coordinate system is generated based on the interaction information and the window interaction event, and sent to the target window.

16. The multi-window processing method in three-dimensional space as claimed in claim 15, characterized in that: The target window includes an immersive mode and a window mode.

17. The multi-window processing method in three-dimensional space as claimed in claim 6, characterized in that: Generates its own logic trigger information when the application's own logic changes.

18. The multi-window processing method in three-dimensional space as claimed in claim 17, characterized in that: The self-logic trigger information is used to modify the transformation matrix, indicate switching between immersive and window modes, or trigger window decoration information.

19. The multi-window processing method in three-dimensional space as claimed in claim 16, characterized in that: When the target window enters the immersive mode, the subspace corresponding to the target window is converted into the three-dimensional space.

20. The multi-window processing method in three-dimensional space as claimed in claim 19, characterized in that: The other windows remain unchanged, so that the content of the target window can be overlaid on the content of other windows.

21. The multi-window processing method in three-dimensional space as claimed in claim 19, characterized in that: When the property information of the window beyond the range of the target window is received, the interaction information is converted to the target window coordinate system, and the converted interaction information and interaction events are sent to the target window for processing.

22. The multi-window processing method in three-dimensional space as claimed in claim 1, characterized in that: The attribute information of the window is updated based on user operations or application logic triggers.

23. A multi-window processing system in three-dimensional space, characterized in that: include: A receiving unit, configured to receive attribute information of one or more windows, wherein the windows include two-dimensional windows and three-dimensional windows, or the windows include three-dimensional windows, and the windows are from the same application or different applications; the three-dimensional window is a subspace in a three-dimensional space capable of display and interaction, and the subspace has an independent coordinate system; A transformation matrix generating unit, which is used to generate a corresponding transformation matrix according to the attribute information of one or more windows, and the subspace can be embedded into a three-dimensional space through the transformation matrix; The processing unit is configured to process one or more windows based on the transformation matrix.

24. An extended reality device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the instructions of the method according to any one of claims 1 to 22.

25. A multi-window processing system in three-dimensional space, characterized in that: include: The extended reality device and server as claimed in claim 24, wherein the server comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the instructions of the method described in any one of claims 1 to 22 when executing the computer program.

26. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of an extended reality device, the computer program executes instructions of the method according to any one of claims 1 to 22.

Images