Method, device and application for implementing real-time editing based on extended virtual window
By creating an unreal window corresponding to the viewport of the virtual space in the virtual engine, displaying and rendering the virtual screen content in real time, the problem of frequently switching to the virtual space to view after modifying the content in the virtual engine is solved, real-time editing and viewing of the virtual screen content is realized, and design efficiency is improved.
Patent Information
- Application Number
- CN202210783384.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-05
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2042-07-05
AI Technical Summary
When designing the virtual screen content of a virtual space, the map data, object materials or design blueprint modified in the virtual engine need to be clicked to run before viewing the rendered screen in the virtual space, resulting in designers having to frequently switch between the virtual engine and the virtual space, which is inefficient.
By creating an unreal window corresponding to the viewport of the virtual space in the virtual engine, the virtual screen content is displayed and rendered in real time, and the viewport rendering data is used to realize real-time editing and viewing of the virtual screen content.
Real-time editing and viewing of virtual screen content is realized, reducing the number of switching between the virtual engine and the virtual space by designers, and improving design efficiency.
Smart Images
Figure CN115049773B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual engines, and in particular to a method, device and application for implementing real-time editing based on an extended virtual window. Background Art
[0002] Naked-eye fully immersive mixed reality products refer to products that allow users to experience fully immersed virtual images in real physical space without wearing any auxiliary equipment. In order to achieve the above effects, it is often necessary to configure projection equipment to present a fully immersive virtual space with real scale and real proportion in real physical space, and the image content of the virtual space is realized through a powerful virtual engine behind it.
[0003] However, there is a fatal problem in designing virtual screen content in virtual space: when modifying virtual screen content such as map data, object materials or design blueprints in the virtual engine, it is necessary to click Run in the virtual engine before viewing the rendered screen in the virtual space. However, the virtual space and the virtual engine are often separated by a distance, which results in designers having to return to the virtual space to view the modified content every time they modify the content on the virtual engine, resulting in very low efficiency of the entire design process. Summary of the invention
[0004] The embodiments of the present application provide a method, device and application for achieving real-time editing based on an extended virtual window. By extending the virtual window in a virtual engine, the virtual window can be used as a real-time response device to achieve the effect of real-time viewing of virtual screen content, thereby improving the design efficiency of virtual screen content.
[0005] In a first aspect, an embodiment of the present application provides a method for achieving real-time editing based on an extended virtual window, the method comprising: creating at least one virtual window in a virtual engine that corresponds one-to-one to a viewport of a virtual space; displaying the virtual screen content corresponding to each viewport in the corresponding virtual window, and passing in the corresponding viewport rendering data, wherein the viewport rendering data includes at least a rendering mode, a viewport orientation, and a viewport matrix; the virtual engine renders the viewport of the corresponding virtual space based on the viewport rendering data and the virtual screen content, and projects it onto the corresponding viewport.
[0006] In the second aspect, an embodiment of the present application provides a device for realizing real-time editing in a virtual space based on an extended virtual window, comprising: a virtual window extension unit, for creating at least one virtual window corresponding to a viewport of the virtual space in a virtual engine; an editing unit, for displaying the virtual screen content corresponding to each viewport in the corresponding virtual window, and transmitting the corresponding viewport rendering data, wherein the viewport rendering data includes at least a rendering mode, a viewport orientation, and a viewport matrix; a rendering unit, for enabling the virtual engine to render the viewport of the corresponding virtual space based on the viewport rendering data and the virtual screen content, and project it onto the corresponding viewport.
[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the method described in the computer program to implement real-time editing in a virtual space based on an extended virtual window.
[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a method for implementing real-time editing in a virtual space based on an extended virtual window.
[0009] The main contributions and innovations of the present invention are as follows:
[0010] The embodiment of the present application achieves the effect of real-time editing and real-time viewing by introducing a virtual window corresponding to the real viewport of each virtual space in the virtual engine, and using viewport rendering data to convert the editing screen content into virtual screen content that can be displayed in the virtual window. The operator can edit the virtual screen content such as map object materials or blueprints in real time in the virtual engine without repeatedly switching back and forth between the server controlled by the virtual engine and the virtual space; it solves the problem of only being able to passively accept data in the virtual space but not being able to actively modify and timely view the content.
[0011] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0013] Figure 1 is a schematic diagram of multiple viewports of a virtual space provided in an embodiment of the present application;
[0014] Figure 2is a flow chart of a method for implementing real-time editing based on an extended virtual window according to an embodiment of the present application;
[0015] Figure 3 is a structural block diagram of a device for implementing real-time editing in a virtual space based on an extended virtual window according to an embodiment of the present application;
[0016] Figure 4 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0018] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0019] Embodiment 1
[0020] Virtual space is a space for projecting virtual screen content. In an embodiment of the present solution, the virtual space includes multiple viewports, and the virtual screen content corresponding to each viewport is projected into the corresponding viewport to form a fully immersive virtual space. The present solution aims to realize real-time editing of virtual screen content in the virtual space.
[0021] refer to Figure 3 The method for implementing real-time editing based on extended virtual window provided by this solution includes:
[0022] Creating at least one virtual window in the virtual engine that corresponds one-to-one to the viewport of the virtual space;
[0023] Displaying the virtual screen content corresponding to each viewport in the corresponding virtual window, and passing in the corresponding viewport rendering data, wherein the viewport rendering data at least includes a rendering mode, a viewport orientation, and a viewport matrix;
[0024] The virtual engine renders a viewport of the corresponding virtual space based on the viewport rendering data and the virtual screen content, and projects it on the corresponding viewport.
[0025] It is worth mentioning that in “creating at least one virtual window in the virtual engine that corresponds one-to-one to the viewport of the virtual space”, the number of virtual windows is the same as the number of viewports of the virtual space, and each of the viewports corresponds to a virtual window.
[0026] In a specific application scenario, such as Figure 1 As shown, the virtual space includes five viewports, namely, the ground viewport, the east viewport, the south viewport, the west viewport and the north viewport. The east viewport, the south viewport, the west viewport and the north viewport are arranged in the order of the directions of east, south, west and north. The ground viewport is placed on the bottom planes of the east viewport, the south viewport, the west viewport and the north viewport to enclose and form a bowl-shaped space. The projection device can turn to different viewport directions and project the corresponding virtual screen content. At this time, five virtual windows corresponding to the ground viewport, the east viewport, the south viewport, the west viewport and the north viewport are created in the virtual engine.
[0027] The editing of the virtual engine can be done in the virtual engine editor. The virtual window is directly displayed in the virtual space, and the designer can edit the virtual engine in the virtual space and get real-time feedback on the virtual window.
[0028] In the step of "displaying the virtual screen content corresponding to each viewport in the corresponding virtual window", the screen content is edited in the editor of the virtual engine, the screen content is converted into virtual screen content, and the virtual screen content is filled into the corresponding virtual window using the rendering interface.
[0029] It is worth noting that the operator can edit the virtual screen content in the virtual engine editor, including but not limited to replacing, modifying, deleting or adding any map data, object materials or blueprint-type virtual screen content. The corresponding virtual screen content will be displayed in the Unreal window so that the operator can see the modified screen in the virtual engine.
[0030] It is particularly important to note that the virtual screen content is content that has been rendered by viewport rendering data, so that the operator can achieve the effect of playing in the virtual space in the editor of the virtual engine. In some embodiments, the virtual screen content is filled into the virtual window through a rendering interface.
[0031] Different from the scene that needs to wear VR or AR equipment, in the virtual space, the viewport of the virtual space and the observation angle of the observer in the virtual space are unbound, and the observer can move freely in the virtual space. The virtual screen content displayed in the virtual space is visually changed according to the coordinate position of the observer. Specifically, when the observer is close to a certain viewport, the virtual screen content projected by the viewport should be wider; when the observer is far away from the certain viewport, the virtual screen content projected by the viewport will be narrower, so that the observer can feel the three-dimensional sense of space in the virtual space, and the change of the virtual screen content needs to be controlled according to the viewport matrix passed in by this solution.
[0032] In the step of generating virtual screen content, not only the viewport matrix of each viewport affects the rendering result, but also the orientation and rendering mode of each viewport. Therefore, this solution will pass in the corresponding viewport rendering data when rendering each virtual window. The viewport rendering data at least includes the rendering mode, viewport orientation and viewport matrix.
[0033] The rendering mode is the way the virtual engine renders the viewport, which can be selected as Perspective projection; the viewport orientation is the orientation of the rendered viewport, such as the viewport orientation of the east viewport mentioned above is east, so as to distinguish and determine each viewport; the viewport matrix is the perspective transformation information related to the user's observation coordinates. The viewport matrix is automatically created through the user's observation coordinate information in the virtual space and the viewport window size. The viewport matrix is shown below:
[0034]
[0035] It should be noted that the matrix elements in the viewport matrix are automatically calculated based on parameters such as the viewport window size and observation coordinate information. Therefore, the value of a single matrix element is only an example and has no actual meaning.
[0036] Therefore, the virtual engine needs to obtain the user's observation coordinate information, and calculate the viewport matrix of the current viewport based on the viewport window size of the current viewport and the observation coordinate information.
[0037] In the step of "displaying the virtual screen content corresponding to each viewport in the corresponding virtual window and passing the corresponding viewport rendering data", the user's observation coordinate information is obtained and combined with the viewport corresponding to the current virtual window to obtain the viewport matrix, and the viewport matrix is used to process the editing screen content to obtain the virtual screen content.
[0038] It is also worth mentioning that, in order for the virtual engine to process the editing screen content according to the viewport matrix to obtain the virtual screen content, this solution first needs to reload the rendering class of the virtual engine's editor and then pass in the viewport matrix.
[0039] In the step of "the virtual engine renders the viewport of the corresponding virtual space based on the viewport rendering data and the virtual screen content, and projects it onto the corresponding viewport", the viewport to be rendered is determined based on the viewport rendering data, and the virtual screen content is projected onto the corresponding viewport.
[0040] In an embodiment of the present scheme, the virtual space contains five viewports, namely, the ground viewport, the east viewport, the south viewport, the west viewport and the north viewport. The corresponding optical and mechanical equipment can face five directions, namely, due east, due south, due west, due north and the ground, respectively, to project the virtual picture content onto the corresponding viewports.
[0041] In addition, if the size of the viewport window and the Unreal window is inconsistent with the size of the Unreal Engine Editor window, the viewport matrix and the Unreal window cannot be adapted. That is to say, in the Unreal Engine Editor, the viewport matrix is automatically generated according to the size of the corresponding viewport window, and if the size of the viewport window and the Unreal Engine Editor window is inconsistent, the virtual screen content will be distorted.
[0042] In order to solve this problem, in the step of "displaying the virtual screen content corresponding to each viewport in the corresponding virtual window", the virtual engine creates a virtual window with the same viewport size ratio as the viewport, uses the viewport matrix to process the edited screen content to obtain the virtual screen content and renders it into the virtual window, and processes the screen of the virtual window according to the size ratio of the window of the virtual engine editor and fills it into the window.
[0043] In this solution, the virtual engine editor is provided with a function of refreshing the size ratio of the virtual screen content to the size ratio of the window of the virtual engine editor. Since the virtual screen content is generated according to the viewport matrix, when the virtual engine projects the virtual screen content onto the corresponding viewport, the screen content matching the viewport size can still be obtained.
[0044] For example, assume that the ground viewport is a square, and the window of the virtual engine editor is a rectangle. At this time, when the virtual engine renders the virtual window corresponding to the ground viewport, the virtual engine creates a 3840*3840 virtual window, uses the viewport matrix to process the editing screen content to obtain the virtual screen content and render it into the virtual window, so that the content rendered to the ground viewport is square, and the virtual screen content is compressed to a 3840*2160 format and filled into the window of the virtual engine editor. In this way, when the virtual engine performs projection restoration in the virtual space, the pixels of the ground are compressed in the 3840*2160 virtual window, but can be restored to a square area during projection.
[0045] This solution uses the Unreal Engine as a real-time responder of the virtual engine by expanding the Unreal Window, so that the virtual screen content can be viewed in real time on the editor of the virtual engine. In addition, this solution can also make full use of the powerful editing function of the virtual engine to complete the 3D display of the complex editing process.
[0046] Since this solution can realize real-time 3D display, it has very good application in some specific scenarios. For example, in the project of small parts assembly in the field of education, through this solution, the teacher only needs to select a small object in the virtual engine through the mouse and keyboard. By using the built-in functions of Unreal, the selected object will have different special effects, and the movement control is also very precise and convenient. It can enhance the students' understanding of the operation method.
[0047] Embodiment 2
[0048] Based on the same idea, refer to Figure 3 The present application also proposes a device for implementing real-time editing in a virtual space based on an extended virtual window, comprising:
[0049] A virtual window extension unit, used to create at least one virtual window in a virtual engine that corresponds one-to-one to a viewport of the virtual space;
[0050] An editing unit, used to display the virtual screen content corresponding to each viewport in the corresponding virtual window, and to input the corresponding viewport rendering data, wherein the viewport rendering data at least includes a rendering mode, a viewport orientation, and a viewport matrix;
[0051] A rendering unit is used to enable the virtual engine to render the viewport of the corresponding virtual space based on the viewport rendering data and the virtual picture content, and project it on the corresponding viewport.
[0052] The device of this solution for implementing real-time editing in a virtual space based on an extended virtual window can be implemented as a plug-in in a virtual engine. For details of the technical contents of this embodiment 2 that are the same as those of embodiment 1, please refer to the introduction of embodiment 1, and no redundant explanation will be given here.
[0053] Embodiment 3
[0054] This embodiment also provides an electronic device, referring to Figure 4 , including a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute any of the steps in the above-mentioned embodiment of the method for implementing real-time editing in a virtual space based on an extended virtual window.
[0055] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0056] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 404 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory 404 (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.
[0057] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .
[0058] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the methods for implementing real-time editing in a virtual space based on an extended virtual window in the above embodiments.
[0059] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .
[0060] The transmission device 406 can be used to receive or send data via a network. The specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0061] The input and output device 408 is used for editing content, etc. The output information may be virtual screen content on the extended window, content projected on the viewport, etc.
[0062] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:
[0063] Creating at least one virtual window in the virtual engine that corresponds one-to-one to the viewport of the virtual space;
[0064] Displaying the virtual screen content corresponding to each viewport in the corresponding virtual window, and passing in the corresponding viewport rendering data, wherein the viewport rendering data at least includes a rendering mode, a viewport orientation, and a viewport matrix;
[0065] The virtual engine renders a viewport of the corresponding virtual space based on the viewport rendering data and the virtual screen content, and projects it on the corresponding viewport.
[0066] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0067] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0068] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or implemented by hardware, or implemented by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros can be stored in any device readable data storage medium, and they include program instructions for performing specific tasks. Computer program products can include one or more computer executable components configured to perform embodiments when the program is running. One or more computer executable components can be at least one software code or a part thereof. In addition, at this point, it should be noted that any box of the logic flow in the figure can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored in physical media such as memory chips or storage blocks implemented in processors, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and data variants thereof, CDs. Physical media are non-transient media.
[0069] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0070] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for implementing real-time editing based on an extended virtual window, characterized in that: include: Creating at least one virtual window in the virtual engine that corresponds one-to-one to the viewport of the virtual space; Displaying the virtual screen content corresponding to each viewport in the corresponding virtual window, and passing in the corresponding viewport rendering data, wherein the viewport rendering data at least includes a rendering mode, a viewport orientation, and a viewport matrix, wherein the rendering mode is a mode in which the virtual engine renders the viewport, the viewport orientation is an orientation of the rendered viewport, and the viewport matrix is perspective transformation information related to the user's observation coordinates, and the viewport matrix is automatically created according to the user's observation coordinate information in the virtual space and the viewport window size; The viewport to be rendered is determined based on the viewport rendering data, and the virtual screen content is projected onto the corresponding viewport. The operator edits the virtual screen content in the editor of the virtual engine, including replacing, modifying, deleting or adding any map data, object materials or blueprint-type virtual screen content. The corresponding virtual screen content will be displayed in the Unreal window.
2. The method for implementing real-time editing based on an extended virtual window according to claim 1, characterized in that: Each of the viewports corresponds to a virtual window.
3. The method for implementing real-time editing based on an extended virtual window according to claim 1, characterized in that: In the step of "displaying the virtual screen content corresponding to each viewport in the corresponding virtual window and passing the corresponding viewport rendering data", the user's observation coordinate information is obtained and combined with the viewport corresponding to the current virtual window to obtain the viewport matrix, and the viewport matrix is used to process the editing screen content to obtain the virtual screen content.
4. The method for implementing real-time editing based on an extended virtual window according to claim 1, characterized in that: The screen content is edited in the editor of the virtual engine, the edited screen content is converted into virtual screen content, and the virtual screen content is filled into the corresponding virtual window using a rendering interface.
5. The method for implementing real-time editing based on an extended virtual window according to claim 1, characterized in that: In the step of "displaying the virtual screen content corresponding to each viewport in the corresponding virtual window", the virtual engine creates an intermediate window with the same viewport size ratio as the viewport, uses the viewport matrix to process the edited screen content to obtain the virtual screen content and renders it into the intermediate window, and fills the screen of the intermediate window into the virtual window according to the size ratio of the virtual window.
6. A device for implementing real-time editing in a virtual space based on an extended virtual window, characterized in that: include: A virtual window extension unit, used to create at least one virtual window in a virtual engine that corresponds one-to-one to a viewport of the virtual space; An editing unit, used to display the virtual screen content corresponding to each viewport in the corresponding virtual window, and to input the corresponding viewport rendering data, wherein the viewport rendering data at least includes a rendering mode, a viewport orientation, and a viewport matrix, wherein the rendering mode is a mode in which the virtual engine renders the viewport, the viewport orientation is an orientation of the rendered viewport, and the viewport matrix is perspective transformation information related to the user's observation coordinates, and the viewport matrix is automatically created according to the user's observation coordinate information in the virtual space and the viewport window size; The rendering unit determines the viewport to be rendered based on the viewport rendering data, and projects the virtual screen content onto the corresponding viewport; the operator edits the virtual screen content in the editor of the virtual engine, including replacing, modifying, deleting or adding any map data, object material or blueprint-type virtual screen content, and the corresponding virtual screen content will be displayed in the unreal window.
7. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the method for implementing real-time editing in a virtual space based on an extended virtual window as described in any one of claims 1 to 5.
8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes a method for implementing real-time editing in a virtual space based on an extended virtual window according to any one of claims 1 to 5.