Transparent window interaction method, system and device based on DirectX and storage medium

Abstract

The invention discloses a DirectX-based transparent window interaction method, a DirectX-based transparent window interaction system, DirectX-based transparent window interaction equipment and a storage medium. The method comprises the following steps: integrating a DirectX API (Application Program Interface) and a DComp API; detecting an operation environment, and selecting to use DirectX11 or DirectX12 in combination with the detected operation environment; initializing an API (Application Program Interface); creating a transparent window; finishing rendering of display content of the transparent window by utilizing a DirectX11 or DirectX12 API (Application Program Interface), and performing parallel processing on different rendering tasks in combination with a UE (User Equipment) multi-channel rendering technology; the DComp API is synchronously utilized to complete transparency adjustment of the display content of the transparent window; processing multi-channel transparent synthesis rendering based on the DComp and the UE by using the self-defined hybrid model; the HOOK technology is adopted to intercept a mouse interaction event, a mouse interaction event judgment function is called, whether penetration processing is carried out or not is judged, penetration processing is correspondingly executed according to the judgment result, and real-time interaction is completed through a bottom layer operation system. According to the method and the device, the mouse event can penetrate to the bottom layer of the desktop while the high-quality rendering content is reserved, so that the immersion and the interaction fluency are improved.

Description

Technical Field

[0001] The present application relates to the field of computer graphics and human-computer interaction technology, and in particular to a DirectX-based transparent window interaction method, system, device and storage medium. Background Art

[0002] The fields of computer graphics and virtual reality have made great progress in recent years, especially in terms of image quality and interactive experience. With the widespread application of virtual reality in multiple industries such as games, education, and medical care, users have higher and higher requirements for immersion and authenticity. However, traditional window management technology faces many challenges in achieving transparent interaction, which limits the deep integration of the virtual and real worlds.

[0003] In order to solve the display problem of transparent windows, there are currently two common methods: one is to achieve it through software-level transparent windows, and the other is to use hardware acceleration technology. Software-level transparent windows usually rely on the API provided by the operating system, such as the SetLayeredWindowAttributes function of the Windows system, which can achieve a certain degree of window transparency. However, this method is limited by the performance and functions of the operating system and cannot maintain good performance under high load conditions. Another method is to achieve a higher level of transparency through hardware acceleration technology, such as DirectX and other graphics interfaces. Although this type of method can provide better performance, it often requires developers to have high programming skills and has poor compatibility on different hardware platforms.

[0004] Although the above methods can achieve the effect of transparent windows to a certain extent, there are still obvious deficiencies in practical applications. Traditional technical solutions often bring significant performance losses while realizing transparent windows, especially when dealing with complex image synthesis and multi-threaded tasks. In addition, these solutions are difficult to adapt to different hardware platforms, resulting in poor cross-platform support. Therefore, how to make mouse events penetrate to the bottom layer of the desktop while retaining the rendering content, and realize the realistic interaction of transparent windows without obvious performance loss, has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to retain high-quality rendering content while allowing mouse events to penetrate to the bottom layer of the desktop, thereby improving the immersion and interaction fluency in virtual reality and augmented reality scenes, the present application provides a transparent window interaction method, system, device and storage medium based on DirectX.

[0006] In a first aspect, the present application provides a transparent window interaction method based on DirectX, comprising: Integrate DirectX API and DComp API; the DirectX includes DirectX11 or DirectX12; Check the operating environment and choose to use DirectX11 or DirectX12 based on the detected operating environment; Initialize the DirectX11 or DirectX12 API and DComp API to be used; Create a transparent window, including: determine the transparent window display content, specify the transparent window style and initialize the window transparency; Obtain the rendering requirements of the transparent window display content and generate rendering tasks, use the DirectX11 or DirectX12 API selected after initialization to complete the rendering of the transparent window display content, and combine UE multi-channel rendering technology to process different rendering tasks in parallel; synchronously use DComp API to complete the transparency adjustment of the transparent window display content; customize the hybrid model to process multi-channel transparent composite rendering based on DComp and UE; HOOK technology is used to intercept mouse interaction events and call the mouse interaction event judgment function to determine whether to perform penetration processing and execute penetration processing accordingly based on the judgment result, and use the underlying operating system to complete real-time interaction.

[0007] By adopting the above solution, integrating and utilizing DirectX API and DComp API, transparent windows can retain high-quality rendering content while having the ability to penetrate mouse events, thereby improving the immersion and interaction fluency in virtual reality and augmented reality applications. At the same time, UE multi-channel rendering technology and a custom hybrid model are used to process multi-channel transparent composite rendering, which can efficiently schedule resources in a multi-threaded environment and ensure the stable output of complex visual effects.

[0008] Preferably, the parallel processing of different rendering tasks in combination with the UE multi-channel rendering technology also includes: Use UE's built-in performance analysis technology to monitor the rendering process data of each channel in real time, including: frame rate, GPU utilization, and memory usage; Dynamically adjust the Swap Chain parameters and buffer allocation strategy based on the rendering process data of each channel monitored in real time, including: when the frame rate is detected to be lower than the preset first frame rate threshold, increase the swap interval of the Swap Chain; when the memory occupancy is detected to be higher than the first preset occupancy threshold, reduce the buffer size; when the GPU utilization is detected to be lower than the preset first GPU utilization threshold, reduce the number of buffers.

[0009] By adopting the above solution, the UE's built-in performance analysis technology is used to monitor the rendering process data of each channel in real time, promptly identify potential performance bottlenecks, and dynamically adjust the Swap Chain parameters and buffer allocation strategies to effectively respond to changes in performance requirements in different scenarios, ensuring stable output in complex visual effects and multi-threaded environments.

[0010] Preferably, the parallel processing of different rendering tasks using UE multi-channel rendering technology also includes: In a multi-GPU configuration, monitor the load and data transfer requirements of each GPU in real time; Based on the real-time monitoring of the load and data transmission requirements of each GPU, a machine learning algorithm is used to predict the PCIe channel bandwidth requirements of each GPU. Dynamically allocate bandwidth resources of PCIe channels according to predicted demands of PCIe channel bandwidth of each GPU; including: evaluating the bandwidth resource status of the current PCIe channel, including total bandwidth, allocated bandwidth and available bandwidth; calculating the bandwidth increment or decrement required by each GPU according to predicted demands of PCIe channel bandwidth of each GPU and current bandwidth resource status, combined with preset priorities of each GPU; Combined with DMA technology, the rendering data is transferred to the GPU corresponding to the allocated PCIe channel through the dynamically allocated PCIe channel.

[0011] By adopting the above solution, the load and data transmission requirements of each GPU are monitored in real time, and the machine learning algorithm is used to predict the demand of each GPU for PCIe channel bandwidth. The bandwidth resource allocation is dynamically adjusted in multi-task parallel processing, and efficient data transmission and resource scheduling are achieved under multi-GPU configuration, improving rendering performance and system response speed. DMA technology is used to efficiently transfer rendering data to the corresponding GPU, further improving the speed and efficiency of data processing, and ensuring stable output under complex visual effects and multi-threaded environments.

[0012] Preferably, the parallel processing of different rendering tasks using UE multi-channel rendering technology also includes: For the rendering process of each channel, decompose each frame rendering task into multiple subtasks, analyze the dependency of each subtask, build a task dependency graph, determine the execution order of each subtask and whether parallel execution is allowed; Setting a scheduling strategy, wherein the scheduling strategy includes giving priority to scheduling subtasks that have no dependencies or whose dependencies have been satisfied; Based on the asynchronous computing capabilities of the CPU and GPU, subtasks with no dependencies or whose dependencies have been satisfied will be transferred to the CPU and GPU respectively according to the set scheduling strategy to complete asynchronous computing processing.

[0013] By adopting the above solution, each frame rendering task is subdivided and its dependencies are analyzed, a task dependency graph is constructed, and the asynchronous computing capabilities of the CPU and GPU are utilized to prioritize subtasks with no dependencies or whose dependencies have been satisfied to the CPU and GPU for processing, thereby achieving higher parallelism and lower latency, and improving the efficiency and performance of transparent window rendering.

[0014] Preferably, the rendering process further includes: Writing asynchronous compute shaders that allow asynchronous execution and handle typical rendering tasks; The overall rendering task is divided into multiple subtasks, each subtask is assigned to a rendering channel and an asynchronous compute shader adapted to the current subtask, and data synchronization between different rendering channels and asynchronous compute shaders is ensured.

[0015] By adopting the above solution, writing asynchronous compute shaders and applying them to multiple subtasks, we can fully utilize the parallel computing capabilities of the GPU, improve the parallel processing capabilities of rendering tasks, and effectively reduce the blocking time of the main thread; and synchronize data between different rendering channels and asynchronous compute shaders, effectively avoiding rendering errors caused by inconsistent data, thereby ensuring image quality and stability.

[0016] Preferably, the calling of the mouse interaction event judgment function setting includes: setting the judgment function based on the mouse interaction event attribute or setting the judgment function according to the hot zone mechanism or setting the judgment function based on the intelligent prediction result of the user behavior.

[0017] By adopting the above solution, the judgment conditions of the mouse interaction events are flexibly set, so that the system can select a suitable judgment method according to different application scenarios and user needs to achieve penetrating interaction.

[0018] Preferably, it also includes: Design a transparent window manager; Register each created transparent window with the transparent manager and display it in the transparent window list of the transparent window manager with the assigned unique identifier; Associating identifiers of different transparent windows in the transparent window list of the window manager according to user needs; Add synchronization logic to the mouse interaction event judgment function based on the mouse interaction event attribute setting, and broadcast the current mouse interaction event to the transparent windows associated with all registered transparent windows when the synchronization operation is triggered; After receiving the broadcast event, each associated transparent window responds to the mouse interaction event to achieve synchronous interactive operations.

[0019] By adopting the above solution, a transparent window manager is designed and each transparent window is registered, so as to centrally manage and coordinate the display and interactive operations of multiple transparent windows. When a transparent window receives a mouse interaction event, the event is broadcast to all associated transparent windows to ensure that all related transparent windows can respond to the same event immediately, thereby achieving a consistent interactive experience across windows and realizing efficient synchronous penetration interaction between multiple transparent windows.

[0020] In a second aspect, the present application provides a DirectX-based transparent window interaction method system, comprising: An API integration module, used for integrating DirectX API and DComp API; the DirectX includes DirectX11 or DirectX12; The operating environment detection module detects the operating environment and selects DirectX11 or DirectX12 based on the detected operating environment; API initialization module, used to initialize the selected DirectX11 or DirectX12 API and DCompAPI; A transparent window creation module is used to create a transparent window, including: determining the transparent window display content, specifying the transparent window style and initializing the window transparency; The transparent window rendering operation module is used to obtain the rendering requirements of the transparent window display content and generate rendering tasks. It uses the DirectX11 or DirectX12 API selected after initialization to complete the rendering of the transparent window display content, and combines the UE multi-channel rendering technology to process different rendering tasks in parallel; synchronously uses the DComp API to complete the transparency adjustment of the transparent window display content; and uses a custom hybrid model to process multi-channel transparent composite rendering based on DComp and UE; The transparent window penetration operation module is used to intercept mouse interaction events using HOOK technology and call the mouse interaction event judgment function to determine whether to perform penetration processing and execute the penetration processing accordingly according to the judgment result, and use the underlying operating system to complete real-time interaction.

[0021] By adopting the above solution, it is possible to accurately control the penetration processing of mouse events without affecting the rendering performance, thereby improving the user's interactive experience in virtual reality and augmented reality environments.

[0022] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0023] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0024] In summary, this application has the following beneficial effects: 1. Integrate DirectX 11 and DirectX 12 APIs, and combine with Windows desktop composition API to retain high-quality rendering content while enabling mouse events to penetrate to the bottom layer of the desktop, improving the immersion and interaction fluency in virtual reality and augmented reality scenes; and use UE multi-channel rendering technology and custom blending mode to process multi-channel transparent composite rendering based on DComp and UE, improve rendering throughput, reduce main thread blocking, and ensure stable output and high performance in complex environments; 2. Monitor rendering data in real time and dynamically adjust the frame rate and buffer allocation strategy of the image exchange chain to ensure high-quality rendering effects and smooth user experience during transparent window interaction; 3. Monitor rendering data in real time, dynamically adjust PCIe channel bandwidth resource allocation, and efficiently transfer rendering data to the GPU based on PCIe channel DMA technology, reduce bandwidth bottleneck problems, improve rendering efficiency in multi-threaded environments, and ensure stable output in complex visual effects processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a DirectX-based transparent window interaction method described in a specific embodiment; Figure 2 It is a structural diagram of the DirectX-based transparent window interaction system described in a specific embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] like Figure 1 As shown, the embodiment of the present application discloses a transparent window interaction method based on DirectX, including: S1. Integrate DirectX API and DComp API.

[0028] Specifically, integrating DirectX API and DComp API is the basis of the whole method.

[0029] DirectX API can choose DirectX11 or DirectX12 to provide efficient GPU accelerated image rendering; the specific choice depends on the user's operating environment, such as: if the user's hardware supports DirectX12, DirectX12 is preferred, otherwise DirectX11 is selected.

[0030] The Windows Desktop Composition API, or DComp API, provides efficient visual effects manipulation, including transparency control, visual animation, and synthesis of multi-level visual objects, enabling dynamic adjustment of window transparency and synthesis of visual effects.

[0031] S2. Detect the operating environment and select the DirectX version.

[0032] Specifically, the system detects the operating system environment of the current user, and selects the appropriate DirectX API based on the queried operating system's GPU model, driver version, and other information, that is, checks whether the operating system supports DirectX 12, and whether the GPU driver and hardware are compatible with DirectX 12. If the requirements of DirectX 12 are met, DirectX 12 is used first, otherwise DirectX11 is selected. S3. Initialize the DirectX11 or DirectX12 API and DComp API selected for use.

[0033] S4. Create a transparent window.

[0034] Specifically, it includes: creating a transparent window frame and setting basic window properties. Determining the transparent window display content, adding the display content to the transparent window frame, and selecting the display content to be a dynamic image, a static image, a model, etc. In this embodiment, it is assumed that the transparent window display content is a virtual character model in the game. Specifying the transparent window style can create a custom window style. Using the DComp API to set the initial window transparency, complete the initialization of the window transparency.

[0035] In addition, the window style can be modified by introducing the WS_EX_LAYERED and WS_EX_TRANSPARENT extended flags.

[0036] S5. Use the selected DirectX11 or DirectX12 API to complete the rendering of the transparent window display content, and use the DComp API to dynamically adjust the transparency of the transparent window display content during the rendering process.

[0037] Specifically, the rendering requirements of the determined transparent window display content are obtained and analyzed, and a rendering task is generated. In this embodiment, the rendering task of the virtual character model in the game is mainly determined and generated based on multiple factors such as game logic, scene requirements, and user interaction. For example, in an adventure game, the player needs to control an explorer to explore a mysterious island. Different rendering requirements are determined according to different characters, different positions of the characters (starting point, task point, or user-controlled walking position), and different postures of the characters (walking posture, interactive posture, combat posture) (such as: different character positions and posture change combinations for the rendering requirements of light and shadow projection), and different rendering tasks are generated accordingly.

[0038] The rendering of the transparent window display content is completed by using the DirectX11 or DirectX12 API selected after initialization. Specifically, the rendering process includes: loading resources, including texture data, lighting data, depth data, geometry data, etc. used in the rendering process; setting up the rendering pipeline, configuring the DirectX rendering pipeline, such as vertex shader, geometry shader, pixel shader and rendering state, etc., creating and configuring buffers, such as vertex buffers, index buffers, etc., and using the rendering pipeline to render the transparent window display content.

[0039] In order to further improve rendering efficiency and response speed, UE multi-channel rendering technology is combined to process different rendering tasks in parallel. Specifically, install the latest version of the UE engine; divide the rendering tasks into multiple channels, such as: foreground channel, background channel, special effects channel, etc., configure corresponding rendering parameters for each channel, such as: resolution, color space, rendering order, etc., assign different rendering tasks to each channel, and use UE's multi-threaded rendering capabilities to achieve parallel processing between different channels; in each channel, use DirectX11 or DirectX12 API to render specific content and execute the above rendering process.

[0040] Synchronously use the DComp API to adjust the transparency of the content displayed in the transparent window. Specifically, during the rendering process of the transparent window content, the transparency, size and shape of the window may affect the user's interactive experience and the visual effects of the game. Therefore, the DComp API is used to dynamically adjust the transparency setting throughout the process to ensure that the quality of the game screen is not affected, including: creating a DComp device and a visual tree, adding the transparent window to this tree; dynamically calculating the transparency value of the window according to the game logic or user input; among them, the transparency of the content displayed in the transparent window changes dynamically according to the player's progress, environmental interaction and game logic, and different game logics (the player approaches the location of the hidden treasure in the adventure game) and user input (through specific keys on the keyboard (such as "Ctrl" + scroll wheel)) correspond to the transparency of the transparent window display content; use the DComp API to set the transparency attribute of the window according to the dynamically calculated window transparency value.

[0041] In addition, in order to further ensure stable output and high performance in complex visual effects and multi-threaded environments, a custom hybrid model is used to process multi-channel transparent synthesis rendering based on DComp and UE, including: customizing the transparent synthesis method of each channel content of UE (such as: defining which independent rendering channels are merged together, and adjusting the mixing method between channels according to transparency, color or other parameters), according to the customized transparent synthesis method of each channel content, using DCompAPI to dynamically adjust the transparency of each channel, and then perform superposition and fusion of multiple channels. In this embodiment, the real-time video stream is rendered as the background, and the virtual character in the transparent window is rendered as the foreground. The transparency of different channels is adjusted so that the user can see the real-time video stream behind the virtual character.

[0042] S6. Realize interactive penetration.

[0043] Specifically, HOOK technology is used to intercept mouse interaction events, and a mouse interaction event judgment function is called according to the obtained mouse interaction events; the calling of the mouse interaction event judgment function setting includes: setting a judgment function based on mouse interaction event attributes, such as: mouse click position, mouse movement speed, mouse click frequency and other attributes, and setting whether to perform penetration processing judgment function conditions based on specific mouse interaction event attributes, such as: if the mouse click occurs in the edge area of ​​the transparent window, or the movement speed exceeds the preset speed threshold, and the mouse click frequency exceeds the preset click frequency, then it is set to perform penetration processing judgment function conditions, that is, interact with the underlying window or desktop.

[0044] Alternatively, a judgment function may be set according to a hot zone mechanism, such as defining specific hot zones in a transparent window. These hot zones are designed not to respond to the mouse, and a judgment function condition is set based on whether the position belongs to the hot zone or not. If the position belongs to the hot zone, the judgment function condition is set to perform penetration processing.

[0045] Or a judgment function is set based on the intelligent prediction results of user behavior, such as: analyzing and learning the user's historical mouse interaction behavior to determine the interaction intention corresponding to the user's mouse interaction behavior, and setting a judgment function condition for whether to perform penetration processing based on whether the interaction intention corresponding to the user's mouse interaction behavior belongs to the interaction penetration intention. If the user's mouse interaction behavior belongs to the interaction penetration intention, it is set as the judgment function condition for performing penetration processing.

[0046] By calling the mouse interaction event judgment function, it is judged whether to perform penetration processing and the penetration processing is performed accordingly according to the judgment result, and the real-time interaction is completed by using the underlying operating system.

[0047] In addition, considering that there are situations where users need to perform synchronous interactive operations on multiple transparent windows, in order to further achieve efficient synchronous interaction between multiple transparent windows, it also includes: selecting and designing a transparent window manager; registering each created transparent window with the transparent manager, and displaying it in the transparent window list of the transparent window manager with an assigned unique identifier; associating identifiers of different transparent windows in the transparent window list of the window manager according to user needs; adding synchronization logic (such as: judging to perform penetration processing) to the mouse interaction event judgment function set based on the mouse interaction event attribute setting, and when the synchronization operation is triggered, broadcasting the current mouse interaction event to the transparent windows associated with all registered transparent windows; after each associated transparent window receives the broadcast event, it responds to the mouse interaction event to achieve synchronous interactive operations.

[0048] In a specific embodiment, in order to effectively cope with performance fluctuations in different scenarios and ensure that high rendering efficiency and system stability can be maintained under complex visual effects and high load conditions, the Swap Chain parameters and buffer allocation strategy are dynamically adjusted during the rendering process to improve response speed and optimize resource utilization; therefore, the parallel processing of different rendering tasks in combination with the UE multi-channel rendering technology also includes: Use UE's built-in performance analysis technology to monitor the rendering process data of each channel in real time, including: frame rate, GPU utilization, and memory usage; According to the rendering process data of each channel monitored in real time, the Swap Chain parameters and buffer allocation strategy are dynamically adjusted; specifically, the following are included: When the frame rate is detected to be lower than the preset first frame rate threshold, the swap interval of the Swap Chain is increased; that is, when the frame rate is detected to be lower than expected, the burden of the GPU is reduced by increasing the swap interval of the Swap Chain, giving more rendering time, thereby improving the frame quality. Correspondingly, when the frame rate is detected to be higher than the preset second frame rate threshold, the swap interval of the Swap Chain is reduced. The increased or decreased Swap Chain swap interval duration can be pre-set according to the size of the frame rate exceeding the preset second frame rate threshold or being lower than the preset first frame rate threshold.

[0049] When it is monitored that the memory occupancy rate is greater than the first preset occupancy rate threshold, the buffer size is reduced; that is, when the memory is tight, the buffer size is reduced to reduce unnecessary memory occupation.

[0050] When it is monitored that the GPU utilization is lower than a preset first GPU utilization threshold, the number of buffers is reduced, that is, by reducing the number of buffers to release more memory resources for other tasks. On the contrary, if the GPU utilization is monitored to be higher than a preset second GPU utilization threshold, the number of frame buffers is increased to reduce rendering waiting time and improve rendering efficiency, wherein the increased or decreased number of frame buffers is pre-set according to the size exceeding the preset second GPU utilization threshold or the size lower than the preset first GPU utilization threshold.

[0051] In a specific embodiment, in order to improve the data transmission efficiency and rendering performance under the multi-GPU configuration and ensure the stability and compatibility of the system on various hardware platforms, the bandwidth resources of the PCIe channel can be dynamically allocated according to the load of each GPU under the multi-GPU configuration to reduce the bandwidth bottleneck and improve the integrity; and the DMA technology based on the PCIe channel is used to reduce the bandwidth bottleneck of image data transmission and realize high-performance data exchange. The method also includes the following steps for parallel processing of different rendering tasks using the UE multi-channel rendering technology: In a multi-GPU configuration, monitor each GPU's load (GPU utilization, temperature, etc.) and data transfer requirements (memory occupancy, I / O requests, etc.) in real time.

[0052] Based on the real-time monitored load conditions and data transmission requirements of each GPU, a machine learning algorithm is used to predict the demand for PCIe channel bandwidth of each GPU. Specifically, a machine learning algorithm suitable for processing time series data, such as the long short-term memory network LSTM, is used to perform feature processing on the monitored load conditions and data transmission requirements of each GPU, extract features such as load peak value and data transmission rate change, and predict the demand for PCIe channel bandwidth of each GPU in the future. The long short-term memory network LSTM is trained and generated through the historical load conditions and data transmission requirements of each GPU and the historical demand for PCIe channel bandwidth of each GPU.

[0053] Dynamically allocate bandwidth resources of PCIe channels according to predicted demands of PCIe channel bandwidth of each GPU; including: evaluating the bandwidth resource situation of the current PCIe channel, including total bandwidth, allocated bandwidth and available bandwidth; calculating the bandwidth increment or decrement required by each GPU according to predicted demands of PCIe channel bandwidth of each GPU and current bandwidth resource situation, combined with preset priorities of each GPU; calculating the bandwidth increment or decrement required by each GPU specifically includes: first satisfying the bandwidth demand of high-priority GPUs, and satisfying the predicted demand of low-priority GPUs as much as possible if available bandwidth permits; if the predicted demand of a GPU exceeds its currently allocated bandwidth and the system has sufficient available bandwidth, calculating the bandwidth increment required by the GPU; if the currently allocated bandwidth of a GPU exceeds its predicted demand, considering reducing its bandwidth allocation to release excess bandwidth resources for use by other GPUs.

[0054] Combined with DMA technology, rendering data (such as texture data) is transferred to the assigned GPU through the dynamically allocated PCIe channel, realizing direct data transfer between GPU memory and host memory. In order to further optimize the rendering process, DMA transfer parameters such as block size and transfer timing are optimized to match the data transfer requirements of the rendering task or the time window of DMA transfer is intelligently adjusted to avoid conflicts with the key rendering cycle of the main GPU and minimize bandwidth pressure, such as predicting the key rendering cycle of the main GPU based on the GPU load, and then avoiding arranging DMA transfers in the same period.

[0055] In addition, in addition to the multi-GPU configuration, a comprehensive compatibility test was conducted for different GPU architectures, display devices and operating systems, and the memory allocation strategy of the rendering pipeline was optimized. The compatibility test includes: selecting the test scope, that is, clearly testing the GPU architecture (such as NVIDIA's CUDA, AMD's RDX or Intel's integrated graphics processor, etc.), display devices (including displays with different resolutions and refresh rates, VR devices, projectors, etc.) and operating systems (different versions of Windows, macOs, and Linux); creating test cases, designing a series of test cases for rendering scenes and functions, including basic graphics drawing, complex lighting and texture processing, and physical simulation, executing test results, analyzing test results, monitoring the memory usage of the rendering pipeline under different hardware and operating systems, and if performance degradation is identified (such as memory leaks, unnecessary memory allocations, or frequent memory requests and releases), designing a dynamic memory allocation strategy to adapt to different GPU architectures and operating systems, applying the formulated memory allocation strategy to the rendering pipeline, adjusting the relevant parameters and performing compatibility testing again, verifying that the optimized memory allocation strategy meets the memory performance requirements of the test case, and then obtaining the memory allocation strategy of the rendering pipeline under different GPU architectures, display devices, and operating system conditions.

[0056] In a specific embodiment, in order to ensure that the rendering logic can effectively utilize the asynchronous computing capabilities of the CPU and GPU and achieve stable output of rendering quality in complex scenes, the relevant rendering task scheduling can be completed based on the frame-dependent task scheduling algorithm to effectively utilize the asynchronous computing capabilities of the CPU and GPU; in the specific method: the parallel processing of different rendering tasks using the UE multi-channel rendering technology also includes: For the rendering process of each channel, decompose the rendering task of each frame into multiple subtasks, such as: the rendering process is decomposed into subtasks such as geometry processing, texture loading, lighting calculation, and shadow generation; analyze the dependency of each subtask, that is, which tasks can be processed in parallel, such as: texture loading and lighting calculation can be processed in parallel, and which tasks need to be processed sequentially, such as: lighting calculation and shadow generation; build a task dependency graph to determine the execution order of each subtask and whether parallel execution is allowed.

[0057] Set a scheduling strategy, which includes giving priority to subtasks that have no dependencies or whose dependencies have been satisfied; based on the asynchronous computing capabilities of the CPU and GPU, subtasks that have no dependencies or whose dependencies have been satisfied will be transferred to the CPU and GPU respectively to complete asynchronous computing processing according to the set scheduling strategy. For example, geometry processing and texture loading tasks will be given priority, and after geometry processing and texture loading are completed, lighting calculation and shadow generation will be scheduled at the same time.

[0058] In a specific embodiment, asynchronous computing shaders are used to accelerate visual effect generation, reduce main thread blocking, and improve rendering throughput. The rendering process also includes: Write asynchronous compute shaders that allow asynchronous execution and handle typical rendering tasks; wherein the written asynchronous compute shaders run independently of the main rendering thread, are integrated into the UE's rendering pipeline, and seamlessly connect with the multi-channel rendering framework. The typical rendering tasks may be depth calculations, lighting calculations, etc.

[0059] The overall rendering task is divided into multiple subtasks, each subtask is assigned to a rendering channel and an asynchronous compute shader adapted to the current subtask, and data synchronization between different rendering channels and asynchronous compute shaders is ensured to avoid rendering errors; the asynchronous compute shader adapted to the current subtask is the asynchronous compute shader that can be used by the current rendering subtask to complete rendering.

[0060] like Figure 2 As shown, the embodiment of the present application discloses a transparent window interaction method system based on DirectX, including: API integration module 101, used for integrating DirectX API and DComp API; the DirectX includes DirectX11 or DirectX12; An operating environment detection module 102 detects an operating environment and selects to use DirectX11 or DirectX12 based on the detected operating environment; API initialization module 103, used to initialize the DirectX11 or DirectX12 API and DCompAPI selected for use; The transparent window creation module 104 is used to create a transparent window, including: determining the transparent window display content, specifying the transparent window style and initializing the window transparency; The transparent window rendering operation module 105 is used to obtain the rendering requirements of the transparent window display content and generate a rendering task, use the DirectX11 or DirectX12 API selected after initialization to complete the rendering of the transparent window display content, and combine the UE multi-channel rendering technology to process different rendering tasks in parallel; synchronously use the DComp API to complete the transparency adjustment of the transparent window display content; customize the hybrid model to process the multi-channel transparent synthesis rendering based on DComp and UE; The transparent window penetration operation module 106 is used to intercept mouse interaction events using HOOK technology and call the mouse interaction event judgment function to determine whether to perform penetration processing and execute the penetration processing accordingly according to the judgment result, and use the underlying operating system to complete real-time interaction.

[0061] In a specific embodiment, the transparent window rendering operation module 105 is also used to write an asynchronous computing shader that allows asynchronous execution and processing of typical rendering tasks; the overall rendering task is divided into multiple subtasks, each subtask is assigned to a rendering channel and an asynchronous computing shader adapted to the current subtask, and data synchronization between different rendering channels and asynchronous computing shaders is ensured.

[0062] In a specific embodiment, the transparent window creation module 104 is also used to design a transparent window manager; register each created transparent window in the transparent manager and display it in the transparent window list of the transparent window manager with an assigned unique identifier; associate the identifiers of different transparent windows in the transparent window list of the window manager according to user needs; add synchronization logic to the mouse interaction event judgment function based on the mouse interaction event attribute setting, and when the synchronization operation is triggered, broadcast the current mouse interaction event to the transparent windows associated with all registered transparent windows; after receiving the broadcast event, each associated transparent window responds to the mouse interaction event to achieve synchronous interaction operation.

[0063] The embodiment of the present application also discloses a computer-readable storage medium.

[0064] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above-mentioned DirectX-based transparent window interaction method. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0065] The embodiment of the present application also discloses a computer device.

[0066] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above DirectX-based transparent window interaction method.

[0067] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A transparent window interaction method based on DirectX, characterized in that: include: Integrate DirectX API and DComp API; the DirectX includes DirectX11 or DirectX12; Check the operating environment and choose to use DirectX11 or DirectX12 based on the detected operating environment; Initialize the DirectX11 or DirectX12 API and DComp API to be used; Create a transparent window, including: determine the transparent window display content, specify the transparent window style and initialize the window transparency; Obtain the rendering requirements of the transparent window display content and generate rendering tasks, use the DirectX11 or DirectX12 API selected after initialization to complete the rendering of the transparent window display content, and combine UE multi-channel rendering technology to process different rendering tasks in parallel; synchronously use DComp API to complete the transparency adjustment of the transparent window display content; customize the hybrid model to process multi-channel transparent composite rendering based on DComp and UE; HOOK technology is used to intercept mouse interaction events and call the mouse interaction event judgment function to determine whether to perform penetration processing and execute penetration processing accordingly based on the judgment result, and use the underlying operating system to complete real-time interaction.

2. The transparent window interaction method according to claim 1, characterized in that: The parallel processing of different rendering tasks in combination with the UE multi-channel rendering technology also includes: Use UE's built-in performance analysis technology to monitor the rendering process data of each channel in real time, including: frame rate, GPU utilization, and memory usage; Dynamically adjust the Swap Chain parameters and buffer allocation strategy based on the rendering process data of each channel monitored in real time, including: when the frame rate is detected to be lower than the preset first frame rate threshold, increase the swap interval of the Swap Chain; when the memory occupancy is detected to be higher than the first preset occupancy threshold, reduce the buffer size; when the GPU utilization is detected to be lower than the preset first GPU utilization threshold, reduce the number of buffers.

3. The transparent window interaction method according to claim 1, characterized in that: The parallel processing of different rendering tasks using UE multi-channel rendering technology also includes: In a multi-GPU configuration, monitor the load and data transfer requirements of each GPU in real time; Based on the real-time monitoring of the load and data transmission requirements of each GPU, a machine learning algorithm is used to predict the PCIe channel bandwidth requirements of each GPU. Dynamically allocate bandwidth resources of PCIe channels according to predicted demands of PCIe channel bandwidth of each GPU; including: evaluating the bandwidth resource status of the current PCIe channel, including total bandwidth, allocated bandwidth and available bandwidth; calculating the bandwidth increment or decrement required by each GPU according to predicted demands of PCIe channel bandwidth of each GPU and current bandwidth resource status, combined with preset priorities of each GPU; Combined with DMA technology, the rendering data is transferred to the GPU corresponding to the allocated PCIe channel through the dynamically allocated PCIe channel.

4. The transparent window interaction method according to claim 1, characterized in that: The parallel processing of different rendering tasks using UE multi-channel rendering technology also includes: For the rendering process of each channel, decompose each frame rendering task into multiple subtasks, analyze the dependency of each subtask, build a task dependency graph, determine the execution order of each subtask and whether parallel execution is allowed; Setting a scheduling strategy, wherein the scheduling strategy includes giving priority to scheduling subtasks that have no dependencies or whose dependencies have been satisfied; Based on the asynchronous computing capabilities of the CPU and GPU, subtasks with no dependencies or whose dependencies have been satisfied will be transferred to the CPU and GPU respectively according to the set scheduling strategy to complete asynchronous computing processing.

5. The transparent window interaction method according to claim 1, characterized in that: The rendering process also includes: Writing asynchronous compute shaders that allow asynchronous execution and handle typical rendering tasks; The overall rendering task is divided into multiple subtasks, each subtask is assigned to a rendering channel and an asynchronous compute shader adapted to the current subtask, and data synchronization between different rendering channels and asynchronous compute shaders is ensured.

6. The transparent window interaction method according to claim 1, characterized in that: The calling of the mouse interaction event judgment function setting includes: setting the judgment function based on the mouse interaction event attribute or setting the judgment function according to the hot zone mechanism or setting the judgment function based on the intelligent prediction result of the user behavior.

7. The transparent window interaction method according to claim 1, characterized in that: Also includes: Design a transparent window manager; Register each created transparent window with the transparent manager and display it in the transparent window list of the transparent window manager with the assigned unique identifier; Associating identifiers of different transparent windows in the transparent window list of the window manager according to user needs; Add synchronization logic to the mouse interaction event judgment function based on the mouse interaction event attribute setting, and broadcast the current mouse interaction event to the transparent windows associated with all registered transparent windows when the synchronization operation is triggered; After receiving the broadcast event, each associated transparent window responds to the mouse interaction event to achieve synchronous interactive operations.

8. A transparent window interaction method system based on DirectX, characterized in that: include: An API integration module, used for integrating DirectX API and DComp API; the DirectX includes DirectX11 or DirectX12; The operating environment detection module detects the operating environment and selects DirectX11 or DirectX12 based on the detected operating environment; API initialization module, used to initialize the selected DirectX11 or DirectX12 API and DComp API; A transparent window creation module is used to create a transparent window, including: determining the transparent window display content, specifying the transparent window style and initializing the window transparency; The transparent window rendering operation module is used to obtain the rendering requirements of the transparent window display content and generate rendering tasks. It uses the DirectX11 or DirectX12 API selected after initialization to complete the rendering of the transparent window display content, and combines the UE multi-channel rendering technology to process different rendering tasks in parallel; synchronously uses the DComp API to complete the transparency adjustment of the transparent window display content; and uses a custom hybrid model to process multi-channel transparent composite rendering based on DComp and UE; The transparent window penetration operation module is used to intercept mouse interaction events using HOOK technology and call the mouse interaction event judgment function to determine whether to perform penetration processing and execute the penetration processing accordingly according to the judgment result, and use the underlying operating system to complete real-time interaction.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images