A multi-screen linkage VR immersive interaction method, device and medium

Abstract

The application discloses a multi-screen linkage VR immersive interaction method and device and medium, and is applied to the VR interaction field. The method comprises the following steps: acquiring three-dimensional scene data of a healing environment; rendering a first visual angle picture based on the three-dimensional scene data, and performing edge splicing on sub-pictures displayed on each display screen corresponding to a multi-screen PC end by using an edge fusion algorithm; establishing a communication connection with a VR end, and sending a healing environment adjustment instruction, so that the VR end adjusts a second visual angle picture displayed according to the healing environment adjustment instruction; or receiving operation state data fed back by the VR end, and checking the operation state data, wherein the healing environment adjustment instruction and the operation state data both comprise corresponding time stamps. It can be seen that the multi-screen and the edge splicing of the sub-pictures displayed on the multi-screen can greatly improve the participation and immersion experience of the same person, and the time stamp mechanism can further realize the synchronization of the visual content of the two ends.

Description

Technical Field

[0001] This application relates to the field of VR interaction, and in particular to a multi-screen interactive VR immersive interaction method, device and medium. Background Technology

[0002] In scenarios such as psychological relaxation, cognitive adjustment, and rehabilitation assistance, virtual reality (VR) technology is increasingly being applied in the field of environmental (e.g., forest) healing. Its core principle is to balance the deeply immersive healing experience of the user with the synchronized immersive observation of accompanying personnel (medical staff, instructors, etc.), achieving the dual goal of "user immersion + accompanying personnel collaborative perception." Large-screen displays, as the key medium connecting these two aspects, need to accurately convey the first-person scene and weather atmosphere from the VR perspective.

[0003] Currently, the implementation of VR and large-screen interaction mainly falls into two categories: one is to transmit the image to a single display device through the VR device's built-in screen projection function, and the other is to use third-party software to achieve synchronized streaming between VR and a single PC (Personal Computer). However, for PC multi-screen systems, most are independent architectures, only able to enlarge or copy the single-screen image, failing to achieve deep interaction and stereoscopic adaptation with the VR device, and lacking the ability to synchronize control for environmental healing scenes and weather modes. Furthermore, current technology suffers from several core defects: First, accompanying personnel lack immersion; single-screen flat displays cannot reproduce the three-dimensional spatial sense and weather atmosphere of forest scenes, and accompanying personnel can only passively watch, unable to perceive the layering of the VR first-person perspective scene; second, the experience is disconnected from multi-screen adaptation; multi-screen systems cannot enhance the experiencer's sense of immersion, and the large-screen display becomes merely a simple image transmission tool; third, weak interactivity and insufficient stability; transmission is mostly unidirectional, the PC cannot actively control the VR device's state, and common protocols easily lead to screen stuttering and command loss, making it difficult to guarantee the continuity of a long-term healing experience.

[0004] In view of the above-mentioned technologies, seeking a method to improve the synchronization accuracy and interactivity of both ends, as well as to take into account the VR healing experience of the user and the immersive viewing needs of accompanying personnel, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a multi-screen interactive VR immersive experience method, device, and medium. This can solve the problems in existing technologies, such as the inability of a single-screen planar display to reproduce the three-dimensional spatial sense and weather atmosphere of a healing scene, the disconnect between the user and the multi-screen adaptation and the lack of immersion for accompanying personnel, and the difficulty in ensuring the continuity of a long-term healing experience due to the inability of the PC to actively control the VR device's state.

[0006] To address the aforementioned technical problems, this application provides a multi-screen interactive VR immersive experience method, applicable to the multi-screen PC portion of a system consisting of a multi-screen PC and a VR device. The method includes: Obtain the 3D scene data corresponding to the predefined healing environment; First-person view is generated based on 3D scene data rendering, and edge blending algorithm is used to stitch together the sub-views displayed on multiple display screens of the multi-screen PC so that the sub-views displayed on multiple display screens together constitute the first-person view. Establish a communication connection with the VR device and send healing environment adjustment commands to the VR device based on the communication connection, so that the VR device can adjust the displayed second-view screen according to the healing environment adjustment commands; or receive operation status data fed back by the VR device based on the communication connection and verify the operation status data. Both the healing environment adjustment commands and the operation status data include corresponding timestamps.

[0007] Preferably, acquiring the 3D scene data corresponding to the predefined healing environment includes: The image acquisition device acquires raw image data corresponding to multiple different environments. Based on the original image data, an initial 3D geometric model and texture map of the environment are generated through a 3D reconstruction algorithm; Generate corresponding 3D scene data based on the initial 3D geometric model and texture map; The healing environment for each environment is determined based on the healing needs corresponding to the VR device; The three-dimensional scene data corresponding to the healing environment is determined based on the healing environment.

[0008] Preferably, generating corresponding 3D scene data based on the initial 3D geometric model and texture map includes: The initial 3D geometric model is meshed to generate optimized geometric model data. The texture map is normalized for resolution and color corrected to generate optimized texture map data. Establish a mapping relationship between geometric model data and texture mapping data so that the texture mapping data accurately covers the surface of the geometric model data; Determine the corresponding dynamic environment parameter set and spatial semantic labels based on geometric model data and texture mapping data; Geometric model data, texture mapping data, mapping relationships, dynamic environment parameter sets, and spatial semantic labels are encapsulated to generate corresponding 3D scene data.

[0009] Preferably, the initial three-dimensional geometric model is subjected to mesh optimization processing to generate optimized geometric model data, including: The initial 3D geometric model is sequentially processed by smoothing filtering, hole detection and repair, and surface reduction to generate a multi-detail model adapted for multi-terminal rendering. Generate corresponding geometric model data based on the multi-level detail model.

[0010] Preferably, a first-person perspective image is generated based on 3D scene data rendering, and an edge blending algorithm is used to stitch together the sub-images displayed on multiple display screens corresponding to a multi-screen PC, including: Receives multiple live video streams output from cameras corresponding to each display screen; Based on the vertical spatial structure of each display screen, the corresponding image segmentation and stereo mapping algorithm is executed for each real-scene video stream to generate a sub-image that matches the physical size and resolution of the corresponding display screen. An edge blending algorithm is performed between adjacent sub-screens corresponding to adjacent display screens to stitch the edges together, thus forming a first-person perspective image.

[0011] Preferably, after performing edge blending algorithms to stitch edges between adjacent sub-screens corresponding to adjacent display screens, the method further includes: Adjust the transparency and color of the image being stitched together at the edges; After compensating for transparency and color, the brightness and contrast of each sub-image are calibrated and adjusted.

[0012] On the other hand, this application provides a multi-screen interactive VR immersive interaction method, applied to the VR end of a system consisting of multi-screen PC and VR end interaction. The method includes: Obtain the 3D scene data corresponding to the predefined healing environment; Based on 3D scene data, the VR device tracks the user's head posture in real time, dynamically captures the corresponding field of view based on the head posture, renders and generates a stereoscopic second-view image, and displays it in the VR headset. Establish a communication connection with the multi-screen PC client and send the user's operation status data to the multi-screen PC client based on the communication connection so that the multi-screen PC client can verify the operation status data; or receive the healing environment adjustment command sent by the multi-screen PC client based on the communication connection so as to adjust the displayed second-view screen according to the healing environment adjustment command; wherein, both the healing environment adjustment command and the operation status data include the corresponding timestamp.

[0013] Preferably, based on 3D scene data, the VR device tracks the user's head posture in real time, dynamically captures the corresponding field of view based on the head posture, renders and generates a stereoscopic second-view image, and displays it on the VR headset, including: Real-time tracking of the user's head pose in six degrees of freedom, including three-dimensional spatial position coordinates and three-dimensional rotation angle; Based on the six degrees of freedom posture, the virtual camera parameters of the experiencer in the healing environment are dynamically determined; Scene data within the corresponding view frustum range is extracted from 3D scene data based on virtual camera parameters; The scene data within the captured view frustum area is rendered according to different viewpoints to generate stereo image pairs with parallax. The stereoscopic image is output to the screen of the VR headset for display, so that a second-person perspective view can be rendered in the VR headset.

[0014] On the other hand, this application also provides an electronic device, including a memory for storing computer programs; The processor is used to execute computer programs to implement the steps of the above-mentioned multi-screen linkage VR immersive interaction method.

[0015] On the other hand, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described multi-screen linkage VR immersive interaction method.

[0016] The multi-screen interactive VR immersive experience method provided in this application is specifically applied to a system consisting of a multi-screen PC and a VR device. Compared to traditional single-screen solutions, the hardware system design of this multi-screen PC constructs a continuous elevation display area corresponding to the virtual healing environment in physical space, significantly expanding the vertical field of view of fellow participants and enabling them to simultaneously perceive the complete spatial hierarchy of the healing environment. Simultaneously, multiple participants can observe and communicate together from different angles within a spacious visual space. Combined with the visual mapping of the VR user's gaze point on the corresponding screen, true spatial empathy and shared experience are achieved, greatly enhancing the participants' sense of participation and immersion. Furthermore, during the rendering of the first-person perspective image on the multi-screen PC, an edge blending algorithm eliminates the stitching marks between adjacent display screens, forming a seamless immersive first-person perspective image, improving the immersion for both participants and the user. The communication connection established between the multi-screen PC and the VR device allows adjustment of the second-person perspective image displayed on the VR device via the multi-screen PC, as well as feedback on VR device operations via the multi-screen PC. Moreover, timestamps are included in the data and command transmission process, further synchronizing the visual content between the two ends. Furthermore, the timestamp mechanism can accurately map the gaze point information of the VR user to the same spatiotemporal location on the corresponding screen of the multi-screen PC, enabling companions to perceive the user's focus in real time. This achieves synergy between the visual content and interaction status of both ends, creating a synchronous and real-time shared immersive healing space for the user and companions. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the system structure consisting of multi-screen PC terminal and VR terminal interaction provided in the embodiments of this application; Figure 2 A flowchart illustrating a multi-screen interactive VR immersive interaction method provided in this application embodiment; Figure 3 A flowchart illustrating a multi-screen interactive VR immersive interaction method according to another embodiment of this application; Figure 4 A complete flowchart of a multi-screen interactive VR immersive interaction method provided in this application embodiment; Figure 5 A structural diagram of an electronic device provided in another embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] The core of this application is to provide a multi-screen interactive VR immersive interaction method, device, and medium.

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In the embodiments, the multi-screen linkage VR immersive interaction method provided in this application is specifically applied to a system consisting of multi-screen PC terminals and VR terminals interacting, such as... Figure 1As shown. The hardware and software structure design of the multi-screen (e.g., three-screen) PC mainly includes: three vertically spliced ​​display screens of the same specifications (supporting vertical splicing installation to ensure color consistency and restore weather texture), a PC host (configured with a multi-core processor and a dedicated graphics card to meet the rendering needs of different 3D environmental scenes and weather effects), a Unity engine (a real-time 3D interactive content creation and operation platform), screen brackets (used to fix the screens to achieve vertical arrangement), and a network interface (used to establish a TCP (Transmission Control Protocol) communication link), breaking through the hardware and software limitations of single-screen streaming. The Unity engine has a built-in three-camera linkage module, which deploys three cameras (left, center, and right) on three screens. Through parameter calibration and viewpoint synchronization, it achieves accurate output and stereoscopic stitching of the three screen images. The subsystem as a whole is responsible for constructing cognitive immersive healing scenes (such as forests, snowfields, and deserts) and weather modes (sunny, cloudy, foggy, and rainy days), generating weather switching and synchronization commands, receiving VR terminal status feedback, splitting and blending the three screen images, adjusting Unity camera parameters, and controlling multi-screen collaborative display. It is the core carrier for realizing multi-screen cognitive immersive effects.

[0023] The VR hardware and software architecture design includes: VR / AR terminal devices (with network connectivity and motion capture capabilities, supporting high-definition scene and weather effect rendering) and user-worn devices (such as controllers, VR headsets, etc.) used to collect user operation data and provide an immersive experience. In this design, based on the VR / AR terminal devices, a healing scene software identical to that used on the PC is installed. This software presets weather modes and corresponding visual and sound effects, receiving commands from the PC, executing weather switching and scene actions, collecting user operation data and feeding it back to the multi-screen PC, while simultaneously providing a first-person, immersive, healing forest experience.

[0024] Meanwhile, the connection between the PC and VR devices also includes routers and network cables. Wired connections are prioritized to ensure the stability of command and data transmission, but wireless TCP connections are also supported to adapt to mobile healing scenarios, creating a local area network communication environment between the PC and VR devices. Specifically, a TCP communication subsystem is deployed between both the PC and VR devices to establish a bidirectional TCP communication link, ensuring reliable transmission of weather switching commands, scene synchronization data, and operation feedback. This guarantees the orderly and real-time transmission of data, supporting weather and scene synchronization between the two devices.

[0025] Figure 2 A flowchart illustrating a multi-screen interactive VR immersive interaction method provided in this application embodiment is shown below. Figure 2 As shown, it includes the following steps: S10: Obtain the 3D scene data corresponding to the predefined healing environment.

[0026] S11: Render a first-person perspective image based on 3D scene data, and use an edge blending algorithm to stitch together the sub-images displayed on multiple display screens of the multi-screen PC so that the sub-images displayed on multiple display screens together constitute the first-person perspective image.

[0027] S12: Establish a communication connection with the VR terminal and send a healing environment adjustment command to the VR terminal based on the communication connection, so that the VR terminal can adjust the displayed second-view screen according to the healing environment adjustment command; or receive operation status data fed back by the VR terminal based on the communication connection and verify the operation status data, wherein both the healing environment adjustment command and the operation status data include the corresponding timestamp.

[0028] In a specific embodiment, steps S10-S12 are applied to the multi-screen PC. First, the user on the multi-screen PC determines the healing environment based on the needs of the VR user, such as a forest, snowfield, or desert, and determines the corresponding 3D scene data. Next, the VR user renders a second-view image based on the 3D scene data and displays it on the corresponding VR headset. Then, the PC renders a first-view image based on the 3D scene data and transmits it to the multiple display screens on the multi-screen PC. The main process involves using an edge blending algorithm to stitch the sub-images displayed on the multiple display screens of the multi-screen PC together. This can be understood as performing pixel-level edge blending processing on the stitching edges of adjacent sub-images to eliminate stitching marks, thereby forming a continuous, seamless healing scene image, i.e., the first-view image.

[0029] The steps for rendering a second-view image on the VR device and displaying it in the corresponding VR headset are as follows: Based on the same 3D scene data as the PC device, the VR device tracks the user's head posture in real time, dynamically captures the corresponding field of view based on the head posture, renders and generates a stereoscopic second-view image, and displays it in the VR headset. In other words, the specific viewpoint displayed in the VR headset worn by the user is adjusted in real time according to the user's head posture.

[0030] After multiple display screens jointly create a first-person perspective view and the VR headset creates a second-person perspective view, a communication connection is established between the two to achieve synchronization accuracy and interactivity. Based on this connection, the multi-screen PC (the user) can adjust the specific scene in the first-person perspective view using controllers or user interfaces, such as changing a forest scene to a desert scene or a sunny day to a rainy day. The multi-screen PC then sends a healing environment adjustment command with weather mode encoding and timestamp information to the VR device via the communication connection, ensuring accurate transmission and execution. The VR device parses the healing environment adjustment command, adjusts the weather mode, and synchronously renders the second-person perspective view and sound effects displayed on the VR headset, creating a deeply immersive healing environment for the user. Simultaneously, the VR device collects the user's operation status data and feeds it back to the multi-screen PC, which verifies the operation status data to achieve the purpose of two-terminal interaction.

[0031] It's easy to understand that after acquiring the predefined 3D scene data corresponding to the healing environment, the order in which the multi-screen PC renders the first-person view on multiple display screens based on the 3D scene data, or the VR device renders the second-person view on the VR headset based on the 3D scene data, does not affect subsequent operations. This is because both devices generate the image based on the 3D scene data, so the first-person and second-person views are identical after rendering. It's important to note that the first-person view displayed on the multi-screen PC does not change with the VR user's movement. In other words, the person on the multi-screen PC can only experience the environment of the VR user based on their own screen and can interact with the VR user when they do. However, the second-person view displayed on the VR device adjusts its viewing angle according to the user's head posture.

[0032] It is also easy to understand that the healing environment adjustment instructions and operation status data both include corresponding timestamps, which can further enable synchronization between the two ends.

[0033] The multi-screen VR immersive interaction method provided in this application is specifically applied to a system consisting of a multi-screen PC and a VR terminal. Compared to traditional single-screen solutions, the hardware system design of this multi-screen PC constructs a continuous elevation display area corresponding to the virtual healing environment in physical space, significantly expanding the vertical field of view of companions and enabling them to simultaneously perceive the complete spatial hierarchy of the healing environment. Simultaneously, multiple companions can observe and communicate together from different angles within a spacious visual space. Combined with the visual mapping of the VR user's gaze point on the corresponding screen, true spatial empathy and shared experience are achieved, greatly enhancing the participation and immersive experience of companions. Furthermore, during the first-person perspective rendering process on the multi-screen PC, edge blending algorithms eliminate splicing artifacts between adjacent display screens, forming a seamless immersive first-person perspective image, improving the immersion of companions and the user. The communication connection established between the PC and VR terminal allows adjustment of the second-person perspective image displayed on the VR terminal via the multi-screen PC, as well as feedback on VR terminal operations via the multi-screen PC. Moreover, timestamps are included in the data and command transmission process, further synchronizing the visual content on both ends. Furthermore, the timestamp mechanism can accurately map the gaze point information of the VR user to the same spatiotemporal location on the corresponding screen of the multi-screen PC, enabling companions to perceive the user's focus in real time. This achieves synergy between the visual content and interaction status of both ends, creating a synchronous and real-time shared immersive healing space for the user and companions.

[0034] Based on the above embodiments, step S10, obtaining the three-dimensional scene data corresponding to the predefined healing environment, is specifically implemented as follows: acquiring original image data corresponding to multiple different environments through an image acquisition device; generating an initial three-dimensional geometric model and texture map of the environment based on each original image data through a three-dimensional reconstruction algorithm; generating corresponding three-dimensional scene data based on the initial three-dimensional geometric model and texture map; determining the healing environment in each environment according to the healing requirements of the VR terminal; and determining the three-dimensional scene data corresponding to the healing environment based on the healing environment.

[0035] In a specific embodiment, this application pre-collects raw image data of multiple different natural environments (such as forests, streams, and beaches) using image acquisition devices. Based on a 3D reconstruction algorithm, it generates initial 3D geometric models and texture maps corresponding to each environment, thereby constructing complete 3D scene data of a digital healing environment. The system automatically matches or allows companions or therapists to select the most suitable healing environment from multiple candidate environments based on the specific healing needs of the VR user (such as anxiety relief, stress release, and emotion regulation), and retrieves the corresponding 3D scene data for that environment, providing a data foundation for subsequent multi-device synchronous presentation.

[0036] It should be noted that the embodiments provided in this application are only one possible implementation method, but are not limited to this only implementation method. Users can adjust them according to their own needs.

[0037] This application achieves precise matching and on-demand retrieval of healing environments and healing needs by predefining multiple 3D scene data of different healing environments. On the one hand, the scene data generated based on real-world environment acquisition and 3D reconstruction has a high degree of visual realism and immersion, which can more effectively trigger positive emotional responses in the experiencer. On the other hand, the diverse predefined environments provide a wealth of choices for personalized treatment plans. Companions or therapists can flexibly select the most suitable healing scene according to the specific situation of the experiencer, significantly improving the pertinence and effectiveness of psychological therapy. At the same time, the predefined data storage method avoids the computational overhead of real-time scene construction, ensuring the smoothness and stability of subsequent multi-screen PC rendering and real-time tracking on VR.

[0038] The specific implementation method for generating corresponding 3D scene data based on the initial 3D geometric model and texture map is as follows: The initial 3D geometric model is meshed to generate optimized geometric model data; the texture map is normalized for resolution and corrected for color to generate optimized texture map data; a mapping relationship is established between the geometric model data and the texture map data so that the texture map data accurately covers the surface of the geometric model data; the corresponding dynamic environment parameter set and spatial semantic labels are determined based on the geometric model data and texture map data; the geometric model data, texture map data, mapping relationship, dynamic environment parameter set, and spatial semantic labels are encapsulated to generate the corresponding 3D scene data.

[0039] The initial 3D geometric model is mesh optimized to generate optimized geometric model data. This includes: performing smoothing filtering, hole detection and repair, and surface reduction on the initial 3D geometric model in sequence to generate a multi-level detail model adapted for multi-terminal rendering; and generating corresponding geometric model data based on the multi-level detail model.

[0040] In specific embodiments, this application performs mesh optimization on the initial 3D geometric model, including smoothing filtering to eliminate acquisition noise, hole repair to ensure geometric integrity, and facet reduction to generate a multi-level detail model, thereby obtaining optimized geometric model data. Simultaneously, it performs resolution standardization and color correction on the texture map, including unified resolution conversion of multi-source textures, color balancing and lighting consistency correction of textures acquired from different viewpoints, and tone mapping repair of exposed areas, generating optimized texture map data. It establishes a UV mapping relationship (UV Mapping, the process of establishing the correspondence between 3D model vertices and 2D texture image pixels) between the optimized geometric model data and the texture map data, ensuring that the texture map accurately covers the surface of the geometric model. Based on the optimized geometric model and texture map data, it configures corresponding dynamic environment parameter sets (including lighting parameters, weather particle parameters, visibility parameters, and time axis parameters) and spatial semantic tags (including region type tags, interactive hotspot tags, and viewpoint recommendation tags). Finally, it integrates the geometric model data, texture map data, UV mapping relationship, dynamic environment parameter set, and spatial semantic tags into a unified data encapsulation, generating high-quality 3D scene data with complete structure and rich semantics.

[0041] It should be noted that the embodiments provided in this application are only one possible implementation method, but are not limited to this only implementation method. Users can adjust them according to their own needs.

[0042] This application effectively eliminates noise, holes, and color inconsistencies in the original acquired data through multiple optimization processes of the initial geometric model and texture mapping, significantly improving the geometric accuracy and visual realism of the 3D scene. By establishing precise UV mapping relationships, it ensures perfect fit between texture and geometry, avoiding texture stretching or misalignment during multi-screen presentation. The introduction of dynamic environmental parameter sets and spatial semantic tags enables the 3D scene data to not only contain static visual information but also predefine dynamic and interactive attributes such as weather changes, light and shadow evolution, and interactive hotspots, providing rich data support for subsequent multi-screen PC rendering and VR real-time tracking. The integrated encapsulated data structure facilitates efficient system access and multi-terminal synchronization, laying a high-quality data foundation for achieving an immersive healing experience of spatial empathy and real-time interaction between users and their companions.

[0043] Based on the above embodiments, as a preferred embodiment, the specific implementation of step S11, which involves rendering a first-view image based on 3D scene data and using an edge blending algorithm to stitch together the sub-images displayed on multiple screens corresponding to a multi-screen PC, is as follows: receiving multiple real-scene video streams output by cameras corresponding to each screen; performing corresponding image segmentation and stereo mapping algorithms on each real-scene video stream according to the vertical spatial structure of each screen to generate sub-images that match the physical size and resolution of the corresponding screen; performing edge blending algorithms to stitch together adjacent sub-images corresponding to adjacent screens; performing transparency and color compensation on the edge-stitched images; and after compensating for transparency and color, calibrating and adjusting the brightness and contrast of each sub-image to generate a first-view image.

[0044] In a specific embodiment, the present invention first receives multiple real-scene video streams output by cameras corresponding to each display screen; based on the vertical spatial structure of the multi-screen display array (including screen physical size, resolution, and vertical arrangement position), it performs a dedicated image segmentation and stereo mapping algorithm on each real-scene video stream to generate sub-images that precisely match each screen; then, it performs an edge blending algorithm on the splicing area between adjacent sub-images corresponding to adjacent display screens, eliminating splicing artifacts through pixel-level transparency blending and color compensation; finally, it performs unified brightness and contrast calibration and adjustment on each blended sub-image to ensure that the entire image is visually balanced in brightness and harmonious in contrast, thereby generating a continuous and seamless first-person perspective image and outputting it to the multi-screen display array.

[0045] It should be noted that the embodiments provided in this application are only one possible implementation method, but are not limited to this only implementation method. Users can adjust them according to their own needs.

[0046] This application ensures precise matching of the physical characteristics of each sub-screen with its corresponding display screen through dedicated image segmentation and stereoscopic mapping for each display screen, fundamentally avoiding visual distortion caused by image stretching or compression. The edge blending algorithm performs pixel-level transparency and color compensation on the splicing area, effectively eliminating bright bands, dark areas, or color abrupt changes common in multi-screen splicing. This allows viewers to perceive a seamless, geometrically continuous, and consistently lit overall image, rather than a simple patchwork of multiple independent screens. Subsequent global brightness and contrast calibration further unifies the display effect of each screen, eliminating visual inconsistencies caused by individual screen differences or ambient light. This series of processes provides viewers with a wide field of view, high continuity, and visually uniform immersive viewing experience, enabling them to truly experience the complete spatial hierarchy and atmosphere of a healing environment, significantly improving visual comfort and empathy when multiple people are together.

[0047] On the other hand, the multi-screen interactive VR immersive interaction method provided in this application is specifically applied to the VR terminal in a system consisting of multi-screen PC terminals and VR terminals interacting. This method includes the following steps: Figure 3 As shown: S20: Obtain the 3D scene data corresponding to the predefined healing environment.

[0048] S21: Based on 3D scene data, the VR device tracks the user's head posture in real time, dynamically captures the corresponding field of view based on the head posture, renders and generates a stereoscopic second-view image, and displays it in the VR headset.

[0049] S22: Establish a communication connection with the multi-screen PC and send the user's operation status data to the multi-screen PC based on the communication connection so that the multi-screen PC can verify the operation status data; or receive the healing environment adjustment command sent by the multi-screen PC based on the communication connection so as to adjust the displayed second-view screen according to the healing environment adjustment command; wherein, both the healing environment adjustment command and the operation status data include the corresponding timestamp.

[0050] Since the embodiments corresponding to the operation of the VR terminal in the entire multi-screen linkage VR immersive interaction method are described in the above embodiments, this application will not repeat them here.

[0051] Based on 3D scene data, it tracks the user's head posture in real time on the VR device, dynamically extracts the corresponding field of view based on the head posture, renders and generates a stereoscopic second-view image, and displays it on the VR headset. This includes: real-time tracking of the user's six degrees of freedom (DOF) head posture, including 3D spatial position coordinates and 3D rotation angles; dynamically determining the virtual camera parameters of the user in the therapeutic environment based on the six DDF posture; extracting scene data within the corresponding frustum range from the 3D scene data based on the virtual camera parameters; rendering the extracted scene data within the frustum range according to different perspectives to generate stereoscopic image pairs with parallax; and outputting the stereoscopic image pairs to the VR headset screen for display, so as to render and generate a stereoscopic second-view image in the VR headset.

[0052] In a specific embodiment, this application tracks the six-degree-of-freedom posture of the user's head in real time (including three-dimensional spatial position coordinates and three-dimensional rotation angles), and dynamically determines the virtual camera parameters (position, orientation, and field of view) of the user in the healing environment based on the posture; based on these parameters, scene data within the current visual cone range is accurately extracted from the three-dimensional scene data, and corresponding stereoscopic image pairs are generated according to the preset interpupillary distance offset of the left and right eye perspectives respectively; finally, the stereoscopic image pairs with parallax are output in real time to the left and right eye screens of the VR headset for display, so that the user can naturally explore the healing environment by turning their head and obtain visual feedback and spatial perception consistent with the real world.

[0053] This application achieves highly natural visual interaction between the user and the virtual healing environment through real-time tracking of head posture with six degrees of freedom and dynamic frustum capture. Turning the head changes the perspective, and walking changes the position, greatly enhancing the sense of immersion and presence. The stereoscopic images generated based on the same 3D scene data source accurately simulate the binocular parallax of the human eye. Combined with the optical system of the VR headset, it creates a realistic spatial depth perception for the user, which helps to relax the mind and body and immerse the emotions. At the same time, the rendering mechanism ensures that every frame of the VR end is generated synchronously with the screen displayed on the multi-screen PC end based on the same source data and coaxial time. This provides a precise visual basis for spatial empathy and real-time interaction (such as gaze mapping and guided communication) between the user and companions, significantly improving the consistency and effectiveness of the dual-end collaborative healing experience.

[0054] In summary, the multi-screen interactive VR immersive interaction method provided in this application is applicable to both multi-screen PCs and VR devices. For the multi-screen PC, it specifically comprises four points: First, it uses 3D modeling software to construct a cognitive immersive healing environment model, presets multiple weather modes, defines the corresponding visual parameters (light intensity, visibility, raindrop density, etc.) and sound effect parameters for each mode, and exports them in a dual-end compatible format to ensure consistency of scene and weather effects between the multi-screen PC and VR devices, supporting real-time adjustment of weather parameters. Second, it generates healing environment adjustment commands such as weather switching, perspective adjustment, and start / stop based on user operations (mouse, keyboard, brain-computer interface, etc.) on the multi-screen PC. The command format includes command type, weather mode code, timestamp, etc., where the timestamp is used to ensure the temporal consistency of command execution between the two devices, ensuring accurate synchronization between weather switching and scene. The third point is that the Unity engine is responsible for controlling the parameters and synchronization status of the left, center, and right cameras to ensure precise synchronization between the three screens and the VR perspective. It also optimizes rendering parameters based on weather patterns. The image blending submodule runs dedicated image segmentation, stereoscopic mapping, and edge blending algorithms to further calibrate the soothing environmental images and corresponding weather effects output by the three cameras according to the vertical spatial structure of the three screens. It performs pixel blending processing on the stitching edges to eliminate artifacts and dynamically adjusts the brightness and contrast consistency of each screen to ensure a collaborative, immersive stereoscopic effect. The fourth point is establishing a TCP communication connection with the VR device, sending command data, and receiving operation status data from the VR device. It verifies the transmitted data to ensure data integrity and prioritizes the transmission of weather switching commands to guarantee timely synchronization.

[0055] For the VR end, there are three main points: First, it loads a healing environment model identical to that of the multi-screen PC. Based on the healing environment adjustment commands sent by the multi-screen PC, it renders the scene with the corresponding weather mode (synchronously updating visuals and sound effects), while responding to the user's actions (such as moving the viewpoint with the controller or touching objects in the scene), ensuring a first-person immersive experience. Second, it collects the user's operation data (viewpoint rotation angle, controller actions, position movement information, etc.) through the VR end's corresponding sensors, converts the data into a standardized format, and synchronously uploads it to the multi-screen PC, supporting cross-platform status interaction. Third, it establishes a TCP communication connection with the multi-screen PC, receives and parses synchronization commands and weather switching commands from the multi-screen PC, and executes them. Simultaneously, it feeds back the collected operation status data to the multi-screen PC, ensuring real-time data interaction and scene / weather synchronization between the two ends.

[0056] In summary, the workflow of a system consisting of multi-screen PC and VR interaction is as follows: Figure 4 As shown, it includes the following steps: S30: System initialization.

[0057] S31: Establishment of TCP communication link + verification and synchronization.

[0058] S32: Generate / Adjust instructions in a loop.

[0059] S33: Render corresponding visual images on the VR side and the multi-screen PC side.

[0060] S34: Continuously maintain synchronization for the VR side and the multi-screen PC side in real time, and return to step S32 without termination conditions.

[0061] S35: Under the condition of need for termination, the multi-screen PC side issues a shutdown instruction, disconnects the link, and the devices shut down in sequence.

[0062] Among them, it is not difficult to understand that for steps S30 - S31, start the multi-screen PC side and the VR side devices, respectively load the homologous healing environment model and weather mode parameters, the multi-screen PC side completes parameter settings, establishes a two-way communication link between the multi-screen PC side and the VR side, completes connection verification and data synchronization, and ensures normal communication, consistent scene and weather parameters. For step S32, the multi-screen PC side user (such as a staff member) issues instructions such as weather switching and scene startup through the operation interface according to the healing needs or the feedback of the experiencer. The instructions carry the weather mode code and timestamp information to ensure the accuracy of transmission and execution. For step S33, the multi-screen PC side transmits the weather switching instruction and the synchronization instruction to the VR side through the TCP communication connection. After the VR side analyzes the instruction, it switches to the corresponding weather mode, synchronizes the rendered image and sound effect, and constructs a deeply immersive healing environment for the experiencer; at the same time, collects the operation data of the experiencer and feeds it back to the multi-screen PC side; while the multi-screen PC side performs segmentation, stereoscopic mapping and edge fusion processing on the scene image in the current weather mode, converts the scene of the healing environment into a three-screen stereoscopic image, and the accompanying personnel obtain an immersive viewing experience through the three screens, completely solving the problem of lack of immersion in traditional large-screen displays. For step S34, the two sides continuously exchange data through the TCP link. The multi-screen PC side can issue instructions such as weather switching and perspective adjustment in real time according to the feedback of the experiencer or the healing needs. The VR side corrects the scene and weather states according to the instructions, and at the same time continuously feeds back the operation data to form a closed-loop synchronization, ensuring real-time linkage of the scenes, weather and the actions of the experiencer on both sides without obvious delay. For step S35, the multi-screen PC side issues a stop instruction, the two sides stop scene rendering and weather effect output, the TCP communication link is disconnected, and the devices shut down in sequence.

[0063] Therefore, this application addresses the pain point of lack of immersion in large-screen displays by constructing an integrated system consisting of a multi-screen PC, a VR terminal, and TCP communication. It pre-builds a shared, immersive, healing environment scene and multiple switchable weather modes on both terminals. The multi-screen PC acts as the master control terminal, issuing weather switching and status synchronization commands. Real-time linkage between the scenes and weather on both terminals is achieved via the TCP protocol. This not only provides VR users with a deeply immersive healing experience but also transforms the healing scene and weather effects into a 3D image through a multi-screen-specific adaptation algorithm and a Unity three-camera control scheme. This presents accompanying personnel with an immersive viewing experience identical to that of the user, completely resolving the issues of lack of immersion for accompanying personnel and disconnect between the user and the multi-screen setup caused by single-screen solutions. Furthermore, during the first-person perspective rendering process on the multi-screen PC, an edge blending algorithm eliminates splicing artifacts between adjacent display screens, forming a seamless immersive first-person perspective image, enhancing the immersion experience for both accompanying personnel and the user. Simultaneously, the communication connection established between the PC and VR terminals allows for adjustment of the second-view image displayed on the VR terminal via the multi-screen PC, as well as feedback on VR terminal operations via the multi-screen PC. Furthermore, timestamps are included in the data and command transmission process, further synchronizing the visual content on both ends. The timestamp mechanism accurately maps the VR user's gaze point information to the corresponding spatiotemporal location on the multi-screen PC screen, enabling companions to perceive the user's focus in real time. This achieves synergy between the visual content and interaction status on both ends, creating a synchronized, real-time shared immersive healing space for both the user and their companions.

[0064] Figure 5 A structural diagram of an electronic device provided in another embodiment of this application, such as... Figure 5 As shown, the electronic device includes: a memory 20 for storing computer programs; The processor 21 is used to execute computer programs to implement the steps of the multi-screen linkage VR immersive interaction method mentioned in the above embodiments.

[0065] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0066] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0067] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the multi-screen linkage VR immersive interaction method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc.

[0068] In some embodiments, the electronic device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0069] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0070] The electronic device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can realize the above-mentioned multi-screen linkage VR immersive interaction method and has the same beneficial effects.

[0071] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0072] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The foregoing has provided a detailed description of a multi-screen interactive VR immersive interaction method, device, and medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0074] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A multi-screen interactive VR immersive experience method, characterized in that, The method, applied to the multi-screen PC terminal in a system consisting of multi-screen PC terminals and VR terminals interacting, includes: Obtain the 3D scene data corresponding to the predefined healing environment; The first-view image is generated based on the three-dimensional scene data, and the edge blending algorithm is used to stitch together the sub-images displayed on the multiple display screens corresponding to the multi-screen PC so that the sub-images displayed on the multiple display screens together constitute the first-view image. A communication connection is established with the VR terminal, and a healing environment adjustment command is sent to the VR terminal based on the communication connection, so that the VR terminal adjusts the displayed second-view screen according to the healing environment adjustment command; or the operation status data fed back by the VR terminal is received based on the communication connection, and the operation status data is verified, wherein both the healing environment adjustment command and the operation status data include a corresponding timestamp.

2. The VR immersive interaction method with multi-screen linkage according to claim 1, characterized in that, The acquisition of the three-dimensional scene data corresponding to the predefined healing environment includes: The image acquisition device acquires raw image data corresponding to multiple different environments. Based on the original image data, an initial three-dimensional geometric model and texture map of the environment are generated using a three-dimensional reconstruction algorithm; Generate corresponding 3D scene data based on the initial 3D geometric model and the texture map; The healing environment is determined based on the healing needs corresponding to the VR terminal in each of the aforementioned environments; Based on the healing environment, the corresponding three-dimensional scene data is determined.

3. The VR immersive interaction method with multi-screen linkage according to claim 2, characterized in that, The step of generating the corresponding 3D scene data based on the initial 3D geometric model and the texture map includes: The initial three-dimensional geometric model is subjected to mesh optimization processing to generate optimized geometric model data; The texture map is subjected to resolution normalization and color correction to generate optimized texture map data; Establish a mapping relationship between the geometric model data and the texture mapping data so that the texture mapping data accurately covers the surface of the geometric model data; The corresponding dynamic environment parameter set and spatial semantic labels are determined based on the geometric model data and the texture mapping data; The geometric model data, texture mapping data, mapping relationships, dynamic environment parameter set, and spatial semantic tags are encapsulated to generate the corresponding 3D scene data.

4. The VR immersive interaction method with multi-screen linkage according to claim 3, characterized in that, The step of performing mesh optimization on the initial three-dimensional geometric model to generate optimized geometric model data includes: The initial three-dimensional geometric model is sequentially subjected to smoothing filtering, hole detection and repair, and surface reduction to generate a multi-detail model adapted for multi-terminal rendering. The corresponding geometric model data is generated based on the multi-level detail model.

5. The VR immersive interaction method with multi-screen linkage according to any one of claims 1-4, characterized in that, The process of rendering a first-view image based on the 3D scene data and then stitching together the sub-images displayed on multiple display screens corresponding to the multi-screen PC using an edge blending algorithm includes: Receive multiple live video streams output by the cameras corresponding to each of the aforementioned display screens; Based on the vertical spatial structure of each display screen, a corresponding screen segmentation and stereo mapping algorithm is executed for each of the real-scene video streams to generate the sub-screen that matches the physical size and resolution of the corresponding display screen. The edge blending algorithm is performed on adjacent sub-screens corresponding to adjacent display screens to stitch the edges together, thus forming the first viewpoint image.

6. The VR immersive interaction method with multi-screen linkage according to claim 5, characterized in that, After performing edge blending algorithm to stitch edges between adjacent sub-screens corresponding to adjacent display screens, the method further includes: Adjust the transparency and color of the image being stitched together at the edges; After compensating for the transparency and color, the brightness and contrast of each sub-image are calibrated and adjusted.

7. A multi-screen interactive VR immersive experience method, characterized in that, The method, applied to the VR terminal in a system consisting of multi-screen PC and VR terminal interaction, includes: Obtain the 3D scene data corresponding to the predefined healing environment; Based on the three-dimensional scene data, the VR device tracks the user's head posture in real time, dynamically captures the corresponding field of view according to the head posture, renders and generates a stereoscopic second-view image, and displays it in the VR headset. A communication connection is established with the multi-screen PC client, and the user's operation status data is sent to the multi-screen PC client based on the communication connection so that the multi-screen PC client can verify the operation status data; or the user receives a healing environment adjustment command sent by the multi-screen PC client based on the communication connection so as to adjust the displayed second perspective screen according to the healing environment adjustment command; wherein, both the healing environment adjustment command and the operation status data include a corresponding timestamp.

8. The VR immersive interaction method with multi-screen linkage according to claim 7, characterized in that, The process of tracking the user's head posture in real time on the VR device based on the 3D scene data, dynamically capturing the corresponding field of view according to the head posture, rendering and generating a second-view stereoscopic image, and displaying it on the VR headset includes: The six-degree-of-freedom pose of the user's head is tracked in real time, and the six-degree-of-freedom pose includes three-dimensional spatial position coordinates and three-dimensional rotation angles; Based on the six degrees of freedom posture, the virtual camera parameters of the experiencer in the healing environment are dynamically determined; Based on the virtual camera parameters, scene data within the corresponding view frustum range is extracted from the 3D scene data; The scene data within the captured view frustum area is rendered according to different viewpoints to generate stereoscopic image pairs with parallax. The stereoscopic image is output to the screen of the VR headset for display, so as to render and generate a second perspective image with a stereoscopic view in the VR headset.

9. An electronic device, characterized in that, Includes memory used to store computer programs; A processor, configured to execute the computer program to implement the steps of the multi-screen VR immersive interaction method as described in any one of claims 1 to 6, or the steps of the multi-screen VR immersive interaction method as described in claim 7 or 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-screen VR immersive interaction method as described in any one of claims 1 to 6, or the steps of the multi-screen VR immersive interaction method as described in claim 7 or 8.

Images