Image fusion stereoscopic display method and system based on unity engine

Abstract

The application provides an image fusion stereoscopic display method and system based on a Unity engine, and relates to the fields of computer graphics and virtual reality technology.The method configures left-eye and right-eye cameras in a Unity scene, collects left and right view angle images according to a pupil distance parameter, and generates left-eye and right-eye rendering textures;UI textures are generated according to the Canvas types of UI elements, and parallax compensation, pixel alignment and image fusion processing are performed through Compute Shader, the UI texture transparency is superimposed, and a composite display image is generated;the fusion image introduces the results of the previous frame, and time-continuous output is realized based on an inter-frame smoothing function.The composite texture is output to a device supporting multiple stereoscopic display modes through a graphics rendering pipeline.The application realizes efficient fusion of 3D images and UI, improves rendering efficiency and visual immersion, and is suitable for virtual reality, augmented reality and digital twin scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer graphics and virtual reality technology, and particularly relates to an image fusion stereoscopic display method and system based on a Unity engine. BACKGROUND

[0002] With the rapid development of virtual reality (VR), augmented reality (AR) and 3D content creation (such as games, virtual simulation, digital twin, etc.), the Unity engine, as a mainstream real-time 3D development tool, is widely used in 3D stereoscopic display, game development and virtual reality fields. However, the existing 3D stereoscopic display technology has many problems in the cooperation of images and UI elements, the efficiency of resource adaptation, hardware device compatibility, rendering efficiency and visual personalization. Specifically, the cooperation of UI elements and 3D images in the prior art is usually fragmented, especially in the Overlay mode, the UI often completely blocks the stereoscopic picture, resulting in poor display effect, and the developer needs to manually adjust the coordinates, levels and other parameters of the UI elements, causing the development cycle to be prolonged. In addition, the traditional hardware adaptation method also cannot effectively compatible with different types of display devices, which is easy to cause picture tearing and synchronization delay, affecting the user experience.

[0003] At the same time, the rendering method in the prior art mostly relies on CPU for texture synthesis processing, which leads to the inability to meet the rendering requirements of high-resolution and high-frame-rate scenes (such as 4K / 120fps games), resulting in problems such as picture lag and rendering delay. The prior art also fails to fully utilize the parallel computing capability of GPU, limiting the improvement of image processing speed and rendering efficiency, further affecting the smoothness and stability of the display effect. In addition, the distance between the existing binocular cameras usually adopts a fixed value (such as 0.065 meters), which is difficult to adapt to children, myopic users or special viewing angle requirements, resulting in insufficient immersion of the visual effect and failing to meet the personalized needs of users.

[0004] Therefore, how to solve these problems and improve the flexibility of 3D stereoscopic display and UI cooperation, the efficiency of resource adaptation, hardware compatibility, rendering efficiency and visual personalization has become the core problem to be solved in the current technology. SUMMARY

[0005] The present application aims to provide an image fusion stereoscopic display method and system based on a Unity engine, to solve the limitations of the prior art, such as the fragmentation of 3D and UI cooperation, low resource adaptation efficiency, weak stereoscopic display hardware adaptation, rendering efficiency bottleneck and lack of visual personalization, etc. Through innovative UI adaptation and image fusion technology, GPU accelerated rendering, dynamic pupil distance adjustment and multi-display device compatibility, the smoothness, compatibility and user experience of the display effect are significantly improved, while the development cycle is shortened and the computing efficiency of the system is improved.

[0006] The present application realizes the above-mentioned purpose through the technical scheme as follows:

[0007] In one aspect, the present application provides an image fusion stereoscopic display method based on a Unity engine, which comprises:

[0008] Step S1: configuring a left-eye camera and a right-eye camera in a Unity scene, respectively collecting a left-view image and a right-view image based on a set interpupillary distance parameter, and generating a left-eye rendering texture and a right-eye rendering texture;

[0009] Step S2: configuring a corresponding display mode mark according to the Canvas type to which each UI element in a user interface is bound, the display mode mark being used to indicate the display mode of the UI element in the left-eye image and the right-eye image, the display mode including binocular synchronous display, left-eye-only display and right-eye-only display; and generating a user interface texture based on a UI rendering process;

[0010] Step S3: inputting the left-eye rendering texture, the right-eye rendering texture and the user interface texture into a Compute Shader to perform image fusion processing, the fusion processing including: dynamically calculating the fusion weight of the left and right views based on the image content features; performing weighted operation on the previous frame image and the current frame image, the weight of the weighted operation being determined by an adaptive smoothing coefficient; performing disparity compensation on the right-eye image based on a depth map and performing pixel alignment; generating a fusion image by fusing the left-eye image and the right-eye image according to the fusion weight; and superimposing the UI layer according to the transparency of the user interface texture to obtain a composite display image;

[0011] Step S4: writing the composite display image into a composite display texture, and outputting the composite display texture to a stereoscopic display device, the stereoscopic display device supporting frame sequence type, shutter type or line alternation type stereoscopic image display mode.

[0012] Preferably, the step S1 comprises:

[0013] The left-eye camera and the right-eye camera are respectively generated by copying the main camera, the left-eye camera and the right-eye camera are symmetrically offset relative to the main camera along the horizontal direction, the offset distance being an adjustable parameter, the default value being 0.065 meters, and the value range being 0.03 meters to 0.08 meters;

[0014] The interpupillary distance parameter value is adjusted in real time through a control interface, and the position offset of the left-eye camera and the right-eye camera in the horizontal direction is dynamically updated;

[0015] The field of view angle of the left-eye camera and the right-eye camera is set to 60 degrees;

[0016] The left-eye camera and the right-eye camera respectively collect left-view images and right-view images, and generate left-eye rendering textures and right-eye rendering textures through rendering, the rendering textures being two-dimensional textures readable and writable by the GPU, and the data type being Texture2D <float4>.

[0017] Preferably, the step S2 comprises:

[0018] identifying the Canvas type to which the UI element belongs based on the display mode mark, when the Canvas type is Overlay mode, generating a separate user interface texture bound to a Render Texture object; when the Canvas type is World Space mode or Screen Space-Camera mode, respectively collecting images by the left eye camera and the right eye camera and generating a unified user interface texture;

[0019] automatically configuring rendering parameters of the user interface texture according to the Canvas type, the rendering parameters including rendering priority, scaling factor and occlusion level information;

[0020] the user interface texture is a two-dimensional texture readable and writable by the GPU, and the data type is Texture2D <float4>.

[0021] Preferably, the step S3 comprises:

[0022] inputting the left-eye rendering texture, the right-eye rendering texture and the user interface texture into a Compute Shader, performing image fusion processing in the Compute Shader, the image fusion processing comprising the following steps:

[0023] dynamically calculating a fusion weight of the left view and the right view based on image content features;

[0024] calculating a parallax compensation displacement of the right-eye image based on a depth map and performing pixel alignment;

[0025] performing fusion on the aligned left and right images to obtain a fusion image;

[0026] superimposing the user interface image into the fusion image according to its transparency information to generate a final display image, and the fusion calculation formula is:

[0027] ;

[0028] wherein: is a color value of the left-eye image at a pixel position ; is a color value of the right-eye image at a pixel position ; is a color value of the user interface image at a pixel position ; is an Alpha channel value of the user interface image at a position , ranging from 0 to 1, for controlling transparency; is a weight coefficient of left and right image fusion, dynamically calculated based on image content, ranging from 0 to 1; is a parallax compensation displacement of the right-eye image at a position , derived from a depth map; is a pixel value after left and right image fusion; is a color value of the final synthesized display image at a pixel position .

[0029] The step S3 further comprises:

[0030] obtaining a fusion weight change, a parallax compensation displacement change and a user visual parameter between a current frame and a previous frame; and calculating an adaptive inter-frame smoothing coefficient of each pixel position according to a basic smoothing coefficient, a weight change adjustment coefficient, a parallax change adjustment coefficient and a user parameter adjustment coefficient;

[0031] Based on the adaptive inter-frame smoothing coefficient, corresponding pixel values of a current frame fusion image and a previous frame fusion image are subjected to a weighted operation to generate current frame output pixel values.

[0032] Preferably, the step S4 comprises:

[0033] The synthesized display texture is bound to a target frame buffer of a graphics rendering pipeline as an output texture, and is transmitted to a stereoscopic display device; the stereoscopic display device supports a frame sequence type, a shutter type or a line alternation type stereoscopic image display mode.

[0034] Preferably, the left eye rendering texture, the right eye rendering texture, the user interface texture and the synthesized display texture are all two-dimensional textures readable and writable by the GPU, and the data type is Texture2D <float4>And configured as a continuous memory layout to support multi-thread parallel access and fast processing of image data.

[0035] Preferably, the method further comprises a resource adaptation step, which comprises:

[0036] For each user interface element, a metadata component containing a display mode flag and a scaling factor is configured, the display mode flag is used to indicate the display mode of the interface element in the left eye image and the right eye image, the display mode includes binocular synchronization, left eye display only and right eye display only;

[0037] In the resource preprocessing stage before image rendering, the Canvas type bound by each user interface element is identified based on the editor extension script, and the RectTransform attribute and Canvas Renderer attribute of the interface element are automatically adjusted according to the content in the metadata component; Before image fusion processing, based on the interpupillary distance parameter and the gaze point depth parameter of the user, the horizontal pixel offset of each user interface element in the left image and the right image is calculated , which is used to adjust the position of the interface element in the user interface texture, the formula is:

[0038] ;

[0039] Wherein: The interpupillary distance parameter of the user; The gaze point depth parameter; The distance from the user's eyes to the screen; The scaling factor of the interface element; The calibration parameter.

[0040] In another aspect, the present application provides an image fusion stereoscopic display system based on Unity engine, which is applied to the image fusion stereoscopic display method based on Unity engine as described above, and is characterized in that it comprises:

[0041] A binocular rendering module is configured to configure a left eye camera and a right eye camera in a Unity scene, and to collect left view images and right view images based on a set interpupillary distance parameter, respectively, to generate left eye rendering texture and right eye rendering texture;

[0042] An interface texture generation module is configured to identify the corresponding display mode flag according to the Canvas type bound by each UI element in the user interface, and to generate user interface texture;

[0043] An image fusion module performs an image fusion process, which includes a weighted operation based on a previous frame image and a current frame image, a weight of the weighted operation being determined by an adaptive smoothing coefficient, and an overlay process of a fused image based on Alpha channel information of a user interface texture, to generate a composite display texture;

[0044] A time-continuous fusion module is configured to perform a weighted operation on corresponding pixel values of a previous frame fused image and a current frame fused image in the image fusion module, to realize time-continuous image fusion.

[0045] A resource adaptation module is configured to configure a metadata component containing a display mode mark and a scaling coefficient for each UI element, and identify a Canvas type bound to the UI element through an editor extension script in a resource preprocessing stage, and adjust RectTransform attributes and Canvas Renderer attributes.

[0046] A position control module is configured to calculate a horizontal offset of each UI element in a composite image according to a user interpupillary distance parameter and a gaze point depth parameter before the fusion process, and adjust a position of a corresponding UI element in the interface texture based on the offset.

[0047] A scheduling control module is configured to calculate a number of two-dimensional thread groups according to a target display resolution, and trigger ComputeShader parallel processing of an image fusion task.

[0048] An output module is configured to bind the composite display texture to a target frame buffer of a graphics rendering pipeline, and transmit the composite display texture to a stereoscopic display device supporting frame sequential, shutter or line alternate display modes.

[0049] A computer readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the image fusion stereoscopic display method based on the Unity engine.

[0050] The beneficial effects of this invention are as follows: Through innovative binocular image rendering and user interface (UI) adaptation technology, this invention significantly improves the effect and user experience of 3D stereoscopic display. By configuring left and right eye cameras in a Unity scene and acquiring left and right view images according to set interpupillary distance parameters, this invention can accurately generate left and right eye rendering textures, while supporting dynamic adaptation of UI elements in different Canvas modes. Through image fusion processing using Compute Shader, this invention combines depth maps to perform parallax compensation on the right eye image and calculates fusion weights based on image content features, ensuring accurate alignment and fusion of left and right eye images. Simultaneously, by using the fused image from the previous frame to perform temporally continuous fusion with the current frame, it effectively avoids screen jitter and delay, improving the smoothness and stability of stereoscopic display. Finally, the generated composite display image can adapt to various types of stereoscopic display devices, including frame-sequence, shutter-based, and line-alternating display modes, ensuring broad device compatibility and optimized display effects. Therefore, this invention not only improves the accuracy of image fusion and the adaptation efficiency of UI elements, but also enhances user immersion while optimizing display device compatibility and rendering performance, solving several limitations in existing technologies. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0052] Figure 1 This is a flowchart of the method of the present invention;

[0053] Figure 2 This is a schematic diagram of the overall structure of the system of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0055] Example 1

[0056] like Figure 1 As shown, this is an embodiment of the present invention, which provides an image fusion stereoscopic display method based on the Unity engine, including:

[0057] Step S1: Binocular view and UI texture collection

[0058] Binocular camera configuration: Configure the left eye camera and the right eye camera in the Unity scene, and collect the left view image and the right view image based on the set interpupillary distance parameters.

[0059] Further, duplicate the main camera to generate the left eye camera and the right eye camera, which are symmetrically offset relative to the main camera in the horizontal direction, and the offset distance is an adjustable parameter, with a default value of 0.065 meters and a value range of 0.03 meters to 0.08 meters.

[0060] Adjust the interpupillary distance parameter in real time through the control interface, and dynamically update the position offset of the left eye camera and the right eye camera in the horizontal direction.

[0061] Field of view angle setting: The field of view angles of the left eye camera and the right eye camera are both set to 60 degrees.

[0062] Texture generation: The view images collected by the left eye camera and the right eye camera and the rendering texture, which is a two-dimensional texture readable and writable by GPU, with a data type of Texture2D <float4>

[0063] Step S2: According to the Canvas type bound to each UI element in the user interface, configure the corresponding display mode mark, which is used to indicate the display mode of the UI element in the left eye image and the right eye image, and the display mode includes binocular synchronous display, left eye display only and right eye display only; generate the user interface texture based on the UI rendering process.

[0064] Display mode mark: The display mode mark is used to indicate the display mode of the UI element in the left eye image and the right eye image, and the mark includes: binocular synchronous display, left eye display only and right eye display only.

[0065] Canvas type identification: Based on the display mode mark, the system identifies the Canvas type to which the UI element belongs, and configures different rendering logics according to the different Canvas types:

[0066] When the Canvas type is Overlay mode, generate a separate user interface texture bound to the Render Texture object.

[0067] When the Canvas type is World Space mode or Screen Space-Camera mode, respectively collect images by the left eye camera and the right eye camera and generate a unified user interface texture.

[0068] Rendering parameter configuration of the user interface texture: According to the Canvas type, automatically configure the rendering parameters of the user interface texture, including: rendering priority, scaling factor, occlusion level information, user interface texture generation:

[0069] The user interface texture is a two-dimensional texture readable and writable by GPU, and its data type is Texture2D <float4>, for subsequent image fusion processing.

[0070] Step S3: image fusion processing

[0071] The left-eye rendering texture, the right-eye rendering texture, and the user interface texture are input to the Compute Shader, and image fusion processing is performed, which includes: dynamically calculating the fusion weight of the left and right views based on image content features; performing parallax compensation and pixel alignment on the right-eye image based on the depth map; fusing the left and right eye images to generate a fused image according to the fusion weight; performing weighted operation on the previous frame fused image and the current frame fused image, and the weight of the weighted operation is determined by the adaptive inter-frame smoothing coefficient; superimposing the UI layer according to the transparency of the user interface texture to obtain a synthesized display image;

[0072] Further, the pixel data of the left-eye image, the right-eye image, and the user interface image are input to the Compute Shader, and image fusion processing is performed in the Compute Shader, which includes the following steps:

[0073] Dynamically calculating the fusion weight of the left and right views based on image content features;

[0074] Calculating the parallax compensation displacement of the right-eye image based on the depth map and performing pixel alignment;

[0075] Fusing the aligned left and right images to obtain a fused image;

[0076] Superimposing the user interface image into the fused image according to its transparency information to generate a final display image, and the fusion calculation formula is:

[0077] ;

[0078] Wherein: is the color value of the left-eye image at pixel position ; is the color value of the right-eye image at pixel position ; is the color value of the user interface image at pixel position ; is the Alpha channel value of the user interface image at position , ranging from [0, 1], used to control transparency; is the weight coefficient of left and right image fusion, dynamically calculated based on image content, ranging from [0, 1]; is the parallax compensation displacement of the right-eye image at position , derived from the depth map; is the pixel value after left and right image fusion; To synthesize the final display image at pixel positions with color values.

[0079] In one specific embodiment of the present application, the image fusion processing process is implemented based on Compute Shader in the Unity engine. The image processing module fuses the left eye image, the right eye image and the UI image through GPU parallel computing. The following is a typical Compute Shader program implementation code:

[0080] Texture2D <float4>Input Left Texture; / / left eye image texture

[0081] Texture2D <float4>InputRightTexture; / / right eye image texture

[0082] Texture2D <float4>UITexture; / / model

[0083] SamplerState linearSampler; / / Linear sampler

[0084] RWTexture2D <float4>OutputTexture; / / Output image

[0085] int Width;

[0086] int Height;

[0087] [numthreads(8, 8, 1)]

[0088] void CSMain(uint3 threadID : SV_DispatchThreadID)

[0089] {

[0090] / / Step 1: Synthesize 3D textures line by line

[0091] uint index = threadID.y % 2; / / Parity check

[0092] float4 stereoColor = (index == 1)

[0093] InputLeftTexture[threadID.xy]

[0094] InputRightTexture[threadID.xy];

[0095] / / Step 2: Pattern texture overlay ( mix)

[0096] float2 uv = float2(threadID.x / (float)Width, threadID.y / (float)Height);

[0097] float4 uiColor = UITexture.Sample(linearSampler, uv);

[0098] / / Calculate the final synthesis result

[0099] OutputTexture[threadID.xy] = lerp(stereoColor, uiColor, uiColor.a);

[0100] }

[0101] The above code shows a typical implementation of the present application, which can realize seamless integration of UI and stereoscopic images without sacrificing frame rate and image quality, and adapt to various rendering modes and stereoscopic display devices. The specific implementation process can be adjusted according to the performance requirements of the development platform and display terminal.

[0102] In an embodiment, the time-continuous fusion processing in step S3 specifically includes the following steps:

[0103] Step one: adaptive inter-frame smoothing coefficient calculation

[0104] Obtain the fusion weights, disparity compensation displacements of the current frame and the previous frame, and user visual parameters, including user interpupillary distance parameters and gaze point depth parameters. Based on the preset basic smoothing coefficient, weight change adjustment coefficient, disparity change adjustment coefficient, and user parameter adjustment coefficient, calculate the adaptive inter-frame smoothing coefficient of each pixel position .

[0105] The calculation formula of the adaptive inter-frame smoothing coefficient is:

[0106] Wherein: is the basic smoothing coefficient; and are the fusion weight values of the current frame and the previous frame at pixel ; and are the disparity compensation displacement values of the current frame and the previous frame at pixel ; is the user interpupillary distance parameter; is the user gaze point depth parameter; , , are the adjustment coefficients of image content change, disparity change, and user visual parameters, respectively.

[0107] Step two: time-continuous pixel value weighting operation

[0108] According to the adaptive inter-frame smoothing coefficient, perform weighting operation on the corresponding pixel values of the current frame fusion image and the previous frame fusion image to obtain the current frame output pixel value , and the weighting calculation formula is:

[0109] ;

[0110] Wherein: is the pixel value of the previous frame fusion image at pixel ; is the pixel value of the current frame fusion image at pixel .

[0111] Step three: output pixel value range limitation

[0112] In order to ensure that the output data is within a legal range, the calculated current frame output pixel value is limited in value, so as to be located in a preset pixel effective value range. The limitation formula is as follows:

[0113]

[0114] Through the above steps, the time-continuous inter-frame pixel fusion processing can be completed in the Compute Shader.

[0115] In order to more clearly illustrate the technical solutions of the present application, the following will combine specific computer program codes to describe an embodiment of the image fusion stereoscopic display method of the present application in detail.

[0116] Step S4: write the synthesized display image into a synthesized display texture, and output the synthesized display texture to a stereoscopic display device which supports a frame sequence type, a shutter type or a line alternation type stereoscopic image display mode.

[0117] Further, the step S4 comprises:

[0118] bind the synthesized display texture to a target frame buffer of a graphics rendering pipeline as an output texture, and transmit it to a stereoscopic display device; the stereoscopic display device supports a frame sequence type, a shutter type or a line alternation type stereoscopic image display mode.

[0119] In a further embodiment of the present application, the left eye rendering texture, the right eye rendering texture, the user interface texture and the synthesized display texture are all two-dimensional textures readable and writable by the GPU, and the data type is Texture2D <float4>And configured as a continuous memory layout to support multi-thread parallel access and fast processing of image data.

[0120] The present example further includes a resource adaptation step, which comprises:

[0121] For each user interface element, a metadata component containing a display mode flag and a scaling factor is configured, the display mode flag being used to indicate the display mode of the interface element in the left eye image and the right eye image, the display mode including binocular synchronization, left eye display only and right eye display only;

[0122] In the resource preprocessing stage before image rendering, the Canvas type bound to each user interface element is identified based on the editor extension script, and the RectTransform attribute and Canvas Renderer attribute of the interface element are automatically adjusted according to the content in the metadata component; before image fusion processing, based on the interpupillary distance parameter and the gaze point depth parameter of the user, the horizontal pixel offset of each user interface element in the left image and the right image is calculated , which is used to adjust the position of the interface element in the user interface texture, the formula being:

[0123] ;

[0124] Wherein: is the interpupillary distance parameter of the user; is the gaze point depth parameter; is the distance from the user's eyes to the screen; is the scaling factor of the interface element; is the calibration parameter; is the horizontal offset pixel value of the interface element.

[0125] As Figure 2 shown, another embodiment of the present application provides an image fusion stereoscopic display system based on Unity engine, applied to the image fusion stereoscopic display method based on Unity engine as described above, comprising:

[0126] A binocular rendering module is configured to configure a left eye camera and a right eye camera in the Unity scene, and to collect a left view image and a right view image based on a set interpupillary distance parameter respectively, to generate a left eye rendering texture and a right eye rendering texture;

[0127] An interface texture generation module is configured to identify the corresponding display mode flag according to the Canvas type bound to each UI element in the user interface, and to generate a user interface texture;

[0128] The image fusion module performs image fusion processing, which includes weighted operation based on the previous frame image and the current frame image, the weight of the weighted operation being determined by an adaptive smoothing coefficient, and superposition processing of the fused image based on the Alpha channel information of the user interface texture to generate a composite display texture.

[0129] The time-continuous fusion module is configured to perform weighted operation on corresponding pixel values of the previous frame fused image and the current frame fused image in the image fusion module to realize time-continuous image fusion.

[0130] The resource adaptation module is configured to configure a metadata component containing a display mode tag and a scaling coefficient for each UI element, and identify the Canvas type bound to the UI element through an editor extension script in the resource preprocessing stage, and adjust the RectTransform attribute and the Canvas Renderer attribute.

[0131] The position control module is configured to calculate the horizontal offset of each UI element in the composite image according to the user interpupillary distance parameter and the gaze point depth parameter before the fusion processing, and adjust the position of the corresponding UI element in the interface texture based on the offset.

[0132] The scheduling control module is configured to calculate the number of two-dimensional thread groups according to the target display resolution, and trigger ComputeShader parallel processing of the image fusion task.

[0133] The output module is configured to bind the composite display texture to the target frame buffer of the graphics rendering pipeline, and transmit it to a stereoscopic display device supporting frame sequential, shutter or line alternate display modes.

[0134] A computer readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the image fusion stereoscopic display method based on the Unity engine as described above.

[0135] Embodiment 2

[0136] The specific implementation mode provided in this embodiment is designed to support UI display fusion under different Canvas modes, and introduces an adjustable interpupillary distance parameter to enhance the individualized control ability of stereoscopic vision experience.

[0137] In terms of stereoscopic rendering, the system first creates a left eye camera (Left Camera) and a right eye camera (Right Camera) by copying the main camera. The two cameras are symmetrically offset in the X-axis direction relative to the main camera, and the offset distance is dynamically controlled by the inter Pupillary Distance parameter. The default value of this parameter is 0.065 meters, and users can adjust it in real time through a slider control during runtime, with a range of 0.03 to 0.08 meters. Both left and right eye cameras are bound to their respective RenderTexture objects (Left RT, Right RT), which serve as Input Left Texture and Input Right Texture inputs in subsequent image fusion processing.

[0138] In terms of UI image acquisition and fusion, the system supports UI configuration in Overlay mode and World Space mode. For Overlay mode, a Canvas node named UI Overlay Canvas is created, and user interface elements such as buttons and health bars are added under it. The overall Canvas is bound to the Render Texture (UI Overlay RT) to generate the UITexture texture. For World Space mode, a UI World Canvas is created and set to World Space display mode. It is placed in the three-dimensional scene, such as 1 meter in front of the character, and the Culling Mask of the left and right eye cameras includes the layer where the UI is located, ensuring complete acquisition of the UI texture in 3D space.

[0139] In terms of image fusion processing, the system uses Compute Shader to implement line-by-line alternating left and right image fusion processing, and superimposes UI texture information in the fused image. The core Shader logic is defined in the StereoDisplay.compute file, using a two-dimensional thread structure of [numthreads(8,8,1)]. In the CSMain main function, the left and right image source pixels are switched by judging the parity of the pixel row number, realizing the line interlaced synthesis of 3D stereoscopic images. At the same time, the pixel color and Alpha channel value of the corresponding position in the UI texture are sampled, and based on the lerp function, they are mixed according to the transparency and output to the target texture OutputTexture, realizing the seamless fusion of the UI layer and the 3D image.

[0140] To improve development efficiency and UI resource adaptation capability, the system introduces a metadata component named Stereo UI Meta, which allows users to label the stereoscopic display mode (binocular synchronization, left eye only, right eye only) and scaling factor of each UI element. At the same time, the editor extension script StereoUIEditor.cs is developed to support the "one-click adaptation Canvas mode" function, which automatically identifies the Canvas type of UI elements and adjusts the RectTransform and CanvasRenderer parameters to ensure correct rendering and fusion of UI elements in different Canvas modes.

[0141] This embodiment solves the resource adaptation and fusion problem in different UI display modes while maintaining high real-time performance and high frame rate rendering efficiency. It realizes personalized pupil distance control and multi-device compatible display, providing a good application foundation and scalability for virtual reality, augmented reality, and three-dimensional interactive systems.

[0142] Embodiment 3: Metadata-based UI intelligent adaptation and runtime pupil distance dynamic control method

[0143] To achieve automatic adaptation of UI elements in different Canvas display modes and improve the interactivity and personalized experience of stereoscopic display systems, this embodiment designs and deploys a metadata-based UI adaptation mechanism and runtime pupil distance control function in the Unity platform. This solution is suitable for various rendering modes and display terminals, and has strong versatility and engineering practicality.

[0144] 1. UI display mode metadata identification mechanism

[0145] First, add a custom component StereoUIMeta to each UI element. This component inherits from the MonoBehavior class of the Unity engine and defines an enumeration type for identifying the display mode of UI elements in left and right eye images. The display mode supports three types: binocular synchronous display, left eye only display, and right eye only display. The typical implementation code is as follows:

[0146] / / StereoUIMeta.cs — UI metadata component definition

[0147] public class StereoUIMeta : MonoBehaviour

[0148] {

[0149] / / Enum type: definition Element's visible range

[0150] public enum DisplayMode

[0151] {

[0152] BothEyes, / / Both eyes, both images

[0153] LeftOnly, / / Left eye only, left image

[0154] RightOnly / / Right eye only, right image

[0155] }

[0156] / / Current display mode of the element, default is both eyes

[0157] public DisplayMode displayMode = DisplayMode.BothEyes;

[0158] }

[0159] This component provides independent metadata tags for each UI element, which can be used to determine whether to draw it to the left eye image, the right eye image, or both images during rendering.

[0160] 2. UI element automatically adapts to Canvas type editor extension

[0161] To simplify the configuration process of UI resources in different Canvas modes and improve development efficiency, the system introduces the editor extension script StereoUIEditor.cs, which identifies the RenderMode attribute of the Canvas attached to the UI element and implements the one-key adaptation of UI parameters. The core code is as follows:

[0162] / / StereoUIEditor.cs — Adaptation editor extension

[0163] [CustomEditor(typeof(StereoUIMeta))]

[0164] public class StereoUIEditor : Editor

[0165] {

[0166] public override void OnInspectorGUI()

[0167] {

[0168] base.OnInspectorGUI();

[0169] StereoUIMeta meta = (StereoUIMeta)target;

[0170] if (GUILayout.Button("Auto Adapt to Canvas Mode"))

[0171] {

[0172] Canvas canvas = meta.GetComponentInParent <canvas>();

[0173] switch (canvas.renderMode)

[0174] {

[0175] case RenderMode.Overlay:

[0176] meta.GetComponent <recttransform>().localScale = Vector3.one * 1.2f;

[0177] meta.GetComponent <canvasrenderer>().cull = false;

[0178] break;

[0179] case RenderMode.WorldSpace:

[0180] meta.GetComponent <recttransform>().position = new Vector3(0, 1.5f,5f);

[0181] break

[0182] }

[0183] }

[0184] }

[0185] }

[0186] This mechanism achieves compatibility with multiple Canvas display modes by automatically adjusting the coordinates, scaling, and occlusion properties of UI elements, effectively reducing the cost of manual debugging.

[0187] 3. Dynamic adjustment control logic for interpupillary distance parameter during runtime

[0188] To enhance the system's visual adaptability to different users (such as children and people with myopia), the system provides dynamic control of the interpupillary distance (IPD) parameter. The IPD field is defined in the core controller script StereoDisplayController.cs, and the positions of the left and right eye cameras are adjusted in real time during runtime, as follows:

[0189] / / StereoDisplayController.cs — Interpupillary Distance Controller

[0190] public class StereoDisplayController : MonoBehaviour

[0191] {

[0192] public float interPupillaryDistance = 0.065f; / / Initial interpupillary distance, in meters

[0193] public Transform leftCameraTrans;

[0194] public Transform rightCameraTrans;

[0195] void Update()

[0196] {

[0197] float halfIPD = interPupillaryDistance / 2;

[0198] leftCameraTrans.localPosition = new Vector3(-halfIPD, 0, 0);

[0199] rightCameraTrans.localPosition = new Vector3(halfIPD, 0, 0);

[0200] }

[0201] }

[0202] 4. User interactive IPD adjustment interface binding

[0203] On the UI level, create an interactive IPD adjustment interface with the Slider component, and bind the Slider's sliding value to the IPD parameter in the controller through the script IPDAdjuster.cs to realize user-defined adjustment functions. The implementation code is as follows:

[0204] / / IPDAdjuster.cs — Sliding adjuster

[0205] public class IPDAdjuster : MonoBehaviour

[0206] {

[0207] public StereoDisplayController stereoController; / / Controller reference

[0208] public Slider ipdSlider; / / Component reference

[0209] void Start()

[0210] {

[0211] / / Sliding event binding: real-time update of IPD value

[0212] ipdSlider.onValueChanged.AddListener((value) => {

[0213] stereoController.interPupillaryDistance = value;

[0214] });

[0215] }

[0216] }

[0217] The user can flexibly set the pupil distance value between 0.03 and 0.08 meters by sliding the Slider control, and the system automatically synchronously updates the camera position, thereby changing the binocular disparity in real time and enhancing the personalized experience of stereoscopic perception.

[0218] The embodiment introduces a metadata tagging mechanism, editor extension and runtime interaction logic, and constructs a complete "UI intelligent adaptation + pupil distance dynamic control" solution, achieving the following technical effects:

[0219] The UI elements are automatically adapted in different Canvas modes, improving development efficiency;

[0220] The user can individually adjust the pupil distance value, improving the comfort of stereoscopic vision;

[0221] The system has good extensibility and running stability, and is suitable for multiple platforms and multiple resolution devices.

[0222] The solution significantly improves the interactivity, configurability and human-computer adaptation capability of the stereoscopic display system, and has high practical value and engineering landing nature.

[0223] In summary, the present application solves multiple technical problems in the prior art, especially in the cooperative work of 3D stereoscopic display and UI elements, resource adaptation efficiency, rendering performance and individual adjustment. Through innovative image fusion processing, UI automatic adaptation and GPU acceleration technology, the present application can provide efficient 3D display effect and good user experience, especially suitable for virtual reality (VR), augmented reality (AR), game development and digital twin fields. In addition, the present application has strong hardware adaptation capability and can be compatible with different types of stereoscopic display devices, further expanding the application scenarios and market prospects. In general, the present application has wide application value and good commercialization prospects, can promote the technical development of related fields, and provide innovative solutions for related industries.

[0224] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0225] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0226] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / recttransform> < / canvasrenderer> < / recttransform> < / canvas> ​

Claims

1. A method for image fusion and stereoscopic display based on the Unity engine, characterized in that, The method includes: Step S1: Configure the left-eye camera and the right-eye camera in the Unity scene, and capture the left-view image and the right-view image respectively based on the set interpupillary distance parameters, and generate the left-eye rendering texture and the right-eye rendering texture. Step S2: Configure the corresponding display mode flag according to the Canvas type bound to each UI element in the user interface. The display mode flag is used to indicate the display mode of the UI element in the left-eye image and the right-eye image. The display mode includes simultaneous display of both eyes, display of only the left eye, and display of only the right eye; generate the user interface texture based on the UI rendering process; Step S3: Input the left-eye rendering texture, right-eye rendering texture, and user interface texture into the Compute Shader and perform image fusion processing. The fusion processing includes: dynamically calculating the fusion weights of the left and right views based on image content features; performing disparity compensation and pixel alignment on the right-eye image based on the depth map; fusing the left and right eye images according to the fusion weights to generate a fused image; performing a weighted operation on the fused image of the previous frame and the fused image of the current frame, wherein the weights of the weighted operation are determined by an adaptive inter-frame smoothing coefficient; and overlaying a UI layer according to the transparency of the user interface texture to obtain a composite display image. Step S3 further includes: Obtain the fusion weight change, disparity compensation displacement change, and user visual parameters between the current frame and the previous frame; calculate the adaptive inter-frame smoothing coefficient for each pixel position based on the base smoothing coefficient, weight change adjustment coefficient, disparity change adjustment coefficient, and user parameter adjustment coefficient. Based on the adaptive inter-frame smoothing coefficient, the corresponding pixel values ​​of the current frame fused image and the previous frame fused image are weighted and calculated to generate the current frame output pixel value. Step S4: Write the composite display image into the composite display texture and output the composite display texture to the stereoscopic display device. The stereoscopic display device supports frame sequence, shutter, or line alternation stereoscopic image display modes.

2. The image fusion stereoscopic display method based on the Unity engine according to claim 1, characterized in that, Step S1 includes: The main camera is copied to generate a left-eye camera and a right-eye camera. The left-eye camera and the right-eye camera are symmetrically offset from the main camera in the horizontal direction, and the offset distance is an adjustable parameter. The interpupillary distance parameter value is adjusted in real time through the control interface, and the horizontal positional offset between the left and right eye cameras is dynamically updated. The field of view of both the left-eye and right-eye cameras is set to 60 degrees. The left-eye camera and the right-eye camera respectively capture images from the left and right perspectives, and render them to generate left-eye rendering textures and right-eye rendering textures. The rendering textures are two-dimensional textures that can be read and written by the GPU.

3. The image fusion stereoscopic display method based on the Unity engine according to claim 1, characterized in that, Step S2 includes: The UI element's Canvas type is identified based on the display mode marker. When the Canvas type is Overlay mode, an independent user interface texture is generated and bound to the Render Texture object. When the Canvas type is World Space mode or Screen Space-Camera mode, images are captured by the left and right eye cameras respectively and a unified user interface texture is generated. The rendering parameters of the user interface texture are automatically configured according to the Canvas type. The rendering parameters include rendering priority, scaling factor and occlusion level information.

4. The image fusion stereoscopic display method based on the Unity engine according to claim 1, characterized in that, Step S3 includes: The left-eye rendered texture, the right-eye rendered texture, and the user interface texture are input into the Compute Shader. Image fusion processing is performed in the Compute Shader, which includes the following steps: The fusion weights of the left and right views are dynamically calculated based on image content features; The disparity compensation displacement of the right eye image is calculated based on the depth map and pixel alignment is performed. The aligned left and right images are then merged to obtain a merged image; The user interface image is overlaid onto the blended image according to its transparency information to generate the final display image. The blending calculation formula is as follows: ； in: For the left eye image at pixel location The color value; The right eye image at pixel location The color value; User interface image at pixel position The color value; User interface image at pixel position Alpha channel value; These are the weighting coefficients for fusing the left and right images; The right eye image at pixel location Parallax compensation displacement at the location; These are the pixel values ​​after fusing the left and right images; To synthesize and display images at pixel locations The color value.

5. The image fusion stereoscopic display method based on the Unity engine according to claim 1, characterized in that, Step S4 includes: The composite display texture is bound as an output texture to the target frame buffer of the graphics rendering pipeline and transmitted to the stereoscopic display device; the stereoscopic display device supports frame-sequence, shutter-type, or line-alternating stereoscopic image display modes.

6. The image fusion stereoscopic display method based on the Unity engine according to claim 1, characterized in that, The left-eye rendering texture, right-eye rendering texture, user interface texture, and composite display texture are all GPU-readable and writable two-dimensional textures of data type Texture2D. <float4> It is configured as a contiguous memory layout to support multi-threaded parallel access and fast processing of image data.

7. The image fusion stereoscopic display method based on the Unity engine according to claim 1, characterized in that, The method includes a resource adaptation step, which includes: Configure a metadata component for each user interface element, which includes a display mode flag and a scaling factor. The display mode flag is used to indicate how the interface element is displayed in the left-eye image and the right-eye image. The display modes include binocular synchronization, left-eye-only display, and right-eye-only display. In the resource preprocessing stage before image rendering, the editor extension script identifies the Canvas type bound to each user interface element and automatically adjusts the RectTransform and Canvas Renderer properties of the interface elements according to the content in the metadata component. Before image fusion processing, the horizontal pixel offset of each user interface element in the left and right images is calculated based on the user's interpupillary distance and gaze depth parameters. This is used to adjust the position of interface elements within the user interface texture. The formula is: ； in: For user interpupillary distance parameters; For the depth parameter of the gaze point; The distance from the user's eyes to the screen; This is the scaling factor for the interface elements; These are calibration parameters.

8. A Unity engine-based image fusion stereoscopic display system, applied to the Unity engine-based image fusion stereoscopic display method as described in any one of claims 1-7, characterized in that, The system includes: The binocular rendering module is used to configure the left and right eye cameras in a Unity scene, and to acquire left and right view images respectively based on the set interpupillary distance parameters, generating left and right eye rendering textures. The interface texture generation module is used to identify the corresponding display mode marker based on the Canvas type bound to each UI element in the user interface, and generate the user interface texture. The image fusion module is used to perform image fusion processing, which includes a weighted operation based on the previous frame image and the current frame image. The weights of the weighted operation are determined by an adaptive smoothing coefficient. The fused image is then overlaid based on the Alpha channel information of the user interface texture to generate a synthetic display texture. The temporal continuous fusion module is used to perform weighted calculations on the corresponding pixel values ​​of the fused image from the previous frame and the fused image from the current frame in the image fusion module; The resource adaptation module is used to configure metadata components containing display mode tags and scaling factors for each UI element, and to identify the Canvas type bound to the UI element through the editor's extended script during the resource preprocessing stage, and adjust the RectTransform property and Canvas Renderer property. The position adjustment module is used to calculate the horizontal offset of each UI element in the synthesized image based on the user's interpupillary distance parameters and gaze depth parameters before fusion processing, and adjust the position of the corresponding UI element in the interface texture based on the offset. The scheduling and control module is used to calculate the number of two-dimensional thread groups based on the target display resolution and trigger the ComputeShader to process the image fusion task in parallel. The output module is used to bind the synthesized display texture to the target frame buffer of the graphics rendering pipeline and transmit it to a stereoscopic display device that supports frame sequence, shutter, or line alternation display modes.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the image fusion stereoscopic display method based on the Unity engine as described in any one of claims 1 to 7.

Images