A mixed reality virtual-real fusion method, device and system

Abstract

The present application relates to the field of machine vision and image fusion, more particularly to a mixed reality virtual-real fusion method, device and system, and a medium. The method comprises: acquiring a real-world RGB image and a depth image, and acquiring a virtual image and a virtual depth; comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on a history frame of the RGB image and the occlusion relationship graph; performing edge-preserving filtering on the color model using a latest frame of the RGB image to obtain a final fusion weight map; and performing weighted fusion on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image. The present application uses a color modeling and edge-preserving filtering method to calculate image virtual-real fusion parameters, which not only ensures correct matching of the depth image and the RGB image, but also effectively reduces display delay.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image fusion technology, and more specifically, to a mixed reality fusion method, apparatus, system, and medium. Background Technology

[0002] Mixed Reality (MR) technology enhances the realism of the user experience by presenting virtual scene information in a real scene, creating an interactive feedback loop between the real world, the virtual world, and the user.

[0003] Mixed reality systems require a high degree of realism in the display of virtual objects, with the aim of seamlessly integrating virtual and real objects to maintain the immersiveness of the scene and the realism of the virtual objects. This requirement also presents new challenges to display technology.

[0004] In existing mixed reality system architectures, virtual objects are rendered using rendering techniques in virtual-real fusion scenes. To determine the occlusion relationship between virtual and real objects during virtual-real integration, the following three display strategies are typically employed:

[0005] 1) Virtual objects always appear in front of real objects;

[0006] 2) Try to obtain the most accurate depth information possible by directly comparing the depths of real and virtual objects;

[0007] 3) Extract certain information from images of real objects and combine it with incompletely precise depth information for image matting.

[0008] The first display strategy results in an unrealistic display effect and a poor user experience.

[0009] For the second display strategy, the computational cost of processing the depth map is extremely high, potentially making real-time computation impossible. Current algorithms average 48.4ms, with a display latency of approximately 116ms, meaning they can only support a maximum of 20 FPS for virtual-real image processing, and the effect exhibits a noticeable delay perceptible to the human eye.

[0010] For the third display strategy, existing algorithms typically require the depth map and RGB image to be acquired at the same time. However, in wearable devices, due to power and heat limitations, the latency of acquiring the depth map is usually much higher than that of acquiring the RGB image, and the frame rate is often lower than that of the RGB image. Therefore, the algorithm struggles to use the latest RGB image, which also increases the overall latency of the display system.

[0011] Based on the above analysis, the existing technology has poor display effect of virtual and real scene fusion in mixed reality systems, and the delay is difficult to solve effectively. Summary of the Invention

[0012] The purpose of this invention is to provide a mixed reality fusion method, device, system, and medium to solve the problems of poor display effect and large latency in existing virtual reality fusion technologies.

[0013] To achieve the above objectives, the present invention provides a mixed reality virtual-real fusion method, comprising the following steps:

[0014] Acquire real-world RGB and depth images, as well as virtual images and virtual depth;

[0015] Compare depth images and virtual depth to generate occlusion maps, and generate color models based on historical frames of RGB images and occlusion maps;

[0016] The latest frame of the RGB image is used to perform edge-preserving filtering on the color model to obtain the final fused weight map;

[0017] Based on the final fusion weight map, the real-world RGB image and the virtual image are weighted and fused to generate the display image.

[0018] In some embodiments, the comparison of the depth image and the virtual depth to generate an occlusion relationship map, and the generation of a color model based on historical frames of the RGB image and the occlusion relationship map, further include:

[0019] Select the latest frame of the depth image, the virtual depth of the historical frames of the virtual image, and the historical frames of the RGB image;

[0020] The virtual depth of the depth image and the virtual image are compared, and the occlusion relationship is marked in the historical frames of the RGB image to obtain the first occlusion relationship map;

[0021] Select historical frames of the RGB image and divide them into blocks. Generate color models for different occluded areas of each block of the image according to the first occlusion relationship diagram.

[0022] In some embodiments, the step of performing edge-preserving filtering on the color model using the latest frame of the RGB image to obtain the final fusion weight map further includes:

[0023] Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frames of the RGB image, and obtain the occlusion probability map;

[0024] Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain the first weight map;

[0025] The first weight map is used as the final fusion weight map.

[0026] In some embodiments, obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes:

[0027] Obtain the luminance component of the latest frame of the RGB image, and use the luminance component to perform guided filtering on the occlusion probability map to obtain the first weight map.

[0028] In some embodiments, obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes:

[0029] The latest frame of the RGB image is obtained, and joint bilateral filtering is performed on the occlusion probability map to obtain the first weight map.

[0030] In some embodiments, the mixed reality virtual-real fusion method further includes the following steps:

[0031] A morphological transformation is performed on the first occlusion relationship graph to obtain the second weight graph;

[0032] The first and second weight maps are weighted and merged to obtain the third weight map, which is used as the final merged weight map.

[0033] In some embodiments, acquiring real-world RGB images and depth images further includes: continuously acquiring real-world RGB images, depth images, and exposure timestamps at a preset frame rate;

[0034] The acquisition of virtual image and virtual depth further includes: continuously acquiring virtual image and virtual depth of virtual image at a preset frame rate.

[0035] In some embodiments, after acquiring the real-world RGB image, depth image, and virtual image and virtual depth, the process further includes: downsampling all acquired images to a low-resolution image.

[0036] Before weighted fusion of the real-world RGB image and the virtual image based on the final fusion weight map to generate the display image, the method further includes: upsampling the final fusion weight map to the display resolution.

[0037] In some embodiments, bilinear interpolation is used to downsample the acquired RGB image and virtual image.

[0038] In some embodiments, the acquired depth image and virtual depth are downsampled using nearest neighbor interpolation.

[0039] In some embodiments, marking occlusion relationships in the RGB image further includes:

[0040] The RGB image is divided into three parts: a virtual image display area, a real image display area, and a region to be determined.

[0041] The virtual image region will be marked as the first coefficient;

[0042] The region that will display the true image will be marked as the second coefficient;

[0043] The region to be determined is marked as the third coefficient.

[0044] In some embodiments, marking occlusion relationships in the RGB image further includes:

[0045] Regions with missing depth are marked as undetermined regions;

[0046] Regions lacking virtual depth will not be marked.

[0047] In some embodiments, performing a morphological transformation on the first occlusion relationship map to obtain a second weight map further includes:

[0048] Morphological dilation is performed on the undetermined region of the first occlusion relationship map to obtain the second occlusion relationship map;

[0049] Edge processing is performed on the second occlusion relationship graph to obtain the second weight graph.

[0050] In some embodiments, the step of performing edge processing on the second occlusion relationship map to obtain a second weight map further includes:

[0051] The second occlusion relationship map is filtered and ternary processed to obtain the third occlusion relationship map.

[0052] The third occlusion relationship graph is filtered to obtain the second weight graph.

[0053] In some embodiments, the color model includes a color histogram;

[0054] The step of generating color models for different occluded regions in each image further includes: performing color histogram statistics on the unoccluded and occluded regions in each image.

[0055] In some embodiments, color depth is downsampled when performing color histogram statistics.

[0056] In some embodiments, selecting the latest frame of the RGB image and estimating the probability of pixel occlusion based on the color model of historical frames of the RGB image to obtain an occlusion probability map further includes:

[0057] Morphological dilation is performed on the undetermined region of the first occlusion relationship map to obtain the second occlusion relationship map;

[0058] Select the latest frame of the RGB image, and based on the location of the undetermined region in the second occlusion relationship map, estimate the occlusion probability of each pixel in the corresponding region of the latest frame of the RGB image to obtain the occlusion probability map.

[0059] In some embodiments, the step of estimating the occlusion probability of each pixel in the corresponding region of the latest frame of the RGB image based on the location of the undetermined region in the second occlusion relationship map to obtain an occlusion probability map further includes:

[0060] Select several image blocks from the historical frames of the RGB image that are closest to the current pixel;

[0061] Based on the color models of several image blocks, estimate the color model of the region adjacent to the current pixel;

[0062] Based on the estimated color model, calculate the distribution of the current pixel's color in different occlusion regions;

[0063] Estimate the probability that the current pixel is occluded based on the distribution;

[0064] By applying regularization constraints to the probability values, an occlusion probability map is obtained.

[0065] In some embodiments, the luminance component further includes the Y channel in YUV space, the L channel in Lab space, the I channel in HSI space, and the L channel in HSL space.

[0066] To achieve the above objectives, the present invention provides a mixed reality fusion device, comprising an image information processor, a depth information processor, a central processing unit, and a display processing unit:

[0067] The image information processor is connected to the central processing unit and the display processing unit, and is used to acquire real-world RGB images and send them to the central processing unit and the display processing unit.

[0068] The depth information processor is connected to the central processing unit and is used to acquire depth images of the real scene and send them to the central processing unit.

[0069] The central processing unit is connected to the display processing unit and is used to obtain virtual depth, execute the mixed reality virtual-real fusion method as described in any of the above, and send the final fusion weight map to the display processing unit;

[0070] The display processing unit acquires the final fusion weight map sent by the central processing unit, performs weighted fusion of the real-world RGB image and the virtual image, and generates a display image.

[0071] In one embodiment, the image information processor is connected to a scene camera, acquires the data collected by the scene camera, and generates an RGB image;

[0072] The depth information processor is connected to the depth sensor, acquires the data collected by the depth sensor, and generates a depth image.

[0073] To achieve the above objectives, the present invention provides a mixed reality fusion system, comprising a mixed reality fusion device, a scene camera, a depth sensor, a graphics processor, and a display device:

[0074] The mixed reality virtual-real fusion device is a mixed reality virtual-real fusion device as described in any of the above-mentioned items;

[0075] The scene camera is used to capture RGB images of the real scene and send them to the hybrid display virtual and real device;

[0076] The depth sensor is used to acquire depth images of the real scene and send them to the hybrid display virtual-real device;

[0077] The image processor is used to generate virtual images and virtual depth and send them to the hybrid display virtual-real device;

[0078] The display device is used to receive and display the display image generated by the hybrid display virtual and real device.

[0079] To achieve the above objectives, the present invention provides a computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the method described in any of the preceding claims is performed.

[0080] This invention provides a mixed reality fusion method, apparatus, system, and medium. It uses color modeling and edge-preserving filtering to calculate image fusion parameters, taking into account both historical and current RGB images to ensure correct matching between depth and RGB images, while also utilizing the latest image information to effectively reduce display latency. Attached Figure Description

[0081] The above and other features, properties and advantages of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings and embodiments, in which the same reference numerals always denote the same features, wherein:

[0082] Figure 1 A flowchart of a hybrid reality virtual-real fusion method according to an embodiment of the present invention is disclosed;

[0083] Figure 2 A flowchart of a mixed reality virtual-real fusion method based on color model and guided filtering according to a first embodiment of the present invention is disclosed;

[0084] Figure 3 A schematic diagram of image segmentation according to a first embodiment of the present invention is disclosed;

[0085] Figure 4 A flowchart of a mixed reality virtual-real fusion method based on color model and guided filtering according to a second embodiment of the present invention is disclosed;

[0086] Figure 5 A flowchart of the mixed reality virtual-real fusion algorithm based on color model and guided filtering according to the second embodiment of the present invention is disclosed;

[0087] Figure 6 An algorithm block diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention is disclosed;

[0088] Figure 7 A hardware schematic diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention is disclosed.

[0089] The meanings of the labels in the figures are as follows:

[0090] 610 image information processor;

[0091] 620 depth information processor;

[0092] 630 Central Processing Unit;

[0093] 640 display processing unit;

[0094] 650 scene camera;

[0095] 660° depth sensor;

[0096] 670 graphics processor. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0098] Figure 1 A flowchart of a mixed reality virtual-real fusion method according to an embodiment of the present invention is disclosed, such as... Figure 1 As shown, the present invention provides a mixed reality virtual-real fusion method, including the following steps:

[0099] Step S1: Obtain the real-world RGB image and depth image, as well as the virtual image and virtual depth;

[0100] Step S2: Compare the depth image and virtual depth to generate an occlusion relationship map, and generate a color model based on the historical frames of the RGB image and the occlusion relationship map;

[0101] Step S3: Use the latest frame of the RGB image to perform edge-preserving filtering on the color model to obtain the final fusion weight map;

[0102] Step S4: Based on the final fusion weight map, the real-world RGB image and the virtual image are weighted and fused to generate the display image.

[0103] Furthermore, step S2, comparing the depth image and the virtual depth to generate an occlusion relationship map, and generating a color model based on historical frames of the RGB image and the occlusion relationship map, further includes:

[0104] Select the latest frame of the depth image, the virtual depth of the historical frames of the virtual image, and the historical frames of the RGB image;

[0105] The virtual depth of the depth image and the virtual image are compared, and the occlusion relationship is marked in the historical frames of the RGB image to obtain the first occlusion relationship map;

[0106] Select historical frames of the RGB image and divide them into blocks. Generate color models for different occluded areas of each block of the image according to the first occlusion relationship diagram.

[0107] Furthermore, step S3, applying edge-preserving filtering to the color model using the latest frame of the RGB image to obtain the final fusion weight map, further includes:

[0108] Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frames of the RGB image, and obtain the occlusion probability map;

[0109] Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain the first weight map;

[0110] The first weight map is used as the final fusion weight map.

[0111] Edge-preserving filtering methods employ edge-preserving filters to achieve filtering. An edge-preserving filter is a special type of filter that effectively preserves edge information in an image during the filtering process.

[0112] Among them, the joint bilateral filter, the guided image filter, and the weighted least square filter are some of the more well-known edge-preserving filters.

[0113] The following example uses a boundary-preserving filter as the guiding filter to illustrate the hybrid reality virtual-real fusion method proposed in this invention. It should be noted that other boundary-preserving filters can also be used to achieve the hybrid reality virtual-real fusion method proposed in this invention.

[0114] Figure 2 A flowchart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a first embodiment of the present invention is disclosed, as follows: Figure 2 In the first embodiment shown, the mixed reality virtual-real fusion method based on color model and guided filtering proposed in this invention specifically includes the following steps:

[0115] Step S101: Obtain the RGB image and depth image of the real scene, and obtain the virtual image and virtual depth;

[0116] Step S102: Select the latest frame of the depth image, the virtual depth of the historical frames of the virtual image, and the historical frames of the RGB image. Compare the virtual depth of the depth image and the virtual depth of the virtual image. Mark the occlusion relationship in the historical frames of the RGB image to obtain the first occlusion relationship map.

[0117] Step S103: Select historical frames of the RGB image and divide them into blocks. Generate color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram.

[0118] Step S104: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frames of the RGB image, and obtain the occlusion probability map.

[0119] Step S105: Obtain the luminance component of the latest frame of the RGB image, and use the luminance component to perform guided filtering on the occlusion probability map to obtain the first weight map;

[0120] Step S106: Use the first weight map as the final fused weight map;

[0121] Step S107: Based on the final fusion weight map, perform weighted fusion on the real-world RGB image and the virtual image to generate a display image.

[0122] These steps will be described in detail below. It should be understood that, within the scope of this invention, the above-described technical features of this invention and the technical features specifically described below (such as in the embodiments) can be combined and related to each other to form preferred technical solutions.

[0123] Step S101: Obtain the real-world RGB image and depth image, as well as the virtual image and virtual depth.

[0124] Depth information refers to the distance between objects in a real scene and the camera. If the camera's shooting plane is used as the X and Y axes of a three-dimensional coordinate system, then depth represents the distance along the Z axis. Therefore, depth information can be used to determine occlusion relationships.

[0125] In some embodiments, RGB images, depth maps, and corresponding exposure timestamps of a real scene are continuously acquired at a preset frame rate. Preferably, the preset frame rate is a fixed frame rate. In other embodiments, the preset frame rate allows for a certain range of latency but cannot exceed a latency threshold.

[0126] In some embodiments, virtual images and their virtual depth (Z-buffer) are continuously acquired at a preset frame rate. Preferably, the preset frame rate is a fixed frame rate. In other embodiments, the preset frame rate allows for a certain range of latency but cannot exceed a latency threshold.

[0127] In this embodiment, virtual images and their depth are continuously acquired from the graphics processor. A virtual image refers to a rendered image provided by the graphics processor.

[0128] Optionally, after acquiring all images (RGB image, depth image, and virtual image and virtual depth), all acquired images are downsampled to a lower resolution to meet the computational requirements and ensure subsequent running efficiency.

[0129] Clearly, the "lower" resolution here is a relative concept relative to computing power. When an image is downsampled to a resolution that meets the computing power requirements, it can be considered a lower resolution.

[0130] Downsampling methods include, but are not limited to, bilinear interpolation and nearest neighbor interpolation.

[0131] Preferably, bilinear interpolation is used for downsampling of the acquired RGB image and virtual image;

[0132] Preferably, the acquired depth map and virtual depth are downsampled using nearest neighbor interpolation.

[0133] Step S102: Select the latest frame of the depth image, the virtual depth of the historical frames of the virtual image, and the historical frames of the RGB image. Compare the virtual depth of the depth image and the virtual depth of the virtual image. Mark the occlusion relationship in the historical frames of the RGB image to obtain the first occlusion relationship map.

[0134] In this embodiment, the historical frame of the virtual image further refers to the historical frame in the virtual image whose exposure time is closest to the latest frame of the depth image.

[0135] In this embodiment, the historical frame of the RGB image further refers to the historical frame in the RGB image whose exposure time is closest to the latest frame of the depth image, denoted as the first RGB image.

[0136] Obviously, the exposure time can be determined by the exposure timestamp obtained in step S101.

[0137] In this embodiment, the first occlusion relationship diagram is a first tripartite diagram.

[0138] The first tripartite division divides the image into three parts: a region displaying a virtual image, a region displaying a real image, and a region to be determined.

[0139] The virtual image area, i.e. the unobstructed area of ​​the virtual image, is marked as the first coefficient, which is 0 in this embodiment.

[0140] The real image area is displayed, that is, the area where the virtual image is occluded, and is marked as the second coefficient. In this embodiment, the second coefficient is 1.

[0141] The undetermined area, which is the area where it is not yet possible to determine whether to display a virtual image or a real image, is marked as the third coefficient. In this embodiment, the third coefficient is 0.5.

[0142] Furthermore, regions with missing depth are marked as pending.

[0143] Furthermore, regions lacking virtual depth are left unmarked.

[0144] Step S103: Select historical frames of the RGB image and divide them into blocks. Generate color models for different occlusion areas of each block of the image according to the first occlusion relationship map.

[0145] In this embodiment, selecting historical frames of the RGB image and dividing them into blocks further includes:

[0146] The first RGB image is selected based on the exposure timestamp and the exposure time of the latest frame of the depth image. The first RGB image is then divided into several non-overlapping blocks.

[0147] Furthermore, the first RGB image is divided into several non-overlapping blocks. There are not many restrictions on the specific division method. The following are two examples of preferred implementation methods, but it should not be assumed that the division method proposed in this invention is limited to these two:

[0148] For example, the entire image can be divided into a fixed number of rectangular blocks, such as 8 blocks horizontally and 8 blocks vertically, for a total of 8*8=64 blocks, with no overlap between the blocks.

[0149] For example, the entire image can be divided into rectangular blocks of a fixed size, such as 32-pixel x 32-pixel blocks, with no overlap between the blocks.

[0150] In other embodiments, the image can also be divided into irregular blocks or blocks that overlap each other in some way.

[0151] In this embodiment, the color model is a color histogram. The step of generating a color model for different occluded regions in each image further includes: performing color histogram statistics on the unoccluded and occluded regions in each image.

[0152] Color histograms, as a relatively common method, mainly describe the quantitative characteristics of colors in an image. They can reflect the statistical distribution and basic hue of the colors in the image. Their function is to generate color statistics for estimating occlusion probability maps, representing the frequency of various colors appearing in a certain region of the image.

[0153] In other embodiments, the color model may also use other alternatives, such as the Gaussian mixture model (GMM).

[0154] Furthermore, when performing color histogram statistics, the color depth is downsampled. Optionally, the color depth can be downsampled to 4-bit color or lower.

[0155] Step S104: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frames of the RGB image, and obtain the occlusion probability map.

[0156] Step S104 further includes the following steps:

[0157] Morphological dilation is performed on the undetermined region of the first occlusion relationship map to obtain the second occlusion relationship map;

[0158] Select the latest frame of the RGB image, and based on the location of the undetermined region in the second occlusion relationship map, estimate the occlusion probability of each pixel in the corresponding region of the latest frame of the RGB image to obtain the occlusion probability map.

[0159] In this embodiment, the second occlusion relationship diagram is a second trigonometric diagram. The undetermined region in the first trigonometric diagram is morphologically dilated to obtain the second trigonometric diagram.

[0160] Morphological dilation refers to the expansion of the boundary points of a binary object by merging all background points in contact with the object into the object, thus expanding the boundary outward.

[0161] Morphological expansion is generally classified into horizontal expansion, vertical expansion, and omnidirectional expansion.

[0162] In this embodiment, a 3*3 rectangular kernel is used for morphological dilation.

[0163] The most recent RGB image is selected and denoted as the second RGB image. Obviously, in this embodiment, the most recent frame refers to the frame with the most recent exposure time.

[0164] Based on the locations marked as undetermined areas in the second triangulation, the probability of each pixel in the corresponding area of ​​the second RGB image being occluded is estimated to obtain an occlusion probability map. This involves the following steps:

[0165] In step S103, select several image blocks that are closest to the current pixel from the historical frames of the RGB image.

[0166] Based on the color models of these several image blocks, estimate the color models of the regions adjacent to the current pixel.

[0167] Based on the estimated color model, calculate the distribution of the current pixel's color in different occlusion regions;

[0168] Estimate the probability that the current pixel is occluded based on the distribution;

[0169] By applying regularization constraints to the probability values, an occlusion probability map is obtained.

[0170] Figure 3 A schematic diagram of image segmentation according to a first embodiment of the present invention is disclosed, such as... Figure 3 As shown, in this embodiment, the image is divided into non-overlapping rectangular blocks. The four images closest to the current pixel are selected. Based on the color histograms of these four images, the color histogram near the current pixel is estimated. Based on the estimated color histograms, the distribution of the color of the current pixel in different occlusion regions is calculated.

[0171] The method for determining the four images closest to the current pixel is to select the four images whose geometric center has the smallest Euclidean distance from the pixel position.

[0172] For all rectangular grid and non-overlapping block partitioning methods, the four most recent images are clearly the preferred embodiment, such as... Figure 3 As shown, the probability value of the dot pixel is calculated using a color histogram of four gray blocks.

[0173] In other embodiments, more blocks are also permitted, such as 16 blocks.

[0174] Because other block partitioning methods may exist, the definitions of "nearest" and "closest" can be modified, such as the closest blocks in the vertical or horizontal direction, or the closest blocks after weighted distance. Therefore, the scope of "nearest" and "closest" is not limited to the method of "minimum Euclidean distance".

[0175] In this embodiment, calculating the distribution of the current pixel in different occlusion regions and estimating the probability that the current pixel is occluded based on the distribution further includes:

[0176] Using a color histogram, calculate the frequency of occurrence of the color of the current pixel in the 0 and 1 regions of the four nearby rectangular blocks;

[0177] These frequencies are then weighted according to the distance from the geometric center of the block to the "current pixel".

[0178] Sum the frequency of occurrence in region 0 and the frequency of occurrence in region 1 to obtain the sum value;

[0179] The frequency of occurrence of region 1 is compared with the sum of the values ​​obtained, and the ratio is used as the probability that the current pixel is occluded.

[0180] Regularization is to impose constraints on minimizing the empirical error function; such constraints can be interpreted as prior knowledge.

[0181] Step S105: Obtain the luminance component of the latest frame of the RGB image, and use the luminance component to perform guided filtering on the occlusion probability map to obtain the first weight map.

[0182] The luminance component further includes, but is not limited to, the Y channel in YUV space, the L channel in Lab space, the I channel in HSI space, and the L channel in HSL space.

[0183] YUV is a color encoding method, where Y represents luminance. The Lab color space is a color-opposites space with a dimension L representing luminance. HSI refers to a model of a digital image, where luminance I corresponds to the image brightness and grayscale, representing the lightness or darkness of a color. HSL is a representation of points in the RGB color model in cylindrical coordinates, where L represents luminance.

[0184] In this embodiment, the latest frame's RGB image is denoted as the second RGB image. The second RGB image is converted to a color space containing luminance components, and the luminance channel is taken as the guide image. The occlusion probability image is then subjected to guided filtering to obtain the first weight image.

[0185] Guided filtering is used to transfer information from the RGB image to the tri-image, generating fusion weights.

[0186] Guided filtering, also known as guided image filtering, is an image filtering technique that uses a guide image G to filter the target image P (input image), so that the final output image is generally similar to the target image P, but the texture is similar to the guide image G.

[0187] Guided filtering, as a filter with edge-preserving capabilities, has been widely used in image dehazing and matting techniques in recent years. Compared to joint bilateral filters, guided filtering not only achieves comparable filtering effects in edge preservation and detail smoothing, but its advantages are more evident in computational speed and filtering efficiency. This is because its core process can be quickly implemented through summation operations within a window when the integral image is known, resulting in high algorithm efficiency and low complexity. Its edge-preserving capability perfectly meets the requirements of depth map optimization, while its efficient real-time computation makes it more competitive in practice.

[0188] Step S106: Use the first weight map as the final fused weight map;

[0189] The parameters of the final fused weight map are used as the image virtual-real fusion parameters.

[0190] Step S107: Based on the final fusion weight map, perform weighted fusion on the real-world RGB image and the virtual image to generate a display image.

[0191] Furthermore, guided filtering can be used again in the display pipeline to upsample the final fused weight map to the display resolution, blending the RGB image and the virtual image into a display image.

[0192] The first embodiment of the present invention proposes a mixed reality virtual-real fusion method based on color model and guided filtering. It obtains the initial triangulation and color modeling information from the historical RGB image to ensure depth matching. Then, it reconstructs a new triangulation in the latest RGB image using the color modeling information to ensure that the final effect is applied to the latest image and reduces latency.

[0193] Figure 4 A flowchart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a second embodiment of the present invention is disclosed, as follows: Figure 4 In the second embodiment shown, the mixed reality fusion method based on color model and guided filtering proposed in this invention specifically includes the following steps:

[0194] Step S201: Obtain the RGB image and depth image of the real scene, and obtain the virtual image and virtual depth;

[0195] Step S202: Select the latest frame of the depth image, the virtual depth of the historical frames of the virtual image, and the historical frames of the RGB image. Compare the virtual depth of the depth image and the virtual depth of the virtual image. Mark the occlusion relationship in the historical frames of the RGB image to obtain the first occlusion relationship map.

[0196] Step S203: Select historical frames of the RGB image and divide them into blocks. Generate color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram.

[0197] Step S204: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frames of the RGB image, and obtain the occlusion probability map.

[0198] Step S205: Obtain the luminance component of the latest frame of the RGB image, and use the luminance component to perform guided filtering on the occlusion probability map to obtain the first weight map;

[0199] Step S206: Perform morphological transformation on the first occlusion relationship graph to obtain the second weight graph;

[0200] Step S207: The first weight map and the second weight map are weighted and fused to obtain the third weight map, which is used as the final fused weight map;

[0201] Step S208: Based on the final fusion weight map, perform weighted fusion of the real-world RGB image and the virtual image to generate a display image.

[0202] Figure 5 A block diagram of a mixed reality virtual-real fusion algorithm based on a color model and guided filtering according to an embodiment of the present invention is disclosed, such as... Figure 4 and Figure 5 As shown, in the second embodiment and the first embodiment, steps S201-S205 are the same as steps S101-S105, and steps S208 are the same as steps S107. The difference lies in steps S206 and S207, that is... Figure 4 The dashed part in the text.

[0203] Steps S206 and S207 will be described in detail below.

[0204] Step S206 further includes:

[0205] Morphological dilation is performed on the undetermined region of the first occlusion relationship map to obtain the second occlusion relationship map;

[0206] Edge processing is performed on the second occlusion relationship graph to obtain the second weight graph.

[0207] Furthermore, the step of performing edge processing on the second occlusion relationship graph to obtain the second weight graph further includes:

[0208] The second occlusion relationship map is filtered and ternary processed to obtain the third occlusion relationship map.

[0209] The third occlusion relationship graph is filtered to obtain the second weight graph.

[0210] In this embodiment, the first occlusion relationship diagram is a first tripartite diagram, the second occlusion relationship diagram is a second tripartite diagram, and the third occlusion relationship diagram is a third tripartite diagram.

[0211] In this embodiment, filtering and ternary processing of the second triangulation image further includes:

[0212] The second trisection is subjected to mean filtering, and then triarization is performed using the third coefficient as the threshold.

[0213] The third coefficient is 0.5.

[0214] In this embodiment, the ternaryization process is as follows:

[0215] For filtered values ​​greater than the third coefficient, make them the second coefficient; for filtered values ​​less than the third coefficient, make them the first coefficient; and for filtered values ​​equal to the third coefficient, make them the third coefficient.

[0216] The first coefficient is 0, the second coefficient is 1, and the third coefficient is 0.5.

[0217] In this embodiment, filtering the third tripartite graph further includes:

[0218] Apply mean filtering to the third triplot.

[0219] It should be noted that in this embodiment, mean filtering is applied to both the second and third triangulation images. In other embodiments, different filters can be used for the second and third triangulation images. Other low-pass image filters can be used, such as Gaussian filtering or bilateral filtering with precisely designed thresholds.

[0220] Step S207: The first weight map and the second weight map are weighted and merged to obtain the third weight map, which is used as the final fused weight map.

[0221] The first and second weight maps are weighted and merged according to certain calculation rules to obtain the third weight map, which is then output as the final virtual-real fusion weight.

[0222] The calculation rule for weighted fusion can be based on confidence scores. Confidence scores can be generated from the values ​​of the first weighted graph, or they can be externally generated confidence scores, such as depth confidence scores.

[0223] Step S208: Based on the final fusion weight map, perform weighted fusion of the real-world RGB image and the virtual image to generate a display image.

[0224] Furthermore, a guided filter can be used again in the display pipeline to upsample the third weighted image to the display resolution, blending the RGB image and the virtual image into the display image.

[0225] In the first embodiment, a first weight map is used as the final fusion weight map. The first weight map is related to the depth map, but the depth map has lower accuracy and higher latency compared to the RGB map, and can only obtain blurry edges and details.

[0226] The second weighted image is related to the RGB image content and can obtain accurate edges and details.

[0227] Compared to the first embodiment, the second embodiment adopts a multi-weight joint calculation method, which weights and merges the second weight map with the first weight map to ensure that the edge transition is not abrupt.

[0228] For example, in some cases, it is necessary to display a user's hand in a virtual scene, where the scene seen by the user includes their own hand (from an RGB image), while the rest is a virtual scene (from a virtual image). Suppose that due to lighting, coloring, or other factors, one finger in the RGB image of the hand is a significantly different color from the other fingers and the palm.

[0229] In the first embodiment, when only the first weight map is used, due to the accuracy problem of the depth map, the gaps between the user's fingers may not be accurately identified, and the user may see objects from the RGB image between the fingers.

[0230] The second embodiment combines two weights, producing an output effect where "the user can clearly see their normal-colored fingers and palm, the gaps between the fingers are displayed perfectly, and they can also see their different-colored finger, although the edge of that finger is somewhat blurry." This effect is more in line with general human perception than using either the first or second weight map alone. Therefore, the fusion effect of the second embodiment is superior to that of the first embodiment.

[0231] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0232] To achieve the above-mentioned mixed reality virtual-real fusion method, the present invention also provides a mixed reality virtual-real fusion device, which may be composed of a chip architecture or other software and hardware forms.

[0233] Figure 6 An algorithm block diagram of a mixed reality virtual-real fusion device based on a color model and guided filtering according to an embodiment of the present invention is disclosed. Figure 7 A hardware schematic diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention is disclosed, such as... Figure 6 and Figure 7 As shown, the mixed reality fusion device 600 proposed in this invention includes an image information processor 610, a depth information processor 620, a central processing unit 630, and a display processing unit 640.

[0234] The image information processor 610 is connected to the central processing unit 630 and the display processing unit 640, and is used to acquire RGB images of real scenes and send them to the central processing unit 630 and the display processing unit 640.

[0235] The depth information processor 620, connected to the central processing unit 630, is used to acquire depth images of the real scene and send them to the central processing unit 630.

[0236] The central processing unit 630, connected to the display processing unit 640, is used to acquire virtual depth, and execute a mixed reality virtual-real fusion method based on the acquired RGB image, depth image and virtual depth of the real scene, and send the final fusion weight map to the display processing unit 640.

[0237] The display processing unit 640 acquires the final fusion weight map sent by the central processing unit 630, performs weighted fusion of the real-world RGB image and the virtual image, and generates a display image.

[0238] In this embodiment, the image information processor 610 is an AISC ISP chip, which is connected to the scene camera 650 to acquire the data collected by the scene camera 650 and generate an RGB image.

[0239] In this embodiment, the depth information processor 620 is an NPU or other computing unit capable of running depth calculation algorithms, connected to the depth sensor 660, acquiring the data collected by the depth sensor 660 and generating a depth map.

[0240] In this embodiment, the central processing unit 630 is a CPU or other computing unit capable of running mixed reality virtual-real fusion methods.

[0241] like Figure 6 and Figure 7 As shown, the display processing unit 640 generates a display image by fusing image curling and image aliasing. Image curling causes the input image to be deformed, so that pixels corresponding to the same object are located in the same position. Image aliasing combines various input information into the image that the user finally sees through the display device 680.

[0242] In this embodiment, the display processing unit 640 is an AISC display processing unit;

[0243] The display processing unit 640 is connected to the graphics processing unit (GPU) 670 to obtain the rendered image (virtual image) data of the GPU;

[0244] The display processing unit 640 is connected to the image information processor 610 and acquires the RGB image from the image information processor.

[0245] The display processing unit 640 is connected to the central processing unit 630 and obtains the final fusion weights from the central processing unit 630.

[0246] The display processing unit 640 is connected to the display device 680 and outputs the mixed and generated display image to the display device 680.

[0247] In this embodiment, the display device 680 may be a head-mounted display or glasses, etc.

[0248] When the implementation process document of the mixed reality virtual-real fusion method is a computer program, it can also be stored as an article of art in a computer-readable storage medium. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0249] This invention provides a mixed reality fusion method, apparatus, and medium. It uses color modeling and edge-preserving filtering to calculate image fusion parameters, taking into account both historical and current RGB images to ensure correct matching between depth and RGB images, while also utilizing the latest image information to effectively reduce display latency.

[0250] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0251] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0252] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0253] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0254] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0255] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0256] The above embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the protection scope of the present invention is not limited to the above embodiments, but should be the maximum scope that conforms to the innovative features mentioned in the claims.

Claims

1. A method for blending virtual and real reality, characterized in that, Includes the following steps: Acquire real-world RGB and depth images, as well as virtual images and virtual depth; Compare depth images and virtual depth to generate occlusion maps, and generate color models based on historical frames of RGB images and occlusion maps; The latest frame of the RGB image is used to perform edge-preserving filtering on the color model to obtain the final fused weight map; Based on the final fusion weight map, the real-world RGB image and the virtual image are weighted and fused to generate the display image. The step of comparing the depth image and the virtual depth to generate an occlusion relationship map, and generating a color model based on historical frames of the RGB image and the occlusion relationship map, further includes: Select the latest frame of the depth image, the virtual depth of the historical frames of the virtual image, and the historical frames of the RGB image; The virtual depth of the depth image and the virtual image are compared, and the occlusion relationship is marked in the historical frames of the RGB image to obtain the first occlusion relationship map; Historical frames of an RGB image are selected and divided into blocks. Based on the first occlusion relationship graph, a color model is generated for different occlusion regions of each block. The color model includes a color histogram model to characterize the frequency of color occurrence in different occlusion regions of each block, or the color model includes a Gaussian mixture model to characterize the probability of color occurrence in different occlusion regions of each block.

2. The mixed reality fusion method according to claim 1, characterized in that, The step of using the latest frame of the RGB image to perform edge-preserving filtering on the color model to obtain the final fused weight map further includes: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frames of the RGB image, and obtain the occlusion probability map; Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain the first weight map; The first weight map is used as the final fusion weight map.

3. The mixed reality fusion method according to claim 2, characterized in that, The step of obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes: Obtain the luminance component of the latest frame of the RGB image, and use the luminance component to perform guided filtering on the occlusion probability map to obtain the first weight map.

4. The mixed reality fusion method according to claim 2, characterized in that, The step of obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes: The latest frame of the RGB image is obtained, and joint bilateral filtering is performed on the occlusion probability map to obtain the first weight map.

5. The mixed reality fusion method according to claim 1, characterized in that, It also includes the following steps: A morphological transformation is performed on the first occlusion relationship graph to obtain the second weight graph; The first and second weight maps are weighted and merged to obtain the third weight map, which is used as the final merged weight map.

6. The mixed reality fusion method according to claim 1, characterized in that, The acquisition of real-world RGB images and depth images further includes: continuously acquiring real-world RGB images, depth images, and exposure timestamps at a preset frame rate; The acquisition of virtual image and virtual depth further includes: continuously acquiring virtual image and virtual depth of virtual image at a preset frame rate.

7. The mixed reality fusion method according to claim 1, characterized in that, After acquiring the real-world RGB image and depth image, and the virtual image and virtual depth, the process further includes: downsampling all acquired images to a low-resolution image. Before weighted fusion of the real-world RGB image and the virtual image based on the final fusion weight map to generate the display image, the method further includes: upsampling the final fusion weight map to the display resolution.

8. The mixed reality fusion method according to claim 7, characterized in that, For the acquired RGB image and virtual image, bilinear interpolation is used for downsampling.

9. The mixed reality fusion method according to claim 7, characterized in that, For the acquired depth image and virtual depth, downsampling is performed using nearest neighbor interpolation.

10. The mixed reality fusion method according to claim 1, characterized in that, The step of marking occlusion relationships in historical frames of RGB images further includes: The RGB image is divided into three parts: a virtual image display area, a real image display area, and a region to be determined. The virtual image region will be marked as the first coefficient; The region that will display the true image will be marked as the second coefficient; The region to be determined is marked as the third coefficient.

11. The mixed reality virtual-real fusion method according to claim 10, characterized in that, The step of marking occlusion relationships in historical frames of RGB images further includes: Regions with missing depth are marked as undetermined regions; Regions lacking virtual depth will not be marked.

12. The mixed reality fusion method according to claim 5, characterized in that, The step of performing a morphological transformation on the first occlusion relationship graph to obtain the second weight graph further includes: Morphological dilation is performed on the undetermined region of the first occlusion relationship map to obtain the second occlusion relationship map; Edge processing is performed on the second occlusion relationship graph to obtain the second weight graph.

13. The mixed reality virtual-real fusion method according to claim 12, characterized in that, The step of performing edge processing on the second occlusion relationship map to obtain the second weight map further includes: The second occlusion relationship map is filtered and ternary processed to obtain the third occlusion relationship map. The third occlusion relationship graph is filtered to obtain the second weight graph.

14. The mixed reality virtual-real fusion method according to claim 1, characterized in that, The color model includes a color histogram; The step of generating color models for different occluded regions in each image further includes: performing color histogram statistics on the unoccluded and occluded regions in each image.

15. The mixed reality virtual-real fusion method according to claim 14, characterized in that, When performing color histogram statistics, the color depth is downsampled.

16. The mixed reality virtual-real fusion method according to claim 2, characterized in that, The step of selecting the latest frame of the RGB image and estimating the probability of pixel occlusion based on the color model of historical frames of the RGB image to obtain an occlusion probability map further includes: Morphological dilation is performed on the undetermined region of the first occlusion relationship map to obtain the second occlusion relationship map; Select the latest frame of the RGB image, and based on the location of the undetermined region in the second occlusion relationship map, estimate the occlusion probability of each pixel in the corresponding region of the latest frame of the RGB image to obtain the occlusion probability map.

17. The mixed reality fusion method according to claim 16, characterized in that, The step of estimating the occlusion probability of each pixel in the corresponding region of the latest frame of the RGB image based on the location of the undetermined region in the second occlusion relationship map to obtain an occlusion probability map further includes: Select several image blocks from the historical frames of the RGB image that are closest to the current pixel; Based on the color models of several image blocks, estimate the color model of the region adjacent to the current pixel; Based on the estimated color model, calculate the distribution of the current pixel's color in different occlusion regions; Estimate the probability that the current pixel is occluded based on the distribution; By applying regularization constraints to the probability values, an occlusion probability map is obtained.

18. The mixed reality fusion method according to claim 3, characterized in that, The luminance components further include the Y channel in YUV space, the L channel in Lab space, the I channel in HSI space, and the L channel in HSL space.

19. A mixed reality device for fusing virtual and real elements, characterized in that, Includes an image information processor, a depth information processor, a central processing unit, and a display processing unit: The image information processor is connected to the central processing unit and the display processing unit, and is used to acquire RGB images of real scenes and send them to the central processing unit and the display processing unit; The depth information processor is connected to the central processing unit and is used to acquire depth images of the real scene and send them to the central processing unit. The central processing unit is connected to the display processing unit and is used to obtain virtual depth, execute the mixed reality virtual-real fusion method as described in any one of claims 1-18, and send the final fusion weight map to the display processing unit; The display processing unit acquires the final fusion weight map sent by the central processing unit, performs weighted fusion of the real-world RGB image and the virtual image, and generates a display image.

20. The mixed reality fusion device according to claim 19, characterized in that, The image information processor is connected to the scene camera, acquires the data collected by the scene camera, and generates an RGB image; The depth information processor is connected to the depth sensor, acquires the data collected by the depth sensor, and generates a depth image.

21. A mixed reality (MRP) system that blends virtual and real elements, characterized in that, This includes mixed reality devices that blend virtual and real elements, scene cameras, depth sensors, graphics processors, and display devices. The mixed reality virtual-real fusion device is the mixed reality virtual-real fusion device as described in any one of claims 19-20; The scene camera is used to capture RGB images of the real scene and send them to the hybrid display virtual and real device; The depth sensor is used to acquire depth images of the real scene and send them to the hybrid display virtual-real device; The graphics processor is used to generate virtual images and virtual depth and send them to the hybrid display virtual-real device; The display device is used to receive and display the display image generated by the hybrid display virtual and real device.

22. A computer-readable medium having stored thereon computer instructions, wherein when the computer instructions are executed by a processor, the method of any one of claims 1-18 is performed.

Images