Mixed reality virtual-real fusion method, device and system
By generating occlusion relationship diagrams and color models, combined with edge-keeping filtering technology, the weighted fusion of real and virtual images is achieved, solving the poor display effect and delay of virtual and real fusion scenes in the existing technology, and improving the reality of mixed reality systems Feeling and real-time performance.
Patent Information
- Application Number
- CN202310629358.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-30
AI Technical Summary
The existing mixed reality systems have poor display effects in virtual and real fusion scenarios, with large delays, making it difficult to achieve realism and real-time frame rates.
By acquiring the real-world RGB images and depth images, comparing the depth images and virtual depth to generate an occlusion relationship diagram, generating a color model based on the historical frames and occlusion relationship diagram of the RGB image, and using the latest frames of the RGB image to edge filter the color model , obtain the final fusion weight map, perform weighted fusion of real world and virtual images, and generate display images.
It effectively reduces display delay, improves the realism and frame rate of virtual and real fusion, ensures the correct matching of depth images with RGB images, and uses the latest image information to improve the real-time performance of the system.
Smart Images

Figure CN119068155A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine vision and image fusion, and more specifically, to a mixed reality virtual-real fusion method, device, system, and medium. Background Art
[0002] Mixed reality technology (MR) presents virtual scene information in real scenes, building an interactive feedback information loop between the real world, the virtual world and users to enhance the realism of the user experience.
[0003] Mixed reality systems have high requirements for the display realism of virtual objects. The purpose is to seamlessly connect virtual objects with real objects and maintain the immersiveness of the scene and the realism of virtual objects. This requirement also poses new challenges to display technology.
[0004] In the existing mixed reality system architecture, virtual objects are presented through rendering technology in the virtual-reality fusion scene. In order to determine the front and back occlusion relationship between virtual objects and real objects when the virtual and real are combined, the following three display strategies are usually adopted:
[0005] 1) Virtual objects are always displayed before real objects;
[0006] 2) Try to obtain as accurate depth information as possible and directly compare the depth of real objects with virtual objects;
[0007] 3) Obtain certain information from the image of the real object and combine it with the inaccurate depth for cutout.
[0008] For the first display strategy, the display effect is not realistic enough and the user experience is poor;
[0009] For the second display strategy, the amount of calculation required to process the depth map is very large and may not be calculated in real time. According to the algorithm in the prior art, it takes an average of 48.4ms and the display delay is about 116ms, which means that it can only support a maximum of 20FPS of virtual and real combination, and the effect has a delay that is obviously perceptible to the human eye.
[0010] For the third display strategy, according to the algorithms in the prior art, the depth map and the RGB image are usually required to be from the same time. However, in wearable devices, due to power and heat limitations, the delay of obtaining the depth map is usually much higher than the delay of obtaining the RGB image, and the frame rate is often lower than the frame rate of the RGB image. Therefore, it is difficult for the algorithm to use the latest RGB image, which will also increase the overall delay of the display system.
[0011] Based on the above analysis, the existing technology has poor display effects on the virtual-reality fusion scenes of mixed reality systems, and there are delays that are difficult to be effectively solved. Summary of the invention
[0012] The purpose of the present invention is to provide a mixed reality virtual-real fusion method, device, system and medium to solve the problems of poor display effect and large delay in the prior art virtual reality virtual-real fusion.
[0013] In order to achieve the above object, the present invention provides a mixed reality virtual-real fusion method, comprising the following steps:
[0014] Obtain real-world RGB images and depth images, as well as virtual images and virtual depths;
[0015] Compare the depth image and the virtual depth to generate an occlusion relationship graph, and generate a color model based on the historical frame of the RGB image and the occlusion relationship graph;
[0016] The latest frame of the RGB image is used to perform edge-preserving filtering on the color model to obtain the final fusion weight map;
[0017] According to the final fusion weight map, the real-world RGB image and the virtual image are weightedly fused to generate a display image.
[0018] In some embodiments, the comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frames of the RGB image and the occlusion relationship graph, further comprises:
[0019] Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image;
[0020] Compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the history frame of the RGB image, and obtain a first occlusion relationship graph;
[0021] A historical frame of the RGB image is selected and divided into blocks, and a color model is generated for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
[0022] In some embodiments, the step of using the latest frame of the RGB image to perform edge-preserving filtering on the color model to obtain a final fusion weight map further includes:
[0023] Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain the occlusion probability map;
[0024] Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain a first weight map;
[0025] The first weight map is taken as the final fusion weight map.
[0026] In some embodiments, the obtaining of the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes:
[0027] The brightness component of the latest frame of the RGB image is obtained, and the brightness component is used to perform guided filtering on the occlusion probability map to obtain a first weight map.
[0028] In some embodiments, the obtaining of the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes:
[0029] The latest frame of the RGB image is obtained, and the occlusion probability map is subjected to joint bilateral filtering to obtain the first weight map.
[0030] In some embodiments, the mixed reality virtual-real fusion method further includes the following steps:
[0031] Performing morphological transformation on the first occlusion relationship graph to obtain a second weight graph;
[0032] The first weight map and the second weight map are weightedly fused to obtain a third weight map as the final fused weight map.
[0033] In some embodiments, the acquiring of the real-world RGB image and depth image further comprises: continuously acquiring the real-world RGB image, depth image and exposure timestamp at a preset frame rate;
[0034] The acquiring of the virtual image and the virtual depth further includes: continuously acquiring the virtual image and the virtual depth of the virtual image at a preset frame rate.
[0035] In some embodiments, after acquiring the real-world RGB image and depth image and acquiring the virtual image and virtual depth, the method further includes: downsampling all acquired images to low-resolution images;
[0036] The method further includes: performing weighted fusion of the real-world RGB image and the virtual image according to the final fusion weight map, and before mixing and generating the display image, upsampling the final fusion weight map to the display resolution.
[0037] In some embodiments, bilinear interpolation is used to downsample the acquired RGB image and the virtual image.
[0038] In some embodiments, the acquired depth image and virtual depth are downsampled using nearest neighbor interpolation.
[0039] In some embodiments, marking the occlusion relationship in the RGB image further comprises:
[0040] The RGB image is divided into three parts: the virtual image display area, the real image display area and the pending area;
[0041] marking the displayed virtual image area as a first coefficient;
[0042] The area showing the true image is marked as the second coefficient;
[0043] The undetermined area is marked as the third coefficient.
[0044] In some embodiments, marking the occlusion relationship in the RGB image further comprises:
[0045] Mark the area where the depth is missing as a pending area;
[0046] Areas where virtual depth is missing are not marked.
[0047] In some embodiments, performing a morphological transformation on the first occlusion relationship graph to obtain a second weight graph further includes:
[0048] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0049] Edge processing is performed on the second occlusion relationship graph to obtain a second weight graph.
[0050] In some embodiments, performing edge processing on the second occlusion relationship graph to obtain a second weight graph further includes:
[0051] Filtering and ternary processing the second occlusion relationship graph to obtain a third occlusion relationship graph;
[0052] The third occlusion relationship graph is filtered to obtain a second weight graph.
[0053] In some embodiments, the color model includes a color histogram;
[0054] Generating color models for different blocked areas in each image block further includes: performing color histogram statistics on the unblocked areas and blocked areas in each image block.
[0055] In some embodiments, when performing color histogram statistics, the color depth is downsampled.
[0056] In some embodiments, selecting the latest frame of the RGB image, estimating the probability of pixel points being occluded according to the color model of the historical frames of the RGB image, and obtaining the occlusion probability map further includes:
[0057] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0058] The latest frame of the RGB image is selected, and according to the position of the undetermined area in the second occlusion relationship diagram, the occlusion probability of each pixel point in the corresponding area of the latest frame of the RGB image is estimated to obtain an occlusion probability map.
[0059] In some embodiments, the estimating the occlusion probability of each pixel point in the corresponding position area of the latest frame of the RGB image according to the position of the undetermined area in the second occlusion relationship map to obtain the occlusion probability map further includes:
[0060] Select several blocks of images closest to the current pixel in the historical frame of the RGB image;
[0061] According to the color models of several blocks of images, the color model of the area close to the current pixel is estimated;
[0062] According to the estimated color model, calculate the distribution of the color of the current pixel in different occlusion areas;
[0063] Estimate the probability that the current pixel is blocked based on the distribution;
[0064] Regularization constraints are applied to the probability values to obtain the occlusion probability map.
[0065] In some embodiments, the brightness component further includes a Y channel in the YUV space, an L channel in the Lab space, an I channel in the HSI space, and an L channel in the HSL space.
[0066] In order to achieve the above-mentioned object, the present invention provides a mixed reality virtual-real fusion device, including an image information processor, a depth information processor, a central processing unit, and a display processing unit:
[0067] The image information processor is connected to the central processing unit and the display processing unit, and is used to obtain the RGB image of the real world and send it to the central processing unit and the display processing unit;
[0068] The depth information processor is connected to the central processing unit and is used to obtain a depth image of the real scene and send it to the central processing unit;
[0069] The central processing unit is connected to the display processing unit, and is used to obtain virtual depth, execute any of the mixed reality virtual-real fusion methods described above, and send the final fusion weight map to the display processing unit;
[0070] The display processing unit obtains the final fusion weight map sent by the central processing unit, performs weighted fusion on the real-world RGB image and the virtual image, and mixes and generates a display image.
[0071] In one embodiment, the image information processor is connected to the scene camera, obtains the acquisition data of the scene camera and generates an RGB image;
[0072] The depth information processor is connected to the depth sensor, obtains the collected data of the depth sensor and generates a depth image.
[0073] In order to achieve the above-mentioned object, the present invention provides a mixed reality virtual-reality fusion system, including a mixed reality virtual-reality fusion device, a scene camera, a depth sensor, a graphics processor and a display device:
[0074] The mixed reality virtual-reality fusion device adopts any of the mixed reality virtual-reality fusion devices described above;
[0075] The scene camera is used to collect RGB images of the real scene and send them to the mixed display virtual and real device;
[0076] The depth sensor is used to collect the depth image of the real scene and send it to the mixed display virtual and real device;
[0077] The image processor is used to generate a virtual image and a virtual depth and send them to a mixed display virtual-real device;
[0078] The display device is used to receive and display the display image generated by the mixed display virtual and real device.
[0079] In order to achieve the above object, the present invention provides a computer-readable medium on which computer instructions are stored, wherein when the computer instructions are executed by a processor, any of the above methods is executed.
[0080] The present invention provides a mixed reality virtual-real fusion method, device, system, and medium, which use color modeling and edge-preserving filtering methods to calculate image virtual-real fusion parameters, taking into account historical and current RGB images to ensure the correct matching of depth images and RGB images, and can utilize the latest image information to effectively reduce display delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] The above and other features, properties and advantages of the present invention will become more apparent through the following description in conjunction with the accompanying drawings and embodiments, in which the same reference numerals always represent the same features, wherein:
[0082] Figure 1 A flow chart of a mixed reality virtual-real fusion method according to an embodiment of the present invention is disclosed;
[0083] Figure 2 A flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a first embodiment of the present invention is disclosed;
[0084] Figure 3 A schematic diagram of image segmentation according to a first embodiment of the present invention is disclosed;
[0085] Figure 4 A flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a second embodiment of the present invention is disclosed;
[0086] Figure 5 A workflow diagram of a mixed reality virtual-real fusion algorithm based on a color model and guided filtering according to a second embodiment of the present invention is disclosed;
[0087] Figure 6 An algorithm module diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention is disclosed;
[0088] Figure 7 A hardware schematic diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention is disclosed.
[0089] The meanings of the reference numerals in the figures are as follows:
[0090] 610 image information processor;
[0091] 620 depth information processor;
[0092] 630 CPU;
[0093] 640 display processing unit;
[0094] 650 scene cameras;
[0095] 660 depth sensor;
[0096] 670 graphics processor. DETAILED DESCRIPTION
[0097] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention and are not used to limit the invention.
[0098] Figure 1 A flow chart of a mixed reality virtual-real fusion method according to an embodiment of the present invention is disclosed. Figure 1 As shown, the present invention provides a mixed reality virtual-real fusion method, comprising the following steps:
[0099] Step S1, obtaining a real-world RGB image, a depth image, and obtaining a virtual image and a virtual depth;
[0100] Step S2, comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frame of the RGB image and the occlusion relationship graph;
[0101] Step S3, using the latest frame of the RGB image to perform edge-preserving filtering on the color model to obtain a final fusion weight map;
[0102] Step S4: According to the final fusion weight map, the real-world RGB image and the virtual image are weightedly fused to generate a display image.
[0103] Furthermore, the step S2, comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frames of the RGB image and the occlusion relationship graph, further includes:
[0104] Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image;
[0105] Compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the history frame of the RGB image, and obtain a first occlusion relationship graph;
[0106] A historical frame of the RGB image is selected and divided into blocks, and a color model is generated for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
[0107] Furthermore, the step S3, using the latest frame of the RGB image to perform edge-preserving filtering on the color model to obtain the final fusion weight map, further includes:
[0108] Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain the occlusion probability map;
[0109] Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain a first weight map;
[0110] The first weight map is taken as the final fusion weight map.
[0111] The edge preserving filtering method uses an edge preserving filter to achieve filtering. An edge preserving filter is a special type of filter that can effectively preserve the edge information in an image during the filtering process.
[0112] Among them, the joint bilateral filter, the guided image filter, and the weighted least square filter are several well-known edge-preserving filters.
[0113] The following uses an edge-preserving filter as a guide filter as an example to illustrate the mixed reality virtual-real fusion method proposed by the present invention. It should be noted that the mixed reality virtual-real fusion method proposed by the present invention can also be implemented by using other edge-preserving filters.
[0114] Figure 2 A flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a first embodiment of the present invention is disclosed. Figure 2 In the first embodiment shown, the present invention proposes a mixed reality virtual-real fusion method based on color model and guided filtering, which specifically includes the following steps:
[0115] Step S101, obtaining an RGB image and a depth image of a real scene, and obtaining a virtual image and a virtual depth;
[0116] Step S102: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image, compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the historical frame of the RGB image, and obtain a first occlusion relationship diagram;
[0117] Step S103, selecting historical frames of the RGB image, dividing them into blocks, and generating color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram;
[0118] Step S104, selecting the latest frame of the RGB image, estimating the probability of pixel points being occluded according to the color model of the historical frame of the RGB image, and obtaining an occlusion probability map;
[0119] Step S105, obtaining the brightness component of the latest frame of the RGB image, and using the brightness component to perform guided filtering on the occlusion probability map to obtain a first weight map;
[0120] Step S106: taking the first weight map as the final fusion weight map;
[0121] Step S107: weighted fusion is performed on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0122] These steps will be described in detail below. It should be understood that within the scope of the present invention, the above-mentioned technical features of the present invention and the technical features specifically described below (such as embodiments) can be combined and related to each other to form a preferred technical solution.
[0123] Step S101: Acquire a real-world RGB image and a depth image, and acquire a virtual image and a virtual depth.
[0124] Depth information refers to the distance between an object and the camera in the real scene. The shooting plane of the camera can be used as the X and Y axes of the three-dimensional coordinate system. Then, the depth represents the Z axis distance. Therefore, the depth information can be used to determine the occlusion relationship.
[0125] In some embodiments, the RGB image, depth image and corresponding exposure timestamp of the real scene are continuously acquired at a preset frame rate. Preferably, the preset frame rate is a fixed frame rate. In other embodiments, the preset frame rate allows a certain range of delay but cannot exceed the delay threshold.
[0126] In some embodiments, the virtual image and the virtual depth (Z-buffer) of the virtual image are continuously acquired at a preset frame rate. Preferably, the preset frame rate is a fixed frame rate. In other embodiments, the preset frame rate allows a certain range of delay but cannot exceed the delay threshold.
[0127] In this embodiment, the virtual image and the depth of the virtual image are continuously obtained from the graphics processor. The virtual image refers to the rendered image provided by the graphics processor.
[0128] Optionally, after acquiring all images (RGB images, depth images, virtual images, virtual depths), all acquired images are downsampled to lower resolution images to meet the requirements of computing power and ensure subsequent operation efficiency.
[0129] Obviously, the "lower" resolution here is a relative concept relative to computing power. When the image is downsampled to a resolution that meets the computing power requirements, it can be considered a lower resolution.
[0130] Downsampling methods include, but are not limited to, bilinear interpolation and nearest neighbor interpolation methods.
[0131] Preferably, bilinear interpolation is used to downsample the acquired RGB image and virtual image;
[0132] Preferably, the acquired depth map and virtual depth are downsampled using nearest neighbor interpolation.
[0133] Step S102: select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image, compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the historical frame of the RGB image, and obtain a first occlusion relationship diagram.
[0134] In this embodiment, the historical frame of the virtual image further refers to a historical frame in the virtual image that has an exposure time closest to the latest frame of the depth image.
[0135] In this embodiment, the historical frame of the RGB image further refers to a historical frame in the RGB image that has the closest exposure time to the latest frame of the depth image, which is recorded as the first RGB image.
[0136] Obviously, the exposure time can be determined by the exposure timestamp obtained in step S101.
[0137] In this embodiment, the first occlusion relationship graph is a first tri-graph.
[0138] The first three-part image divides the image into three parts: the virtual image area, the real image area, and the pending area:
[0139] The area where the virtual image is displayed, that is, the area where the virtual image is not blocked, is marked as the first coefficient. In this embodiment, the first coefficient is 0;
[0140] The real image area is displayed, that is, the virtual image is blocked, and is marked as the second coefficient. In this embodiment, the second coefficient is 1;
[0141] The undetermined area, that is, the area where it is not yet certain whether to display a virtual image or a real image, is marked as the third coefficient. In this embodiment, the third coefficient is 0.5.
[0142] Furthermore, the depth-missing regions are marked as pending;
[0143] Furthermore, the area where the virtual depth is missing is not marked.
[0144] Step S103: select historical frames of the RGB image and divide them into blocks, and generate color models for different occlusion areas of each image block according to the first occlusion relationship diagram.
[0145] In this embodiment, selecting the historical frames of the RGB image and dividing them into blocks further includes:
[0146] According to the exposure timestamp, a frame of RGB image closest to the exposure time of the latest frame of the depth image is selected, that is, the first RGB image, and the first RGB image is divided into a number of non-overlapping blocks.
[0147] Furthermore, the first RGB image is divided into a number of non-overlapping blocks. The specific division method is not too limited. The following examples illustrate two division methods as preferred implementations, but it cannot be considered that the division methods proposed in the present invention are limited to the following two methods:
[0148] For example, the entire image may be divided into a fixed number of rectangular blocks, such as 8 blocks in the horizontal direction and 8 blocks in the vertical direction, for a total of 8*8=64 blocks, and the blocks do not overlap with each other.
[0149] For example, the entire image may be divided into rectangular blocks of a fixed size, such as blocks of 32 pixels×32 pixels, where the blocks do not overlap with each other.
[0150] In other embodiments, the image may be divided into blocks in a certain manner into irregular blocks or blocks that overlap with each other.
[0151] In this embodiment, the color model is a color histogram, and generating color models for different occluded areas in each image block further includes: performing color histogram statistics on the unoccluded areas and the occluded areas in each image block.
[0152] As a more general method, the color histogram mainly describes the quantitative characteristics of the color in the image, which can reflect the statistical distribution and basic tone of the image color. Its function is to generate color statistical information for estimating the occlusion probability map, which indicates the frequency of occurrence of various colors in a certain area of the image.
[0153] In other embodiments, the color model may also use other alternatives, such as a Gaussian mixture model (GMM).
[0154] Furthermore, when performing color histogram statistics, the color depth is downsampled. Optionally, the color depth can be downsampled to 4-bit color or lower.
[0155] Step S104, selecting the latest frame of the RGB image, estimating the probability of pixel points being occluded according to the color model of the historical frame of the RGB image, and obtaining an occlusion probability map;
[0156] The step S104 further includes the following steps:
[0157] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0158] The latest frame of the RGB image is selected, and according to the position of the undetermined area in the second occlusion relationship diagram, the occlusion probability of each pixel point in the corresponding area of the latest frame of the RGB image is estimated to obtain an occlusion probability map.
[0159] In this embodiment, the second occlusion relationship graph is a second tri-plot, and the undetermined region in the first tri-plot is morphologically expanded to obtain the second tri-plot.
[0160] Morphological dilation refers to the expansion of the boundary points of the binary object, merging all background points that are in contact with the object into the object, and expanding the boundary outward.
[0161] Morphological expansion is generally divided into horizontal expansion, vertical expansion and omnidirectional expansion.
[0162] In this embodiment, a 3*3 rectangular kernel is used for morphological dilation.
[0163] The RGB image of the latest frame is selected and recorded as the second RGB image. Obviously, in this embodiment, the latest frame refers to the frame with the latest exposure time.
[0164] According to the position of the area marked as undetermined in the second three-part image, the probability of each pixel point in the corresponding area of the second RGB image being occluded is estimated to obtain an occlusion probability map, which specifically includes the following steps:
[0165] Selecting several blocks of images closest to the current pixel in the historical frame of the RGB image in step S103;
[0166] According to the color models of the plurality of image blocks, the color model of the area close to the current pixel is estimated;
[0167] According to the estimated color model, calculate the distribution of the color of the current pixel in different occlusion areas;
[0168] Estimate the probability that the current pixel is blocked based on the distribution;
[0169] Regularization constraints are applied to the probability values to obtain the occlusion probability map.
[0170] Figure 3 A schematic diagram of image segmentation according to a first embodiment of the present invention is disclosed. Figure 3 As shown, in this embodiment, the image is divided into non-overlapping rectangular blocks, and the four image blocks closest to the current pixel are selected. The color histogram near the current pixel is estimated based on the color histogram statistics of these four image blocks, and the distribution of the color of the current pixel in different occlusion areas is calculated based on the estimated color histogram.
[0171] The four images closest to the current pixel are determined by the four images with the smallest Euclidean distance between the geometric center and the pixel position.
[0172] For all rectangular grids and non-overlapping block divisions, the last 4 blocks of images are obviously the preferred embodiment, such as Figure 3 As shown, the color histogram of the four gray blocks is used to calculate the probability value of the dot pixel.
[0173] In other embodiments, more blocks are allowed to be used, for example, 16 blocks are used.
[0174] Due to the possible existence of other blocking methods, the definition of "nearest" and "closest" can be modified, such as the nearest blocks in the vertical or horizontal direction, the nearest blocks after weighted distance, etc. Therefore, the representation scope of "nearest" and "closest" is not limited to the "minimum Euclidean distance" method.
[0175] In this embodiment, calculating the distribution of the current pixel in different occlusion areas and estimating the probability of the current pixel being occluded according to the distribution further includes:
[0176] Use the color histogram to calculate the frequency of occurrence of the color of the current pixel in the areas marked as 0 and 1 in the four nearby rectangular blocks;
[0177] And these frequencies are weighted according to the distance from the geometric center of the block to the "current pixel";
[0178] Sum the frequency of occurrence in the 0 region and the frequency of occurrence in the 1 region to obtain the sum value;
[0179] Compare the frequency of occurrence of area 1 with the obtained sum, and use the ratio as the probability that the current pixel is blocked.
[0180] Regularization is to impose constraints on minimizing the empirical error function, and such constraints can be interpreted as prior knowledge.
[0181] Step S105: Obtain the brightness component of the latest frame of the RGB image, and use the brightness component to perform guided filtering on the occlusion probability map to obtain a first weight map.
[0182] The brightness component further includes but is not limited to the Y channel in the YUV space, the L channel in the Lab space, the I channel in the HSI space, the L channel in the HSL space, etc.
[0183] YUV is a color encoding method, where Y represents brightness. Lab color space is a color-opponent space with dimension L representing brightness. HSI refers to a digital image model, where brightness I refers to the corresponding imaging brightness and image grayscale, and is the brightness of the color. HSL is a method of representing points in the RGB color model in a cylindrical coordinate system, where L represents brightness.
[0184] In this embodiment, the RGB image of the latest frame is recorded as the second RGB image. The second RGB image is converted into a color space containing a brightness component, and the brightness channel thereof is taken as a guide image. The occlusion probability image is guided filtered to obtain a first weight image.
[0185] Guided filtering is used to transfer the information in the RGB image to the triplicate map and generate fusion weights.
[0186] Guided filtering, also known as guided graph filtering, is an image filtering technology that filters the target image P (input image) through a guiding graph G, so that the final output image is roughly similar to the target image P, but the texture part is similar to the guiding graph G.
[0187] Guided filtering, as a filter with edge-preserving capability, has been widely used in image dehazing and cutout techniques in recent years. Compared with the joint bilateral filter, guided filtering can not only achieve comparable filtering effects in edge preservation and detail smoothing, but its advantages are more reflected in the calculation speed and filtering efficiency, because its core process can be quickly implemented through the summation operation within the window when the integral image is known, so the algorithm is highly efficient and has low complexity. Its edge-preserving capability just meets the needs of depth map optimization, and its efficient real-time operation also makes it more competitive in practice.
[0188] Step S106: taking the first weight map as the final fusion weight map;
[0189] The parameters of the final fusion weight map are used as the image virtual-real fusion parameters.
[0190] Step S107: weighted fusion is performed on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0191] Furthermore, the guided filtering can be used again in the display pipeline to upsample the final fused weight map to the display resolution, and the RGB image and the virtual image can be mixed into a display image.
[0192] The first embodiment of the present invention proposes a mixed reality virtual-real fusion method based on color model and guided filtering, which obtains the initial tripartite map and color modeling information from the historical RGB image to ensure depth matching, and then reconstructs the color modeling information into a new tripartite map in the latest RGB image to ensure that the final effect acts on the latest image and reduce latency.
[0193] Figure 4 A flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a second embodiment of the present invention is disclosed. Figure 4 In the second embodiment shown, the present invention proposes a mixed reality virtual-real fusion method based on color model and guided filtering, which specifically includes the following steps:
[0194] Step S201, obtaining an RGB image and a depth image of a real scene, and obtaining a virtual image and a virtual depth;
[0195] Step S202: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image, compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the historical frame of the RGB image, and obtain a first occlusion relationship diagram;
[0196] Step S203: select historical frames of the RGB image, divide them into blocks, and generate color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram;
[0197] Step S204, selecting the latest frame of the RGB image, estimating the probability of pixel points being occluded according to the color model of the historical frame of the RGB image, and obtaining an occlusion probability map;
[0198] Step S205, obtaining the brightness component of the latest frame of the RGB image, and using the brightness component to perform guided filtering on the occlusion probability map to obtain a first weight map;
[0199] Step S206, performing morphological transformation on the first occlusion relationship graph to obtain a second weight graph;
[0200] Step S207: weightedly fuse the first weight map and the second weight map to obtain a third weight map as the final fused weight map;
[0201] Step S208: weighted fusion is performed on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0202] Figure 5 A module diagram of a mixed reality virtual-real fusion algorithm based on a color model and guided filtering according to an embodiment of the present invention is disclosed. Figure 4 and Figure 5 As shown, in the second embodiment and the first embodiment, steps S201 to S205 are the same as steps S101 to S105, and step S208 is the same as step S107. The difference lies in step S206 and step S207, that is, Figure 4 The dotted part in .
[0203] Step S206 and step S207 will be described in detail below.
[0204] The step S206 further comprises:
[0205] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0206] Edge processing is performed on the second occlusion relationship graph to obtain a second weight graph.
[0207] Furthermore, the step of performing edge processing on the second occlusion relationship graph to obtain a second weight graph further includes:
[0208] Filtering and ternary processing the second occlusion relationship graph to obtain a third occlusion relationship graph;
[0209] The third occlusion relationship graph is filtered to obtain a second weight graph.
[0210] In this embodiment, the first occlusion relationship graph is a first three-division graph, the second occlusion relationship graph is a second three-division graph, and the third occlusion relationship graph is a third three-division graph.
[0211] In this embodiment, filtering and ternary processing are performed on the second three-division image, further comprising:
[0212] Perform mean filtering on the second three-part image, and then perform ternary processing using the third coefficient as a threshold;
[0213] Among them, the third coefficient is 0.5.
[0214] In this embodiment, the ternary processing method is:
[0215] For the value greater than the third coefficient after filtering, make it the second coefficient, for the value less than the third coefficient after filtering, make it the first coefficient, and for the value equal to the third coefficient after filtering, make it the third coefficient;
[0216] Among them, the first coefficient is 0, the second coefficient is 1, and the third coefficient is 0.5.
[0217] In this embodiment, filtering the third three-part image further includes:
[0218] Perform mean filtering on the third trisection image.
[0219] It should be noted that, in this embodiment, the second and third trisection images are both subjected to mean filtering. In other embodiments, different filters may be used in the second and third trisection images. Other low-pass image filters may be used, such as Gaussian filtering or bilateral filtering with accurately designed thresholds.
[0220] Step S207: Perform weighted fusion on the first weight map and the second weight map to obtain a third weight map as the final fused weight map.
[0221] The first weight map and the second weight map are weightedly fused according to a certain calculation rule to obtain a third weight map, and the third weight map is output as the final virtual-real fusion weight.
[0222] The calculation rule of weighted fusion may be to fuse by confidence. The confidence may be generated according to the value of the first weight map, or may be an externally generated confidence, such as depth confidence.
[0223] Step S208: weighted fusion is performed on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0224] Furthermore, the guided filter may be used again in the display pipeline to upsample the third weight map to the display resolution, and the RGB image and the virtual image may be mixed into a display image.
[0225] In the first embodiment, the first weight map is used as the final fusion weight map. The first weight map is related to the depth map. However, compared with the RGB map, the depth map has lower accuracy and higher latency, and can only obtain blurred edges and details.
[0226] The second weight map is related to the RGB image content and can obtain accurate edges and details.
[0227] Compared with the first embodiment, the second embodiment adopts a multi-weight joint calculation method to perform weighted fusion on the second weight map and the first weight map to ensure that the edge transition is not abrupt.
[0228] For example, in some cases, it is necessary to display the user's hand in a virtual scene, that is, the scene seen by the user includes one of his own hands (from the RGB image), and the rest are virtual scenes (from the virtual image). Suppose that due to light, coloring, etc., the color of one finger of the hand in the RGB image is obviously different from that of other fingers and the palm.
[0229] In the first embodiment, when only the first weight map is used, due to the accuracy of the depth map, the gaps between the user's fingers may not be accurately identified, and the user can see the object from the RGB image between the fingers;
[0230] The second embodiment fuses two weights, and the output effect produced is like "the user can clearly see his fingers and palms of normal colors, the display effect between the fingers is completely normal, and he can also see his finger of different color, but the edge of this finger is blurry". This effect is more in line with the general human feeling than using the first weight map or the second weight map alone. Therefore, the fusion effect of the second embodiment is better than that of the first embodiment.
[0231] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.
[0232] In order to implement the above-mentioned mixed reality virtual-reality fusion method, the present invention also provides a mixed reality virtual-reality fusion device, which can be composed of a chip architecture or other software and hardware forms.
[0233] Figure 6 The algorithm module diagram of the mixed reality virtual-real fusion device based on color model and guided filtering according to an embodiment of the present invention is disclosed. Figure 7 A hardware schematic diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention is disclosed. Figure 6 and Figure 7 As shown, the mixed reality virtual-real fusion device 600 proposed by the present invention includes an image information processor 610, a depth information processor 620, a central processing unit 630, and a display processing unit 640:
[0234] The image information processor 610 is connected to the central processor 630 and the display processing unit 640, and is used to obtain the RGB image of the real scene and send it to the central processor 630 and the display processing unit 640;
[0235] The depth information processor 620 is connected to the central processing unit 630 and is used to obtain a depth image of the real scene and send it to the central processing unit 630;
[0236] The central processing unit 630 is connected to the display processing unit 640, and is used to obtain the virtual depth, execute the mixed reality virtual-real fusion method according to the obtained RGB image, depth image and virtual depth of the real scene, and send the final fusion weight map to the display processing unit 640;
[0237] The display processing unit 640 obtains the final fusion weight map sent by the central processing unit 630, performs weighted fusion on the real-world RGB map and the virtual image, and mixes and generates a display image.
[0238] In this embodiment, the image information processor 610 is an AISC ISP chip, which is connected to the scene camera 650, obtains the collected data of the scene camera 650 and generates an RGB image.
[0239] In this embodiment, the depth information processor 620 is an NPU or other computing unit that can run a depth calculation algorithm, which is connected to the depth sensor 660 to obtain the collected data of the depth sensor 660 and generate a depth map.
[0240] In this embodiment, the central processing unit 630 is a CPU or other computing unit that can run the mixed reality virtual-real fusion method.
[0241] like Figure 6 and Figure 7 As shown, the display processing unit 640 generates a display image by fusing image warping and image aliasing. Image warping causes the input image to deform so that pixels corresponding to the same object are located at the same position. Image aliasing causes the various input information to be combined into a picture that the user finally views through the display device 680.
[0242] In this embodiment, the display processing unit 640 is an AISC display processing unit;
[0243] The display processing unit 640 is connected to the graphics processing unit (GPU) 670 to obtain the GPU's rendered image (virtual image) data;
[0244] The display processing unit 640 is connected to the image information processor 610 to obtain the RGB image of the image information processor;
[0245] The display processing unit 640 is connected to the central processing unit 630 to obtain the final fusion weight of the central processing unit 630;
[0246] The display processing unit 640 is connected to the display device 680 and outputs the mixed display image to the display device 680 .
[0247] In this embodiment, the display device 680 may be a head mounted display or glasses, etc.
[0248] When the implementation process file of the mixed reality virtual-real fusion method is a computer program, it can also be stored in a computer-readable storage medium as a product. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD)), a smart card, and a flash memory device (e.g., an electrically erasable programmable read-only memory (EPROM), a card, a stick, a key drive). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain and / or carry code and / or instructions and / or data.
[0249] The present invention provides a mixed reality virtual-real fusion method, device and medium, which use color modeling and edge-preserving filtering methods to calculate image virtual-real fusion parameters, taking into account historical and current RGB images to ensure the correct matching of depth images and RGB images, and can utilize the latest image information to effectively reduce display delay.
[0250] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.
[0251] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.
[0252] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.
[0253] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.
[0254] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.
[0255] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0256] The above embodiments are provided for persons familiar with the art to implement or use the present invention. Personnel familiar with the art can make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the protection scope of the present invention is not limited to the above embodiments, but should be the maximum scope of the innovative features mentioned in the claims.
Claims
1. A mixed reality virtual-real fusion method, characterized in that: The following steps are involved: Obtain real-world RGB images and depth images, as well as virtual images and virtual depths; Compare the depth image and the virtual depth to generate an occlusion relationship graph, and generate a color model based on the historical frame of the RGB image and the occlusion relationship graph; The latest frame of the RGB image is used to perform edge-preserving filtering on the color model to obtain the final fusion weight map; According to the final fusion weight map, the real-world RGB image and the virtual image are weightedly fused to generate a display image.
2. The mixed reality virtual-real fusion method according to claim 1, characterized in that: The comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frame of the RGB image and the occlusion relationship graph, further comprises: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image; Compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the history frame of the RGB image, and obtain a first occlusion relationship graph; A historical frame of the RGB image is selected and divided into blocks, and a color model is generated for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
3. The mixed reality virtual-real fusion method according to claim 1, characterized in that: The method further comprises: performing edge-preserving filtering on the color model using the latest frame of the RGB image to obtain a final fusion weight map. Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain the occlusion probability map; Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain a first weight map; The first weight map is taken as the final fusion weight map.
4. The mixed reality virtual-real fusion method according to claim 3, characterized in that: The step of obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes: The brightness component of the latest frame of the RGB image is obtained, and the brightness component is used to perform guided filtering on the occlusion probability map to obtain a first weight map.
5. The mixed reality virtual-real fusion method according to claim 3, characterized in that: The step of obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes: The latest frame of the RGB image is obtained, and the occlusion probability map is subjected to joint bilateral filtering to obtain the first weight map.
6. The mixed reality virtual-real fusion method according to claim 2, characterized in that: The following steps are also included: Performing morphological transformation on the first occlusion relationship graph to obtain a second weight graph; The first weight map and the second weight map are weightedly fused to obtain a third weight map as the final fused weight map.
7. The mixed reality virtual-real fusion method according to claim 1, characterized in that: The acquiring of the real-world RGB image and depth image further comprises: continuously acquiring the real-world RGB image, depth image and exposure timestamp at a preset frame rate; The acquiring of the virtual image and the virtual depth further includes: continuously acquiring the virtual image and the virtual depth of the virtual image at a preset frame rate.
8. The mixed reality virtual-real fusion method according to claim 1, characterized in that: After acquiring the real-world RGB image and depth image and acquiring the virtual image and virtual depth, the method further includes: downsampling all acquired images to low-resolution images; The method further includes: performing weighted fusion of the real-world RGB image and the virtual image according to the final fusion weight map, and before mixing and generating the display image, upsampling the final fusion weight map to the display resolution.
9. The mixed reality virtual-real fusion method according to claim 8, characterized in that: For the acquired RGB image and the virtual image, bilinear interpolation is used for downsampling.
10. The mixed reality virtual-real fusion method according to claim 8, characterized in that: The acquired depth image and virtual depth are downsampled using nearest neighbor interpolation.
11. The mixed reality virtual-real fusion method according to claim 2, characterized in that: The marking of occlusion relationships in the RGB image further comprises: The RGB image is divided into three parts: the virtual image display area, the real image display area and the pending area; marking the displayed virtual image area as a first coefficient; The area showing the true image is marked as the second coefficient; The undetermined area is marked as the third coefficient.
12. The mixed reality virtual-real fusion method according to claim 11, characterized in that: The marking of occlusion relationships in the RGB image further comprises: Mark the area where the depth is missing as a pending area; Areas where virtual depth is missing are not marked.
13. The mixed reality virtual-real fusion method according to claim 6, characterized in that: The performing morphological transformation on the first occlusion relationship graph to obtain the second weight graph further includes: Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph; Edge processing is performed on the second occlusion relationship graph to obtain a second weight graph.
14. The mixed reality virtual-real fusion method according to claim 13, characterized in that: The performing edge processing on the second occlusion relationship graph to obtain a second weight graph further includes: Filtering and ternary processing the second occlusion relationship graph to obtain a third occlusion relationship graph; The third occlusion relationship graph is filtered to obtain a second weight graph.
15. The mixed reality virtual-real fusion method according to claim 2, characterized in that: The color model includes a color histogram; Generating color models for different blocked areas in each image block further includes: performing color histogram statistics on the unblocked areas and blocked areas in each image block.
16. The mixed reality virtual-real fusion method according to claim 15, characterized in that: When performing color histogram statistics, the color depth is downsampled.
17. The mixed reality virtual-real fusion method according to claim 3, characterized in that: The selecting the latest frame of the RGB image, estimating the probability of pixel points being occluded according to the color model of the historical frame of the RGB image, and obtaining an occlusion probability map further includes: Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph; The latest frame of the RGB image is selected, and according to the position of the undetermined area in the second occlusion relationship diagram, the occlusion probability of each pixel point in the corresponding area of the latest frame of the RGB image is estimated to obtain an occlusion probability map.
18. The mixed reality virtual-real fusion method according to claim 13, characterized in that: The estimating the occlusion probability of each pixel point in the corresponding position area of the latest frame of the RGB image according to the position of the undetermined area in the second occlusion relationship diagram to obtain the occlusion probability map further includes: Select several blocks of images closest to the current pixel in the historical frame of the RGB image; According to the color models of several blocks of images, the color model of the area close to the current pixel is estimated; According to the estimated color model, calculate the distribution of the color of the current pixel in different occlusion areas; Estimate the probability that the current pixel is blocked based on the distribution; Regularization constraints are applied to the probability values to obtain the occlusion probability map.
19. The mixed reality virtual-real fusion method according to claim 4, characterized in that: The brightness component further includes a Y channel in the YUV space, an L channel in the Lab space, an I channel in the HSI space, and an L channel in the HSL space.
20. A mixed reality virtual-real fusion device, characterized in that: Including image information processor, depth information processor, central processing unit, display processing unit: The image information processor is connected to the central processing unit and the display processing unit, and is used to obtain the RGB image of the real scene and send it to the central processing unit and the display processing unit; The depth information processor is connected to the central processing unit and is used to obtain a depth image of the real scene and send it to the central processing unit; The central processing unit is connected to the display processing unit, and is used to obtain virtual depth, execute the mixed reality virtual-real fusion method according to any one of claims 1 to 19, and send the final fusion weight map to the display processing unit; The display processing unit obtains the final fusion weight map sent by the central processing unit, performs weighted fusion on the real-world RGB image and the virtual image, and mixes and generates a display image.
21. The mixed reality virtual-real fusion device according to claim 20, characterized in that: The image information processor is connected to the scene camera, obtains the collected data of the scene camera and generates an RGB image; The depth information processor is connected to the depth sensor, obtains the collected data of the depth sensor and generates a depth image.
22. A mixed reality virtual-real fusion system, characterized in that: Including mixed reality virtual and real fusion device, scene camera, depth sensor, graphics processor and display device: The mixed reality virtual-reality fusion device adopts the mixed reality virtual-reality fusion device as described in any one of claims 20-21; The scene camera is used to collect RGB images of the real scene and send them to the mixed display virtual and real device; The depth sensor is used to collect the depth image of the real scene and send it to the mixed display virtual and real device; The image processor is used to generate a virtual image and a virtual depth and send them to a mixed display virtual-real device; The display device is used to receive and display the display image generated by the mixed display virtual and real device.
23. A computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 19 is performed.
Citation Information
Patent Citations
Method for generating virtual-real fusion image for stereo display
CN104504671A
Occlusion detection
US11170578B1
Methods for handling occlusion in augmented reality applications using memory and device tracking and related apparatus
US20230014448A1
