Virtual-real fusion method and apparatus for mixed reality, and system
By generating occlusion relationship diagrams and color models, combined with edge preservation filtering technology, the problems of poor display effects and delays in the existing technology of virtual and real fusion scenes are solved, and efficient fusion of virtual objects and real objects is achieved, improving user experience.
Patent Information
- Application Number
- PCT/CN2024/089588
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-04-24
- Publication Date
- 2024-12-05
AI Technical Summary
The existing mixed reality system has poor display effect and has a large delay in the fusion scenario of virtual and real objects, making it difficult to achieve seamless connection between virtual objects and real objects, resulting in poor user experience.
By obtaining real-world RGB images and depth images, comparing depth images and virtual depth to generate occlusion relationship diagrams, generating color models based on historical frames and occlusion relationship diagrams, and processing the color models using edge-preserving filtering technology to obtain the final fusion rights Value graphs, used to weighted fuse real world and virtual images to generate display images.
It effectively reduces display delay, improves the realism and user experience of the fusion scene of virtual and real, ensures the correct matching of depth images with RGB images, and uses the latest image information to improve the frame rate and display effect.
Smart Images

Figure CN2024089588_05122024_PF_FP_ABST
Abstract
Description
A mixed reality virtual-real fusion method, device and system
[0001] This invention claims priority to a patent application with a filing date of May 30, 2023, Chinese application number 202310629358.X, and titled “A method, device and system for the fusion of mixed reality and virtuality”. Technical Field
[0002] The present invention relates to the technical field of machine vision and image fusion, and more specifically, to a mixed reality virtual-real fusion method, device, system, and medium. Background Art
[0003] Mixed reality (MR) technology presents virtual scene information in real scenes, building an interactive feedback information loop between the real world, the virtual world and the user to enhance the realism of the user experience.
[0004] Mixed reality systems have high requirements for the display realism of virtual objects. The purpose is to seamlessly connect virtual objects with real objects, maintain the immersiveness of the scene and the realism of virtual objects. This requirement also poses new challenges to display technology.
[0005] In existing mixed reality system architectures, virtual objects are presented through rendering technology in virtual-reality fusion scenes. To determine the front-to-back occlusion relationship between virtual objects and real objects during the virtual-reality fusion process, the following three display strategies are typically used:
[0006] 1) Virtual objects are always displayed before real objects;
[0007] 2) Try to obtain as accurate depth information as possible and directly compare the depth of real objects with virtual objects;
[0008] 3) Obtain certain information from the image of the real object and combine it with the incompletely accurate depth for cutout.
[0009] For the first display strategy, the display effect is not realistic enough and the user experience is poor;
[0010] The second display strategy requires a significant amount of computation to process the depth map, potentially preventing real-time computation. Existing algorithms typically take an average of 48.4ms, resulting in a display delay of approximately 116ms. This means it can only support a maximum of 20 FPS of virtual-reality integration, with noticeable delays.
[0011] For the third display strategy, existing algorithms typically require that the depth map and RGB image be obtained at the same time. However, in wearable devices, due to power and heat limitations, the latency for acquiring depth maps is often much higher than that for acquiring RGB images, and the frame rate is often lower than that of RGB images. This makes it difficult for the algorithm to use the latest RGB images, which also increases the overall latency of the display system.
[0012] Based on the above analysis, the existing technology has poor display effects on virtual-reality fusion scenes of mixed reality systems, and there are delays that are difficult to be effectively solved.
[0013] Summary of the Invention
[0014] The purpose of the present invention is to provide a mixed reality virtual-reality fusion method, device, system, and medium to solve the problems of poor display effect and large delay in the existing virtual reality fusion technology.
[0015] In order to achieve the above object, the present invention provides a mixed reality virtual-real fusion method, comprising the following steps:
[0016] Obtain real-world RGB images, depth images, and virtual images and virtual depth;
[0017] Compare the depth image and the virtual depth to generate an occlusion relationship graph, and generate a color model based on the historical frames of the RGB image and the occlusion relationship graph;
[0018] The latest frame of the RGB image is used to perform edge-preserving filtering on the color model to obtain the final fusion weight map;
[0019] According to the final fusion weight map, the real-world RGB image and the virtual image are weightedly fused to generate a mixed display image.
[0020] In some embodiments, the comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frames of the RGB image and the occlusion relationship graph, further includes:
[0021] Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image;
[0022] Comparing the depth image with the virtual depth of the virtual image, marking the occlusion relationship in the history frame of the RGB image, and obtaining a first occlusion relationship graph;
[0023] A historical frame of the RGB image is selected and divided into blocks, and a color model is generated for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
[0024] In some embodiments, performing edge-preserving filtering on the color model using the latest frame of the RGB image to obtain a final fusion weight map further includes:
[0025] Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain the occlusion probability map;
[0026] Get the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain the first weight map;
[0027] The first weight map is used as the final fusion weight map.
[0028] In some embodiments, obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes:
[0029] The brightness component of the latest frame of the RGB image is obtained, and the brightness component is used to perform guided filtering on the occlusion probability map to obtain a first weight map.
[0030] In some embodiments, obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes:
[0031] Get the latest frame of the RGB image, perform joint bilateral filtering on the occlusion probability map, and obtain the first weight map.
[0032] In some embodiments, the mixed reality virtual-real fusion method further includes the following steps:
[0033] Performing morphological transformation on the first occlusion relationship graph to obtain a second weight graph;
[0034] The first weight map and the second weight map are weightedly fused to obtain a third weight map as the final fused weight map.
[0035] In some embodiments, acquiring the real-world RGB image and depth image further comprises: continuously acquiring the real-world RGB image, depth image, and exposure timestamp at a preset frame rate;
[0036] The acquiring of the virtual image and the virtual depth further includes: continuously acquiring the virtual image and the virtual depth of the virtual image at a preset frame rate.
[0037] In some embodiments, after acquiring the real-world RGB image and depth image and acquiring the virtual image and virtual depth, the method further includes: downsampling all acquired images to low-resolution images;
[0038] The method further includes: performing weighted fusion of the real-world RGB image and the virtual image according to the final fusion weight map, and before mixing to generate the display image, upsampling the final fusion weight map to the display resolution.
[0039] In some embodiments, bilinear interpolation is used to downsample the acquired RGB image and the virtual image.
[0040] In some embodiments, the acquired depth image and virtual depth are down-sampled using nearest neighbor interpolation.
[0041] In some embodiments, marking the occlusion relationship in the RGB image further includes:
[0042] The RGB image is divided into three parts: the virtual image display area, the real image display area and the pending area;
[0043] Marking the displayed virtual image area as the first coefficient;
[0044] The area showing the true image is marked as the second coefficient;
[0045] The undetermined area is marked as the third coefficient.
[0046] In some embodiments, marking the occlusion relationship in the RGB image further includes:
[0047] Mark the area with missing depth as pending area;
[0048] The areas where virtual depth is missing are not marked.
[0049] In some embodiments, performing a morphological transformation on the first occlusion relationship graph to obtain a second weight graph further includes:
[0050] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0051] Edge processing is performed on the second occlusion relationship graph to obtain a second weight graph.
[0052] In some embodiments, performing edge processing on the second occlusion relationship graph to obtain a second weight graph further includes:
[0053] Filtering and ternary processing the second occlusion relationship graph to obtain a third occlusion relationship graph;
[0054] The third occlusion relationship graph is filtered to obtain a second weight graph.
[0055] In some embodiments, the color model includes a color histogram;
[0056] Generating color models for different blocked areas in each image block further includes: performing color histogram statistics on the unblocked areas and blocked areas in each image block.
[0057] In some embodiments, when performing color histogram statistics, the color depth is downsampled.
[0058] In some embodiments, selecting the latest frame of the RGB image and estimating the probability of pixel occlusion based on the color model of the historical frames of the RGB image to obtain the occlusion probability map further includes:
[0059] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0060] The latest frame of the RGB image is selected, and according to the position of the undetermined area in the second occlusion relationship map, the occlusion probability of each pixel point in the corresponding area of the latest frame of the RGB image is estimated to obtain an occlusion probability map.
[0061] In some embodiments, estimating the occlusion probability of each pixel point in the corresponding position area of the latest frame of the RGB image based on the position of the undetermined area in the second occlusion relationship map to obtain the occlusion probability map further includes:
[0062] Select several blocks of images closest to the current pixel in the historical frame of the RGB image;
[0063] Estimate the color model of the area close to the current pixel based on the color models of several image blocks;
[0064] According to the estimated color model, calculate the distribution of the color of the current pixel in different occlusion areas;
[0065] Estimate the probability that the current pixel is blocked based on the distribution;
[0066] Regularization constraints are applied to the probability values to obtain the occlusion probability map.
[0067] In some embodiments, the brightness component further includes a Y channel in a YUV space, an L channel in a Lab space, an I channel in an HSI space, and an L channel in an HSL space.
[0068] In order to achieve the above-mentioned object, the present invention provides a mixed reality virtual-real fusion device, comprising an image information processor, a depth information processor, a central processing unit, and a display processing unit:
[0069] The image information processor is connected to the central processing unit and the display processing unit, and is used to obtain the RGB image of the real world and send it to the central processing unit and the display processing unit;
[0070] The depth information processor is connected to the central processing unit and is used to obtain a depth image of the real scene and send it to the central processing unit;
[0071] The central processing unit is connected to the display processing unit, and is used to obtain virtual depth, execute any of the mixed reality virtual and real fusion methods described above, and send the final fusion weight map to the display processing unit;
[0072] The display processing unit obtains the final fusion weight map sent by the central processing unit, performs weighted fusion on the real-world RGB image and the virtual image, and mixes them to generate a display image.
[0073] In one embodiment, the image information processor is connected to the scene camera, obtains the collected data of the scene camera and generates an RGB image;
[0074] The depth information processor is connected to the depth sensor, obtains the data collected by the depth sensor and generates a depth image.
[0075] To achieve the above objectives, the present invention provides a mixed reality virtual-reality fusion system, including a mixed reality virtual-reality fusion device, a scene camera, a depth sensor, a graphics processor, and a display device:
[0076] The mixed reality virtual-real fusion device adopts any one of the mixed reality virtual-real fusion devices described above;
[0077] The scene camera is used to capture RGB images of the real scene and send them to the mixed display virtual and real device;
[0078] The depth sensor is used to collect the depth image of the real scene and send it to the mixed display virtual and real device;
[0079] The image processor is used to generate a virtual image and a virtual depth and send them to a mixed display virtual-real device;
[0080] The display device is used to receive and display the display image generated by the mixed display virtual and real device.
[0081] In order to achieve the above object, the present invention provides a computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, any of the above methods is executed.
[0082] The present invention provides a mixed reality virtual-reality fusion method, device, system, and medium, which use color modeling and edge-preserving filtering methods to calculate image virtual-reality fusion parameters, taking into account historical and current RGB images to ensure the correct matching of depth images and RGB images, and can also utilize the latest image information to effectively reduce display delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0083] The above and other features, properties and advantages of the present invention will become more apparent through the following description in conjunction with the accompanying drawings and embodiments, in which like reference numerals represent like features throughout, wherein:
[0084] FIG1 discloses a flow chart of a method for merging virtual and real mixed reality according to an embodiment of the present invention;
[0085] FIG2 discloses a flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a first embodiment of the present invention;
[0086] FIG3 shows a schematic diagram of image segmentation according to a first embodiment of the present invention;
[0087] FIG4 discloses a flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a second embodiment of the present invention;
[0088] FIG5 discloses a workflow diagram of a mixed reality virtual-real fusion algorithm based on a color model and guided filtering according to a second embodiment of the present invention;
[0089] FIG6 discloses an algorithm module diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention;
[0090] FIG7 illustrates a hardware diagram of a mixed reality virtual-real fusion device according to an embodiment of the present invention.
[0091] The meanings of the reference numerals in the figures are as follows:
[0092] 610 image information processor;
[0093] 620 depth information processor;
[0094] 630 CPU;
[0095] 640 display processing unit;
[0096] 650 scene cameras;
[0097] 660 depth sensor;
[0098] 670 graphics processor. DETAILED DESCRIPTION
[0099] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention and are not intended to limit the invention.
[0100] FIG1 discloses a flow chart of a mixed reality virtual-real fusion method according to an embodiment of the present invention. As shown in FIG1 , the present invention provides a mixed reality virtual-real fusion method, comprising the following steps:
[0101] Step S1: Acquire a real-world RGB image and depth image, and acquire a virtual image and virtual depth;
[0102] Step S2: Compare the depth image and the virtual depth to generate an occlusion relationship graph, and generate a color model based on the historical frame of the RGB image and the occlusion relationship graph;
[0103] Step S3: Use the latest frame of the RGB image to perform edge-preserving filtering on the color model to obtain the final fusion weight map;
[0104] Step S4: Perform weighted fusion on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0105] Furthermore, the step S2, comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frames of the RGB image and the occlusion relationship graph, further includes:
[0106] Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image;
[0107] Comparing the depth image with the virtual depth of the virtual image, marking the occlusion relationship in the history frame of the RGB image, and obtaining a first occlusion relationship graph;
[0108] A historical frame of the RGB image is selected and divided into blocks, and a color model is generated for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
[0109] Furthermore, the step S3, performing edge-preserving filtering on the color model using the latest frame of the RGB image to obtain the final fusion weight map, further includes:
[0110] Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain the occlusion probability map;
[0111] Get the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain the first weight map;
[0112] The first weight map is used as the final fusion weight map.
[0113] Edge Preserving Filter is a special filter that can effectively preserve the edge information in the image during the filtering process.
[0114] Among them, the joint bilateral filter, the guided image filter, and the weighted least square filter are several well-known edge-preserving filters.
[0115] The following uses an edge-preserving filter as a guide filter as an example to illustrate the mixed reality virtual-real fusion method proposed in the present invention. It should be noted that the mixed reality virtual-real fusion method proposed in the present invention can also be implemented by using other edge-preserving filters.
[0116] FIG2 discloses a flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a first embodiment of the present invention. In the first embodiment shown in FIG2 , the present invention proposes a mixed reality virtual-real fusion method based on a color model and guided filtering, which specifically includes the following steps:
[0117] Step S101: Acquire an RGB image and a depth image of a real scene, and acquire a virtual image and a virtual depth;
[0118] Step S102: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image, compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the historical frame of the RGB image, and obtain a first occlusion relationship graph;
[0119] Step S103: selecting historical frames of the RGB image and dividing them into blocks, and generating color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram;
[0120] Step S104: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain an occlusion probability map;
[0121] Step S105: Obtain the brightness component of the latest frame of the RGB image, and use the brightness component to perform guided filtering on the occlusion probability map to obtain a first weight map;
[0122] Step S106: using the first weight map as the final fusion weight map;
[0123] Step S107: Perform weighted fusion on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0124] It should be understood that within the scope of the present invention, the above-mentioned technical features of the present invention and the technical features specifically described below (such as in the embodiments) can be combined with each other and interrelated to form a preferred technical solution.
[0125] Step S101: Acquire a real-world RGB image and depth image, and acquire a virtual image and virtual depth.
[0126] Depth information refers to the distance between an object and the camera in a real scene. The camera's shooting plane can be used as the X and Y axes of the three-dimensional coordinate system. Then, the depth represents the Z-axis distance. Therefore, depth information can be used to determine the occlusion relationship.
[0127] In some embodiments, the RGB image, depth image, and corresponding exposure timestamps of the real scene are continuously acquired at a preset frame rate. Preferably, the preset frame rate is a fixed frame rate. In other embodiments, the preset frame rate allows for a certain range of latency but cannot exceed a latency threshold.
[0128] In some embodiments, the virtual image and its virtual depth (Z-buffer) are continuously acquired at a preset frame rate. Preferably, the preset frame rate is a fixed frame rate. In other embodiments, the preset frame rate allows for a certain range of latency but does not exceed a latency threshold.
[0129] In this embodiment, the virtual image and the depth of the virtual image are continuously obtained from the graphics processor. The virtual image refers to the rendered image provided by the graphics processor.
[0130] Optionally, after acquiring all images (RGB images, depth images, virtual images, and virtual depths), all acquired images are downsampled to lower resolution images to meet computing power requirements and ensure subsequent operating efficiency.
[0131] Obviously, the "lower" resolution here is a relative concept relative to computing power. When the image is downsampled to a resolution that meets the computing power requirements, it can be considered as a lower resolution.
[0132] Downsampling methods include but are not limited to bilinear interpolation and nearest neighbor interpolation methods.
[0133] Preferably, bilinear interpolation is used to downsample the acquired RGB image and virtual image;
[0134] Preferably, the acquired depth map and virtual depth are down-sampled using nearest neighbor interpolation.
[0135] Step S102: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image, compare the depth image and the virtual depth of the virtual image, mark the occlusion relationship in the historical frame of the RGB image, and obtain a first occlusion relationship diagram.
[0136] In this embodiment, the historical frame of the virtual image further refers to a historical frame in the virtual image that has an exposure time closest to the latest frame of the depth image.
[0137] In this embodiment, the historical frame of the RGB image further refers to a historical frame in the RGB image that has the closest exposure time to the latest frame of the depth image, which is recorded as the first RGB image.
[0138] Obviously, the exposure time can be determined by the exposure timestamp obtained in step S101.
[0139] In this embodiment, the first occlusion relationship graph is a first trigraph.
[0140] The first three-part image divides the image into three parts: the virtual image area, the real image area, and the undetermined area:
[0141] The area where the virtual image is displayed, that is, the area where the virtual image is not blocked, is marked as the first coefficient. In this embodiment, the first coefficient is 0.
[0142] The area showing the real image, that is, the area blocked by the virtual image, is marked as the second coefficient. In this embodiment, the second coefficient is 1;
[0143] The undetermined area, that is, the area where it is not yet certain whether to display a virtual image or a real image, is marked as the third coefficient. In this embodiment, the third coefficient is 0.5.
[0144] Furthermore, the depth-missing regions are marked as pending;
[0145] Furthermore, the area where the virtual depth is missing is not marked.
[0146] Step S103: select historical frames of the RGB image and divide them into blocks, and generate color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
[0147] In this embodiment, selecting historical frames of the RGB image and dividing them into blocks further includes:
[0148] According to the exposure timestamp, an RGB image frame with the exposure time closest to the latest frame of the depth image is selected as the first RGB image, and the first RGB image is divided into several non-overlapping blocks.
[0149] Furthermore, the first RGB image is divided into a number of non-overlapping blocks. The specific division method is not too limited. The following examples illustrate two division methods as preferred implementations, but it cannot be considered that the division methods proposed in the present invention are limited to the following two:
[0150] For example, the entire image may be divided into a fixed number of rectangular blocks, such as 8 blocks in the horizontal direction and 8 blocks in the vertical direction, for a total of 8*8=64 blocks, and the blocks do not overlap with each other.
[0151] For example, the entire image may be divided into rectangular blocks of a fixed size, such as blocks of 32 pixels×32 pixels, where the blocks do not overlap with each other.
[0152] In other embodiments, the image may be divided into blocks of different shapes or blocks that overlap with each other in a certain manner.
[0153] In this embodiment, the color model is a color histogram, and generating color models for different occluded areas in each image further includes: performing color histogram statistics on the unoccluded area and the occluded area in each image.
[0154] As a more general method, the color histogram mainly describes the quantitative characteristics of the color in the image, which can reflect the statistical distribution and basic hue of the image color. Its function is to generate color statistical information for estimating the occlusion probability map, which indicates the frequency of occurrence of various colors in a certain area of the image.
[0155] In other embodiments, the color model may also use other alternatives, such as a Gaussian mixture model (GMM).
[0156] Furthermore, when performing color histogram statistics, the color depth is downsampled. Optionally, the color depth can be downsampled to 4-bit color or lower.
[0157] Step S104: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain an occlusion probability map;
[0158] The step S104 further includes the following steps:
[0159] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0160] The latest frame of the RGB image is selected, and according to the position of the undetermined area in the second occlusion relationship map, the occlusion probability of each pixel point in the corresponding area of the latest frame of the RGB image is estimated to obtain an occlusion probability map.
[0161] In this embodiment, the second occlusion relationship graph is a second trisection graph, and the second trisection graph is obtained by performing morphological dilation on the undetermined region in the first trisection graph.
[0162] Morphological dilation refers to the expansion of the boundary points of a binary object, merging all background points that are in contact with the object into the object, and expanding the boundary outward.
[0163] Morphological expansion is generally divided into horizontal expansion, vertical expansion and omnidirectional expansion.
[0164] In this embodiment, a 3*3 rectangular kernel is used for morphological dilation.
[0165] The RGB image of the latest frame is selected and recorded as the second RGB image. Obviously, in this embodiment, the latest frame refers to the latest exposure time.
[0166] According to the position of the area marked as undetermined in the second trigraph, the probability of each pixel point in the corresponding area of the second RGB image being occluded is estimated to obtain an occlusion probability map, which specifically includes the following steps:
[0167] Selecting several blocks of images closest to the current pixel in the historical frame of the RGB image in step S103;
[0168] Estimate the color model of the area close to the current pixel based on the color models of the several image blocks;
[0169] According to the estimated color model, calculate the distribution of the color of the current pixel in different occlusion areas;
[0170] Estimate the probability that the current pixel is blocked based on the distribution;
[0171] Regularization constraints are applied to the probability values to obtain the occlusion probability map.
[0172] Figure 3 reveals a schematic diagram of image segmentation according to the first embodiment of the present invention. As shown in Figure 3, in this embodiment, the image is divided into non-overlapping rectangular blocks, and the four image blocks closest to the current pixel are selected. Based on the color histogram statistics of these four image blocks, the color histogram near the current pixel is estimated, and based on the estimated color histogram, the distribution of the color of the current pixel in different occlusion areas is calculated.
[0173] The four images closest to the current pixel are determined by the four images with the smallest Euclidean distance between the geometric center and the pixel position.
[0174] For all rectangular grids and non-overlapping block division methods, the most recent 4 blocks of images are obviously the preferred embodiment. As shown in FIG3 , the color histograms of the four gray blocks are used when calculating the probability values of the dot pixels.
[0175] In other embodiments, more blocks are allowed to be used, for example, 16 blocks are allowed to be used.
[0176] Due to the possibility of other block partitioning methods, the definition of "nearest" and "closest" can be modified, such as the blocks closest in the vertical or horizontal direction, the blocks closest to each other after weighted distance, etc. Therefore, the representation scope of "nearest" and "closest" is not limited to the "minimum Euclidean distance" method.
[0177] In this embodiment, calculating the distribution of the current pixel in different occlusion areas and estimating the probability of the current pixel being occluded based on the distribution further includes:
[0178] Use the color histogram to calculate the frequency of occurrence of the color of the current pixel in the areas marked as 0 and 1 in the four nearby rectangular blocks;
[0179] And these frequencies are weighted according to the distance from the geometric center of the block to the "current pixel";
[0180] Sum the frequency of occurrence of the 0 region and the frequency of occurrence of the 1 region to obtain the sum value;
[0181] Compare the frequency of occurrence of region 1 with the obtained sum, and use the ratio as the probability that the current pixel is occluded.
[0182] Regularization is to impose constraints on the minimization of the empirical error function, and such constraints can be interpreted as prior knowledge.
[0183] Step S105 : Obtain the brightness component of the latest frame of the RGB image, and use the brightness component to perform guided filtering on the occlusion probability map to obtain a first weight map.
[0184] The brightness component further includes but is not limited to the Y channel in the YUV space, the L channel in the Lab space, the I channel in the HSI space, the L channel in the HSL space, etc.
[0185] YUV is a color encoding method, where Y represents brightness. The Lab color space is a color-opponent space with the dimension L representing luminance. HSI is a digital image model, where luminance I refers to the brightness of the image and the grayscale of the image, representing the brightness of the color. HSL is a cylindrical coordinate representation of points in the RGB color model, where L represents luminance.
[0186] In this embodiment, the RGB image of the latest frame is recorded as the second RGB image. The second RGB image is converted into a color space containing a brightness component, and the brightness channel thereof is taken as a guide image. The occlusion probability map is subjected to guided filtering to obtain a first weight map.
[0187] Guided filtering is used to transfer information from the RGB image to the triplicate image and generate fusion weights.
[0188] Guided filtering, also known as guided graph filtering, is an image filtering technique that filters a target image P (input image) through a guide graph G, so that the final output image is generally similar to the target image P, but the texture part is similar to the guide graph G.
[0189] Guided filtering, as an edge-preserving filter, has been widely used in image dehazing and matting techniques in recent years. Compared to joint bilateral filtering, guided filtering not only achieves comparable filtering results in edge preservation and detail smoothing, but also offers advantages in computational speed and filtering efficiency. This is because its core process can be rapidly implemented through a windowed summation operation when the integral image is known, resulting in high algorithmic efficiency and low complexity. Its edge-preserving capability precisely meets the requirements of depth map optimization, while its efficient real-time operation makes it highly competitive in practice.
[0190] Step S106: using the first weight map as the final fusion weight map;
[0191] The parameters of the final fusion weight map are used as the image virtual-real fusion parameters.
[0192] Step S107: Perform weighted fusion on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0193] Furthermore, the guided filter can be used again in the display pipeline to upsample the final fusion weight map to the display resolution, and the RGB image and the virtual image can be mixed into the display image.
[0194] The first embodiment of the present invention proposes a mixed reality virtual-reality fusion method based on a color model and guided filtering. It obtains the initial tripartite map and color modeling information from the historical RGB image to ensure depth matching. The color modeling information is then reconstructed into a new tripartite map in the latest RGB image to ensure that the final effect is applied to the latest image, reducing latency.
[0195] FIG4 discloses a flow chart of a mixed reality virtual-real fusion method based on a color model and guided filtering according to a second embodiment of the present invention. In the second embodiment shown in FIG4 , the present invention proposes a mixed reality virtual-real fusion method based on a color model and guided filtering, which specifically includes the following steps:
[0196] Step S201: Acquire an RGB image and a depth image of a real scene, and acquire a virtual image and a virtual depth;
[0197] Step S202: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image, compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the historical frame of the RGB image, and obtain a first occlusion relationship graph;
[0198] Step S203: select historical frames of the RGB image and divide them into blocks, and generate color models for different occlusion areas of each block of the image according to the first occlusion relationship diagram;
[0199] Step S204: Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain an occlusion probability map;
[0200] Step S205: Obtain the brightness component of the latest frame of the RGB image, and use the brightness component to perform guided filtering on the occlusion probability map to obtain a first weight map;
[0201] Step S206: performing morphological transformation on the first occlusion relationship graph to obtain a second weight graph;
[0202] Step S207: Perform weighted fusion on the first weight map and the second weight map to obtain a third weight map as the final fused weight map;
[0203] Step S208: Perform weighted fusion on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0204] Figure 5 reveals a module diagram of a mixed reality virtual-real fusion algorithm based on a color model and guided filtering according to an embodiment of the present invention. As shown in Figures 4 and 5, in the second embodiment and the first embodiment, steps S201 to S205 are the same as steps S101 to S105, and step S208 is the same as step S107. The difference lies in step S206 and step S207, which are the dotted parts in Figure 4.
[0205] Steps S206 and S207 will be described in detail below.
[0206] The step S206 further includes:
[0207] Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph;
[0208] Edge processing is performed on the second occlusion relationship graph to obtain a second weight graph.
[0209] Furthermore, performing edge processing on the second occlusion relationship graph to obtain a second weight graph further includes:
[0210] Filtering and ternary processing the second occlusion relationship graph to obtain a third occlusion relationship graph;
[0211] The third occlusion relationship graph is filtered to obtain a second weight graph.
[0212] In this embodiment, the first occlusion relationship graph is a first three-division graph, the second occlusion relationship graph is a second three-division graph, and the third occlusion relationship graph is a third three-division graph.
[0213] In this embodiment, filtering and ternary processing are performed on the second three-division image, further comprising:
[0214] Perform mean filtering on the second three-part image, and then perform ternary processing using the third coefficient as the threshold;
[0215] Among them, the third coefficient is 0.5.
[0216] In this embodiment, the ternary processing method is:
[0217] For the value greater than the third coefficient after filtering, make it the second coefficient, for the value less than the third coefficient after filtering, make it the first coefficient, and for the value equal to the third coefficient after filtering, make it the third coefficient;
[0218] Among them, the first coefficient is 0, the second coefficient is 1, and the third coefficient is 0.5.
[0219] In this embodiment, filtering the third three-division image further includes:
[0220] Perform mean filtering on the third trigraph.
[0221] It should be noted that in this embodiment, mean filtering is performed on both the second and third trigraphs. In other embodiments, different filters may be used for the second and third trigraphs. Other low-pass image filters may be used, such as Gaussian filtering or bilateral filtering with a precisely designed threshold.
[0222] Step S207: Perform weighted fusion on the first weight map and the second weight map to obtain a third weight map as the final fused weight map.
[0223] The first weight map and the second weight map are weightedly fused according to a certain calculation rule to obtain a third weight map, and the third weight map is output as the final virtual-real fusion weight.
[0224] The calculation rule of weighted fusion may be to perform fusion by confidence. The confidence may be generated based on the value of the first weight map, or may be an externally generated confidence, such as depth confidence.
[0225] Step S208: Perform weighted fusion on the real-world RGB image and the virtual image according to the final fusion weight map to generate a display image.
[0226] Furthermore, the guided filter can be used again in the display pipeline to upsample the third weight map to the display resolution, and the RGB image and the virtual image can be mixed into a display image.
[0227] In the first embodiment, the first weight map is used as the final fusion weight map. The first weight map is related to the depth map. However, compared with the RGB map, the depth map has lower accuracy and higher latency, and can only obtain blurred edges and details.
[0228] The second weight map is related to the RGB image content and can obtain accurate edges and details.
[0229] Compared with the first embodiment, the second embodiment adopts a multi-weight joint calculation method to perform weighted fusion on the second weight map and the first weight map to ensure that the edge transition is not abrupt.
[0230] For example, in some cases, it is necessary to display the user's hand in a virtual scene. That is, the scene the user sees contains one of their own hands (from the RGB image), and the rest is the virtual scene (from the virtual image). Suppose that due to lighting, coloring, etc., the color of one finger of the hand in the RGB image is significantly different from the other fingers and the palm.
[0231] In the first embodiment, when only the first weight map is used, due to the accuracy of the depth map, the gaps between the user's fingers may not be accurately recognized, and the user may be able to see objects from the RGB image between the fingers;
[0232] The second embodiment fuses the two weights, producing an output effect similar to "the user can clearly see their normally colored fingers and palm, with the interdigital spaces between the fingers displaying perfectly normal. The user can also see their differently colored finger, but the edges of that finger are blurred." This effect is more consistent with human perception than using either the first or second weight map alone. Therefore, the fusion effect of the second embodiment is superior to that of the first embodiment.
[0233] Although the above methods are illustrated and described as a series of acts for simplicity of explanation, it is to be understood and appreciated that these methods are not limited by the order of the acts, as some acts may occur in a different order and / or concurrently with other acts from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art according to one or more embodiments.
[0234] In order to implement the above-mentioned mixed reality virtual-reality fusion method, the present invention also provides a mixed reality virtual-reality fusion device, which can be composed of a chip architecture or other software and hardware forms.
[0235] FIG6 discloses an algorithm module diagram of a mixed reality virtual-reality fusion device based on a color model and guided filtering according to an embodiment of the present invention. FIG7 discloses a hardware schematic diagram of a mixed reality virtual-reality fusion device according to an embodiment of the present invention. As shown in FIG6 and FIG7 , the mixed reality virtual-reality fusion device 600 proposed by the present invention includes an image information processor 610, a depth information processor 620, a central processing unit 630, and a display processing unit 640:
[0236] The image information processor 610 is connected to the central processing unit 630 and the display processing unit 640, and is used to obtain the RGB image of the real scene and send it to the central processing unit 630 and the display processing unit 640;
[0237] The depth information processor 620 is connected to the central processing unit 630 and is used to obtain a depth image of the real scene and send it to the central processing unit 630;
[0238] The central processing unit 630 is connected to the display processing unit 640 and is used to obtain virtual depth, execute the mixed reality virtual and real fusion method based on the obtained RGB image, depth map and virtual depth of the real scene, and send the final fusion weight map to the display processing unit 640;
[0239] The display processing unit 640 obtains the final fusion weight map sent by the central processing unit 630, performs weighted fusion on the real-world RGB map and the virtual image, and mixes them to generate a display image.
[0240] In this embodiment, the image information processor 610 is an AISC ISP chip, which is connected to the scene camera 650 to obtain the collected data of the scene camera 650 and generate an RGB image.
[0241] In this embodiment, the depth information processor 620 is an NPU or other computing unit that can run a depth calculation algorithm, which is connected to the depth sensor 660 to obtain the collected data of the depth sensor 660 and generate a depth map.
[0242] In this embodiment, the central processing unit 630 is a CPU or other computing unit that can run the mixed reality virtual-real fusion method.
[0243] As shown in Figures 6 and 7, the display processing unit 640 generates a display image by fusing image warping and image aliasing. Image warping causes the input image to deform so that pixels corresponding to the same object are located in the same position. Image aliasing combines the various input information into the image that the user finally views through the display device 680.
[0244] In this embodiment, the display processing unit 640 is an AISC display processing unit;
[0245] The display processing unit 640 is connected to the graphics processing unit (GPU) 670 to obtain the GPU's rendered image (virtual image) data;
[0246] The display processing unit 640 is connected to the image information processor 610 and obtains the RGB image of the image information processor;
[0247] The display processing unit 640 is connected to the central processing unit 630 to obtain the final fusion weight of the central processing unit 630;
[0248] The display processing unit 640 is connected to the display device 680 and outputs the mixed display image to the display device 680 .
[0249] In this embodiment, the display device 680 may be a head-mounted display or glasses, etc.
[0250] When the implementation process file of the mixed reality virtual-real fusion method is a computer program, it can also be stored in a computer-readable storage medium as a product. For example, a computer-readable storage medium may include, but is not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memories (EPROMs), cards, sticks, key drives). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain, and / or carry code and / or instructions and / or data.
[0251] The present invention provides a mixed reality virtual-real fusion method, device, and medium, which uses color modeling and edge-preserving filtering methods to calculate image virtual-real fusion parameters. It not only takes into account historical and current RGB images to ensure the correct matching of depth images and RGB images, but also can utilize the latest image information to effectively reduce display delay.
[0252] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.
[0253] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the foregoing description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0254] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.
[0255] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0256] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside in a user terminal as discrete components.
[0257] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0258] The above embodiments are provided to persons familiar with the art for implementing or using the present invention. Personnel familiar with the art may make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the scope of protection of the present invention is not limited to the above embodiments, but should be the maximum scope of the innovative features mentioned in the claims.
Claims
1. A mixed reality virtual-real fusion method, characterized in that: The following steps are involved: Obtain real-world RGB images and depth images, as well as virtual images and virtual depths; Compare the depth image and the virtual depth to generate an occlusion relationship graph, and generate a color model based on the historical frame of the RGB image and the occlusion relationship graph; The latest frame of the RGB image is used to perform edge-preserving filtering on the color model to obtain the final fusion weight map; According to the final fusion weight map, the real-world RGB image and the virtual image are weightedly fused to generate a display image.
2. The mixed reality virtual-real fusion method according to claim 1, characterized in that: The comparing the depth image and the virtual depth to generate an occlusion relationship graph, and generating a color model based on the historical frame of the RGB image and the occlusion relationship graph, further comprises: Select the latest frame of the depth image, the virtual depth of the historical frame of the virtual image, and the historical frame of the RGB image; Compare the depth image with the virtual depth of the virtual image, mark the occlusion relationship in the history frame of the RGB image, and obtain a first occlusion relationship graph; A historical frame of the RGB image is selected and divided into blocks, and a color model is generated for different occlusion areas of each block of the image according to the first occlusion relationship diagram.
3. The mixed reality virtual-real fusion method according to claim 1, characterized in that: The method further comprises: performing edge-preserving filtering on the color model using the latest frame of the RGB image to obtain a final fusion weight map. Select the latest frame of the RGB image, estimate the probability of pixel occlusion based on the color model of the historical frame of the RGB image, and obtain the occlusion probability map; Obtain the latest frame of the RGB image, perform edge-preserving filtering on the occlusion probability map, and obtain a first weight map; The first weight map is taken as the final fusion weight map.
4. The mixed reality virtual-real fusion method according to claim 3, characterized in that: The step of obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes: The brightness component of the latest frame of the RGB image is obtained, and the brightness component is used to perform guided filtering on the occlusion probability map to obtain a first weight map.
5. The mixed reality virtual-real fusion method according to claim 3, characterized in that: The step of obtaining the latest frame of the RGB image and performing edge-preserving filtering on the occlusion probability map to obtain the first weight map further includes: The latest frame of the RGB image is obtained, and the occlusion probability map is subjected to joint bilateral filtering to obtain the first weight map.
6. The mixed reality virtual-real fusion method according to claim 2, characterized in that: The following steps are also included: Performing morphological transformation on the first occlusion relationship graph to obtain a second weight graph; The first weight map and the second weight map are weightedly fused to obtain a third weight map as the final fused weight map.
7. The mixed reality virtual-real fusion method according to claim 1, characterized in that: The acquiring of the real-world RGB image and depth image further comprises: continuously acquiring the real-world RGB image, depth image and exposure timestamp at a preset frame rate; The acquiring of the virtual image and the virtual depth further includes: continuously acquiring the virtual image and the virtual depth of the virtual image at a preset frame rate.
8. The mixed reality virtual-real fusion method according to claim 1, characterized in that: After acquiring the real-world RGB image and depth image and acquiring the virtual image and virtual depth, the method further includes: downsampling all acquired images to low-resolution images; The real-world RGB image and the virtual image are weighted according to the final fusion weight map. Before fusing and mixing to generate a display image, the method further includes: upsampling the final fused weight map to a display resolution.
9. The mixed reality virtual-real fusion method according to claim 8, characterized in that: For the acquired RGB image and the virtual image, bilinear interpolation is used for downsampling.
10. The mixed reality virtual-real fusion method according to claim 8, characterized in that: The acquired depth image and virtual depth are downsampled using nearest neighbor interpolation.
11. The mixed reality virtual-real fusion method according to claim 2, characterized in that: The marking of occlusion relationships in the RGB image further comprises: The RGB image is divided into three parts: the virtual image display area, the real image display area and the pending area; marking the displayed virtual image area as a first coefficient; The area showing the true image is marked as the second coefficient; The undetermined area is marked as the third coefficient.
12. The mixed reality virtual-real fusion method according to claim 11, characterized in that: The marking of occlusion relationships in the RGB image further comprises: Mark the area where the depth is missing as a pending area; Areas where virtual depth is missing are not marked.
13. The mixed reality virtual-real fusion method according to claim 6, characterized in that: The performing morphological transformation on the first occlusion relationship graph to obtain the second weight graph further includes: Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph; Edge processing is performed on the second occlusion relationship graph to obtain a second weight graph.
14. The mixed reality virtual-real fusion method according to claim 13, characterized in that: The performing edge processing on the second occlusion relationship graph to obtain a second weight graph further includes: Filtering and ternary processing the second occlusion relationship graph to obtain a third occlusion relationship graph; The third occlusion relationship graph is filtered to obtain a second weight graph.
15. The mixed reality virtual-real fusion method according to claim 2, characterized in that: The color model includes a color histogram; Generating color models for different blocked areas in each image block further includes: performing color histogram statistics on the unblocked areas and blocked areas in each image block.
16. The mixed reality virtual-real fusion method according to claim 15, characterized in that: When performing color histogram statistics, the color depth is downsampled.
17. The mixed reality virtual-real fusion method according to claim 3, characterized in that: The selecting the latest frame of the RGB image, estimating the probability of pixel points being occluded according to the color model of the historical frame of the RGB image, and obtaining an occlusion probability map further includes: Performing morphological dilation on the undetermined region of the first occlusion relationship graph to obtain a second occlusion relationship graph; The latest frame of the RGB image is selected, and according to the position of the undetermined area in the second occlusion relationship diagram, the occlusion probability of each pixel point in the corresponding area of the latest frame of the RGB image is estimated to obtain an occlusion probability map.
18. The mixed reality virtual-real fusion method according to claim 13, characterized in that: The estimating the occlusion probability of each pixel point in the corresponding position area of the latest frame of the RGB image according to the position of the undetermined area in the second occlusion relationship diagram to obtain the occlusion probability map further includes: Select several blocks of images closest to the current pixel in the historical frame of the RGB image; According to the color models of several blocks of images, the color model of the area close to the current pixel is estimated; According to the estimated color model, calculate the distribution of the color of the current pixel in different occlusion areas; Estimate the probability that the current pixel is blocked based on the distribution; Regularization constraints are applied to the probability values to obtain the occlusion probability map.
19. The mixed reality virtual-real fusion method according to claim 4, characterized in that: The brightness component further includes a Y channel in the YUV space, an L channel in the Lab space, an I channel in the HSI space, and an L channel in the HSL space.
20. A mixed reality virtual-real fusion device, characterized in that: Including image information processor, depth information processor, central processing unit, display processing unit: The image information processor is connected to the central processing unit and the display processing unit, and is used to obtain the RGB image of the real scene and send it to the central processing unit and the display processing unit; The depth information processor is connected to the central processing unit and is used to obtain a depth image of the real scene and send it to the central processing unit; The central processing unit is connected to the display processing unit, and is used to obtain virtual depth, execute the mixed reality virtual-real fusion method according to any one of claims 1 to 19, and send the final fusion weight map to the display processing unit; The display processing unit obtains the final fusion weight map sent by the central processing unit, performs weighted fusion on the real-world RGB image and the virtual image, and mixes and generates a display image.
21. The mixed reality virtual-real fusion device according to claim 20, characterized in that: The image information processor is connected to the scene camera, obtains the collected data of the scene camera and generates an RGB image; The depth information processor is connected to the depth sensor, obtains the collected data of the depth sensor and generates a depth image.
22. A mixed reality virtual-real fusion system, characterized in that: Including mixed reality virtual and real fusion device, scene camera, depth sensor, graphics processor and display device: The mixed reality virtual-reality fusion device adopts the mixed reality virtual-reality fusion device as described in any one of claims 20-21; The scene camera is used to collect RGB images of the real scene and send them to the mixed display virtual and real device; The depth sensor is used to collect the depth image of the real scene and send it to the mixed display virtual and real device; The image processor is used to generate a virtual image and a virtual depth and send them to a mixed display virtual-real device; The display device is used to receive and display the display image generated by the mixed display virtual and real device.
23. A computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 19 is performed.
Citation Information
Patent Citations
Method for generating virtual-real fusion image for stereo display
CN104504671A
Shielding removing method and device and computer equipment
CN110136082A
AR portrait photographing method and system based on deep learning
CN113409231A
Object occlusion relation confirmation method based on augmented reality and application
CN115713702A
Occlusion detection
US11170578B1
Cited By
Image display method and device, USB camera and medium
CN119496994A
Naked eye 3D augmented reality interactive display system
CN121900045A
A naked-eye 3D augmented reality interactive display system
CN121900045B
