A method, system, terminal and medium for generating XR combined scene experience screen
By obtaining the user pose and imaging interval in real time, combining binocular stereoscopic vision and semantic segmentation technology, we can identify the display sub-interval to which the real scene pixel belongs, solving the problem of poor occlusion calculation accuracy when combining real scenes and virtual sub-scenes, and achieving more Accurately combine scene experience screen generation.
Patent Information
- Application Number
- CN202311205254.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-09-19
AI Technical Summary
When the existing XR combined scene experience generation method is combined by real scenes and virtual subscenes, due to the poor depth information accuracy, the occlusion calculation errors and cannot adapt to the complex environment of multiple virtual training scenarios and real campus scenarios.
By generating the experience screen of each subscene, and obtaining the user pose and imaging interval in real time, using binocular stereo vision for occlusion calculation, identifying the display sub-section to which the real scene pixel belongs, combining the depth value error range and object category or instance segmentation, Determine the occlusion relationship and ensure the correct synthesis of real scenes and virtual subscenes.
It effectively avoids occlusion errors caused by poor depth information accuracy, improves the object occlusion effect in the combined scene experience screen, and is suitable for immersive experience in large-scale combined scenes.
Smart Images

Figure CN117315203B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of pan-virtual reality technology, and in particular to an XR combined scene experience screen generation method, system, terminal and medium. Background Art
[0002] Technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) are of great value in solving the problems of being invisible, intangible, and inaccessible in vocational education and training. VR, AR, MR (hereinafter referred to as XR, extended reality) and other technologies are interconnected and integrated. Empowered by high-performance wireless network technologies such as 5G networks and wifi6, the storage, computing, rendering and other services required by XR mobile terminals (5G mobile phones, head-mounted displays, etc.) can be placed in the cloud. As a result, based on cloud services such as cloud storage, cloud computing, and cloud rendering, the computing, storage, and rendering capabilities that a single XR terminal can have can be unlimited.
[0003] Based on XR technology, a combined scene composed of more than two sub-scenes can be constructed, where the sub-scenes can be virtual or real scenes. Combined scenes can break down the barriers between scenes and construct a variety of experience activity types. For example, in the classroom, multiple single-person teaching activity scenes or multi-person collaborative teaching activity scenes can be combined to generate a teaching combination scene containing multiple scenes. By changing the types and number of scenes included, a wide variety of combined scenes can be constructed, including: "learning by doing" scenes, "demonstration-practical training" scenes, "cognition-practical training" scenes, practical training assistance scenes, assessment competition scenes, multi-group practical demonstrations, etc.
[0004] In a combined scene composed of a real scene and a virtual sub-scene, the real scene experience screen is generated in real time by a real camera, and the virtual scene experience screen is rendered by a computing system. The complete experience screen of this combined scene is obtained by synthesizing the real scene experience screen and the virtual sub-scene experience screen. When a combined scene contains multiple virtual sub-scenes, a single computing device may not be able to load the rendering of this combined scene. Deploy and run each virtual sub-scene of the combined scene on multiple servers in a server cluster or on multiple graphics cards of a large server. Each virtual sub-scene is rendered separately on different servers or different graphics cards to generate an experience screen. The experience screens of each sub-scene are aggregated together after transmission, and the experience screen of the combined scene is obtained through image synthesis. Based on cloud services, an immersive experience of an XR combined scene containing a large number of sub-scenes can be generated for users.
[0005] When the sub-scenes in the combined scene are rendered separately, the user experience screen of each sub-scene needs to be synthesized through occlusion calculation to form the user's experience screen of the XR combined scene. In the patent application "A method, system and platform for constructing a new dimensional space based on XR technology" (application number CN202210022608.9), a corresponding depth image is generated while rendering and generating the user experience screen of each sub-scene, and the user experience screen of each sub-scene is occluded based on the depth image. In order to eliminate the generation and transmission of depth images, in the patent application "Ubiquitous training campus construction method, system and storage medium based on XR technology" (application number: CN202210908795.0), it is pre-specified that there must be a dividing surface between each sub-scene. According to the position of the dividing surface of the scene, the occlusion relationship between the experience screens of each virtual scene can be calculated without the need for a depth image. However, many XR combination scenes in actual application needs cannot meet the requirement that each sub-scene has a separation plane. For example, a combination scene composed of a real campus scene and multiple virtual training scenes, where the virtual training scene is deployed in the square, building lobby and other scenes of the real campus. Often, the virtual training scene is surrounded by buildings in the real campus scene. There is no separation plane between the virtual training scene and the real campus scene that can separate the two. Therefore, the construction method of the patent application "Ubiquitous Training Campus Construction Method, System and Storage Medium Based on XR Technology" (Application Number: CN202210908795.0) cannot be applied. Two XR combination scene experience generation methods are given in the patent applications No. 2023110288353 and 2023110286220. For a combination scene composed of a convex scene and a non-convex scene, the transmission of the depth information of the experience picture is effectively reduced by representing the sub-interval experience picture by the depth information of the sub-interval in the display interval of the combination scene or calculating the one-way occlusion user position interval.
[0006] Due to power limitations, the effective range of depth information of existing XR mobile terminal depth cameras does not exceed 10 meters. For imaging content beyond 10 meters, although depth information can be obtained through binocular vision measurement, the accuracy is very poor. In a combined scene composed of a real scene and a virtual sub-scene, when the real scene is non-convex and when the display range of the virtual sub-scene exceeds the effective range of the depth information of the terminal depth camera, due to the insufficient accuracy of the depth information of the real scene picture, it will lead to large-scale errors in the occlusion calculation between the real scene and the virtual sub-scene that may exist in the depth information. The two XR combined scene experience generation methods given in the patent applications No. 2023110288353 and 2023110286220 effectively reduce the transmission of the experience picture depth information, but do not completely avoid the occlusion relationship between the picture pixels obtained by comparing the depth values. Therefore, the XR combined scene experience generation method still needs to be improved. Summary of the invention
[0007] The main purpose of the present invention is to provide a method, system, terminal and medium for generating an XR combined scene experience screen, especially for a large combined scene composed of a real scene and a virtual sub-scene, to avoid obtaining an incorrect object occlusion effect in the combined scene experience screen through direct comparison of depth information with poor precision.
[0008] To achieve the above object, the present invention proposes a method for generating an XR combined scene experience screen, the method comprising the following steps:
[0009] Step S10, generating the experience screen of each sub-scene of the combined scene, wherein all arbitrary virtual sub-scenes s i Obtain the position of user p in its coordinate system in real time, and use this real-time position and s i The imaging interval of user p in s i The pupil distance in the scene space generates the user p to s i The experience screen;
[0010] Step S20: The binocular stereoscopic vision experience images of each sub-scene are occluded and calculated to be synthesized into a combined scene experience image. For any two pixels τ of the experience image that share the user's sight line but belong to different sub-scenes, i With τ j , if τ i With τ j All of them image the non-sky area of the scene content and calculate τ i With τ j The occlusion relationship is as follows: i The display interval of the combined scene is related to τ j The occlusion relationship of the display interval of the combined scene is assigned as τ i With τ j Occlusion relationship;
[0011] Wherein, a single sub-scene may have multiple display sub-intervals in the combined scene, and any sub-scene of the combined scene is a virtual scene or a real scene. For a combined scene sub-scene s having multiple display sub-intervals in the combined scene, i When calculating the occlusion relationship between any pixel τ and pixels of other combined scene sub-scenes, it is also necessary to determine the display sub-interval to which τ belongs, and obtain the occlusion relationship between pixels from the occlusion relationship between the display sub-interval to which τ belongs and the display interval or display sub-interval of other combined scene sub-scenes.
[0012] As a further improvement of the present invention, the combined scene includes a real scene, and there is a step S00 before the step S10: setting the combined scene experience screen generation parameters, including setting the imaging range of each virtual sub-scene, the pupil distance of the user p in each virtual sub-scene, the rotation and translation transformation relationship between the combined scene coordinate system and the virtual sub-scene coordinate system, the display range of each virtual sub-scene in the combined scene, and setting multiple display sub-ranges of the real scene in the combined scene.
[0013] Wherein, the step S20 includes:
[0014] S201, identifying the display sub-interval to which each pixel in the real scene stereoscopic visual experience picture belongs in the combined scene;
[0015] S202, obtaining the occlusion relationship between the display sub-interval of the real scene in the combined scene and the display interval of each virtual sub-scene under the current user posture;
[0016] S203: synthesize the real scene experience picture and the virtual sub-scene experience picture according to the above occlusion relationship.
[0017] As a further improvement of the present invention, step S201 is specifically implemented as follows: for any pixel τ belonging to the real scene experience screen, calculate the set of display sub-display intervals of the real scene where the user's line of sight passes through in the combined scene, and when the depth value error range of τ is consistent with any display sub-display interval in the set, When the depth value ranges intersect on the line of sight, τ belongs to the display subinterval
[0018] As another improved method of the present invention, the step S00 also includes marking the object categories contained in each display sub-interval of the real scene in the combined scene, and the step S201 is specifically implemented as follows: semantic segmentation is performed on the real scene experience screen image to obtain the category to which the imaging content of each pixel belongs, and for any pixel τ belonging to the real scene experience screen, the set of display sub-intervals of the real scene where the user's line of sight passes through where τ is located is calculated in the combined scene, and when the category to which the imaging content of τ belongs belongs to any display sub-interval in the set When the object contained is a body, then τ belongs to the display subinterval
[0019] As another improved method of the present invention, the step S00 also includes marking the object instances contained in each display sub-interval of the real scene in the combined scene, and the step S201 is specifically implemented as follows: performing instance segmentation on the real scene experience screen image to obtain the instance to which the imaging content of each pixel belongs, for any pixel τ belonging to the real scene experience screen, calculating the set of display sub-intervals of the real scene where the user's line of sight passes through in the combined scene where τ is located, when the instance to which the imaging content of τ belongs belongs to any display sub-interval in the set When the object instance is included, τ belongs to the display subinterval
[0020] As a further improvement of the present invention, step S00 also includes ensuring that each virtual sub-scene set in the display interval of the combined scene is unidirectionally occluded with each display sub-interval of the real scene in the combined scene, and in step S20, the occlusion relationship between the real scene personnel and the virtual sub-scene content is processed by removing the real scene personnel from the real scene experience screen and repairing the image or obtaining the occlusion relationship based on the depth information comparison, and step S203 includes:
[0021] S2031, synthesizing all virtual sub-scene experience pictures to obtain a virtual scene synthesized experience picture;
[0022] S2032, the real scene experience picture and the virtual scene synthetic experience picture are synthesized to obtain a combined scene experience picture.
[0023] To achieve the above objectives, the present invention also proposes an XR combined scene experience screen generation system, the system comprising a memory, a processor, and an XR combined scene experience screen generation method program stored in the memory, and the XR combined scene experience screen generation method program executes the steps of the method described above when run by the processor.
[0024] To achieve the above objectives, the present invention also proposes an XR combined scene experience screen generation terminal, which executes step S203 or step S2031 or step S2032 of the XR combined scene experience screen generation method as described above.
[0025] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the steps of the XR combined scene experience screen generation method as described above are executed. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a flow chart of the method for generating an XR combined scene experience screen of the present invention;
[0027] Figure 2 This is an example diagram of binocular stereo vision of the present invention;
[0028] Figure 3 This is an example of binocular stereoscopic vision depth information error of the present invention;
[0029] Figure 4 This is an example of a real scenario of the present invention;
[0030] Figure 5 This is an example of a combined scene of virtual-real fusion in the present invention;
[0031] Figure 6 A schematic diagram of dividing a real scene of the present invention into multiple display sub-intervals;
[0032] Figure 7 is a system configuration diagram of a first embodiment of the present invention;
[0033] Figure 8 A schematic diagram of a sub-interval display of a real scene traversed by a line of sight passing through a pixel of the present invention;
[0034] Fig. 9 It is a system configuration diagram of the eighth embodiment of the present invention. DETAILED DESCRIPTION
[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0036] Reference Figure 1 As shown, the present invention proposes a method for generating an XR combined scene experience screen, and the method comprises the following steps:
[0037] Step S10, generating an experience screen of each sub-scene of the combined scene, wherein all arbitrary virtual sub-scenes s i Obtain the position of user p in its coordinate system in real time, and use this real-time position and s i The imaging interval of user p in s i The pupil distance in the scene space generates the user p to s i The experience screen;
[0038] Step S20: The binocular stereoscopic vision experience images of each sub-scene are occluded and calculated to be synthesized into a combined scene experience image. For any two pixels τ of the experience image that share the user's sight line but belong to different sub-scenes, i With τ j , if τ i With τ j All of them image the non-sky area of the scene content and calculate τ i With τ j The occlusion relationship is as follows: i The display interval of the combined scene is related to τ jThe occlusion relationship of the display interval of the combined scene is assigned as τ i With τ j Occlusion relationship;
[0039] Wherein, a single sub-scene may have multiple display sub-intervals in the combined scene, and any sub-scene of the combined scene is a virtual scene or a real scene. For a combined scene sub-scene s having multiple display sub-intervals in the combined scene, i When calculating the occlusion relationship between any pixel τ and pixels of other combined scene sub-scenes, it is also necessary to determine the display sub-interval to which τ belongs, and obtain the occlusion relationship between pixels from the occlusion relationship between the display sub-interval to which τ belongs and the display interval or display sub-interval of other combined scene sub-scenes.
[0040] Furthermore, the combined scene includes a real scene, and there is a step S00 before the step S10: setting the combined scene experience screen generation parameters, including setting the imaging range of each virtual sub-scene, the pupil distance of the user p in each virtual sub-scene, the rotation and translation transformation relationship between the combined scene coordinate system and the virtual sub-scene coordinate system, the display range of each virtual sub-scene in the combined scene, and setting multiple display sub-ranges of the real scene in the combined scene.
[0041] Wherein, the step S20 includes:
[0042] S201, identifying the display sub-interval to which each pixel in the real scene stereoscopic visual experience picture belongs in the combined scene;
[0043] S202, obtaining the occlusion relationship between the display sub-interval of the real scene in the combined scene and the display interval of each virtual sub-scene under the current user posture;
[0044] S203: synthesize the real scene experience picture and the virtual sub-scene experience picture according to the above occlusion relationship.
[0045] Further, the step S201 is specifically implemented as follows: for any pixel τ belonging to the real scene experience screen, the set of display sub-display intervals of the real scene where the user's line of sight passes through in the combined scene is calculated, and when the depth value error range of τ is consistent with any display sub-display interval in the set, When the depth value ranges intersect on the line of sight, τ belongs to the display subinterval
[0046] Alternatively, the step S00 also includes marking the object categories contained in each display sub-interval of the real scene in the combined scene, and the step S201 is specifically implemented as follows: semantic segmentation is performed on the real scene experience screen image to obtain the category to which the imaging content of each pixel belongs, and for any pixel τ belonging to the real scene experience screen, the set of display sub-intervals of the real scene where the user's line of sight passes through in the combined scene is calculated, and when the category to which the imaging content of τ belongs belongs to any display sub-interval in the set When the object contained is a body, then τ belongs to the display subinterval
[0047] Alternatively, the step S00 also includes marking the object instances contained in each display sub-interval of the real scene in the combined scene, and the step S201 is specifically implemented as follows: performing instance segmentation on the real scene experience screen image to obtain the instance to which the imaging content of each pixel belongs, for any pixel τ belonging to the real scene experience screen, calculating the set of display sub-intervals of the real scene where the user's line of sight passes in the combined scene, when the instance to which the imaging content of τ belongs belongs to any display sub-interval in the set When the object instance is included, τ belongs to the display subinterval
[0048] Further, step S00 also includes ensuring that each virtual sub-scene set in the display interval of the combined scene is unidirectionally occluded with each display sub-interval of the real scene in the combined scene, and in step S20, the occlusion relationship between the real scene personnel and the virtual sub-scene content is processed by removing the real scene personnel from the real scene experience screen and repairing the image or obtaining the occlusion relationship based on the depth information comparison, and step S203 includes:
[0049] S2031, synthesizing all virtual sub-scene experience pictures to obtain a virtual scene synthesized experience picture;
[0050] S2032, the real scene experience picture and the virtual scene synthetic experience picture are synthesized to obtain a combined scene experience picture.
[0051] Important technical terms involved in the present invention are explained below.
[0052] Convex interval
[0053] make is a three-dimensional convex interval, then it has the following properties: Let There are two points a and b, where the coordinates of point a are The coordinates of point b are Point c is on the line connecting points a and b and between points a and b. The coordinates of point c can be expressed as Among them, 1>λ>0, then point c must also belong to
[0054] Scenarios and Scenario Examples
[0055] A scene defines the objects, object states, object operation logic, and interaction logic between objects contained in a three-dimensional space; a scene instance is a program process or thread that the system calls on computer processors, memory, graphics cards and other computing resources to run in real time according to the scene definition. This program process or thread calculates the state of each object in the scene in real time, renders the screen, and responds to user interaction. When a single scene has multiple user experiences at the same time, if the computing resources available to a single scene instance cannot generate experience screens for all users in real time, it is necessary to generate multiple scene instances for this scene and allocate them to each user. These scene instances establish communication connections to synchronize the state of objects in the scene, and each scene instance generates an experience screen for the corresponding user in real time, so that each user can experience this scene together.
[0056] User Field of View and Coordinate System
[0057] The user's field of view is the spatial range visible to the user. The user's field of view changes with the movement of the user's head and eyeballs. The present invention defines the user's field of view coordinate system with reference to the user's head, so that the movement of the user's head and eyeballs does not affect the position relationship of the user's field of view coordinate system relative to the user's head. In an embodiment of the present invention, the user's field of view coordinate system is set as follows: on the line connecting the binocular center of the human eye, the midpoint between the binocular center of the binocular eye is used as the origin of the coordinate system, the direction of the binocular line connecting the human eye from the left eye to the right eye is defined as the positive direction of the x-axis, and the direction of the face is defined as the positive direction of the z-axis, and the positive direction of the y-axis is defined as perpendicular to the x-axis and the z-axis and satisfies the left-hand coordinate system.
[0058] XR scene imaging range
[0059] User p in the combined scene For an immersive interactive experience, i To form a combined scene One of the sub-scenes of When experiencing i Maybe only part of the scene content needs to be in the combined scene In order to limit s i Need to be in the combined scene The scene content presented to user p is set in a three-dimensional interval, scene s i Only the content in this three-dimensional interval needs to be imaged when generating the experience screen of user p, and this set three-dimensional interval is the XR scene imaging interval. Among them, the imaging interval can be infinite, or it can be any three-dimensional shape interval such as a cube, a sphere, etc.
[0060] Display range of XR scenes in combined scenes
[0061] Combination Scene Contains multiple sub-scenes. In order to make the sub-scenes No overlap or reduced overlap, for any sub-scene s i , when setting the combined scene A three-dimensional interval Ω is set inside to present s i , limit s i exist When setting the combined scene Coordinate system and subscenes i When the rotation, translation and scaling relationship between the coordinate systems is used, the three-dimensional interval Ω is mapped to the sub-scene s according to this rotation, translation and scaling relationship. i The obtained three-dimensional interval Can be used as scenes i The imaging interval, in turn, according to this rotation, translation and scaling relationship, the scene s i Imaging range Mapped to the field of view of viewer p, the obtained three-dimensional interval can be used as scene s i In the combined scene Display interval.
[0062] One-way occlusion and two-way occlusion
[0063] Scenes i and j For combined scenes Two sub-scenes of k , if in all sight lines of user p, if s i With s j Occlusion occurs, all s i The scene content occludes s j scene content, then at time t k , for user p, scenario s i and j One-way occlusion, and the occlusion relationship is s i One-way occlusion j ; Conversely, if in all sight lines of user p, if s i With s j Occlusion occurs, all s j The scene content occludes s i scene content, then at time t k , for user p, scenario s i and j One-way occlusion, and the occlusion relationship is s j One-way occlusion i ; Relatively speaking, if in all sight lines of user p, if s i With s j Occlusion has occurred, possibly si The scene content occludes s j Scene content, or possibly s j The scene content occludes s i scene content, then at time t k , for user p, scenario s i and j Bidirectional occlusion. If at all times, no matter the user p is in the combined scene Whatever position you are in, there are scenes i and j Non-bidirectional occlusion is called: in the combined scene Inside, scenes i and j It is one-way occlusion. Two non-intersecting convex intervals must be one-way occluded. For details, please refer to the patent application "Ubiquitous Training Campus Construction Method, System and Storage Medium Based on XR Technology" (Application No.: CN202210908795.0) Specification 115-116.
[0064] The principle of the present invention is described below.
[0065] Binocular vision depth information accuracy analysis
[0066] by Figure 2 For example, the line from object point A to the optical center of the left eye camera is perpendicular to the imaging plane of the left eye camera. The imaging position of the left eye camera is the origin of the coordinate system. The position of object point A on the imaging plane of the right eye camera is point u, and point u is in the photosensitivity interval of the image pixel τ of the right eye camera.
[0067] like Figure 3 As shown, since the image value is discrete, the image coordinate value of the image pixel τ corresponds to the point u′ on the imaging surface, and the line connecting the right eye optical center and u′ is and The intersection point is A′. Therefore, according to the binocular images generated by the left and right eye cameras, the depth value of A is calculated to be |O′A′|, while the actual depth value of A is |O′A|, and there is a deviation between the two. For example, the left and right eye images are both 4k resolution, the number of horizontal pixels is 4096 pixels, and the field of view is 120 degrees. On the XR headset, the distance between the binocular cameras |OO′| is close to the pupil distance of the human eye. The calculation is based on |OO′| as the average pupil distance of the human eye of 63mm, |O′A| is 50m, and the imaging error range is half a pixel deviation. The depth value range is 37m to 75m, which shows that the error range is extremely large. For details, please refer to Section 5.2 Accuracy Analysis of Binocular Stereo Vision in Machine Vision edited by Zhang Guangjun (ISBN 7-03-014717-0, published in 2005).
[0068] Figure 4Given a real scene consisting of buildings, trees, and a square, user P enters this real scene, where the length and width of the square are both 100 meters. Figure 5 In the figure, virtual scenes A, B, C, and D are deployed in the square of the real scene. In order to avoid the overlap of virtual objects between virtual scenes, a three-dimensional display range is limited to each virtual scene. In this way, a combined scene including virtual scenes and real scenes is constructed. When user P uses an XR headset to immerse himself in this combined scene, the virtual scene can easily generate high-precision depth information when generating the experience screen. However, in the real scene experience screen, the depth camera in the XR headset can obtain high-precision close-range depth information (for example, within 10 meters), but the accuracy of long-distance depth information is very poor. If the occlusion relationship between scene contents is judged directly by comparing the depth information, Figure 5 In the user experience of the combined scenes, it is possible that the virtual scene D blocks the real trees or the real buildings block the virtual scenes A and B, which seriously deviates from the real occlusion relationship.
[0069] like Figure 6 As shown, the method provided by the present invention divides the real scene into multiple three-dimensional intervals. In the user's line of sight at the position of user P, each three-dimensional interval of the real scene and all virtual sub-scenes in the combined scene are unidirectionally occluded. The XR head display collects the experience picture of user P of the real scene, and correctly determines the sub-interval to which each pixel belongs through a certain method. The correct occlusion relationship between each pixel in the real scene experience picture and the corresponding pixel of each virtual sub-scene experience picture in the combined scene is obtained from the unidirectional occlusion relationship between the sub-interval and each virtual scene in the combined scene in the display interval of the combined scene. The combined scene experience picture is thus synthesized. Among them, when judging the sub-interval to which each pixel of the real scene belongs, the real scene sub-interval to which this pixel belongs is first judged based on the pixel depth information and its error range. When the pixel depth information and its error range cannot correctly judge the real scene sub-interval to which this pixel belongs, the sub-interval to which this pixel belongs can be further judged based on visual information such as the imaging object category.
[0070] In the embodiments of the present invention, it is assumed that the imaging field of view and resolution of all images are the same, so that only pixels of the same image coordinates between different images will have an occlusion relationship; if the imaging field of view and resolution between different images are different, the imaging field of view and resolution between the images can be converted to the same by interpolation or sampling, which is a common method used by technicians in the field.
[0071] First embodiment:
[0072] Combination scenarios of this embodiment It is a virtual-real hybrid scene. The combined scene consists of a real scene s0 and N virtual sub-scenes, where N is a natural number greater than 1. These virtual sub-scenes are s1, s2, ..., s N .like Figure 7 As shown, in this embodiment, the combined scene experience generation system includes: a user interface 10, an image synthesis module 20, a real scene imaging module 30, a combined scene parameter setting module 40, an interval identification module 50, an instance of the first virtual sub-scene (60-1), an instance of the second virtual sub-scene (60-2), ..., an instance of the Nth virtual sub-scene (60-N). The instance of the first virtual sub-scene (60-1) is an instance of the virtual sub-scene s1, the instance of the second virtual sub-scene (60-2) is an instance of the virtual sub-scene s2, and the instance of the Nth sub-scene (60-N) is a virtual sub-scene s N For video perspective MR headsets, the real scene imaging module 30 generates a binocular stereoscopic visual experience screen of the user and a corresponding depth image of the real scene through optoelectronic devices such as a camera module on the user's MR headset terminal. For optical perspective MR headsets, the real scene imaging module 30 does not need to generate a binocular stereoscopic visual experience screen but needs to generate a binocular stereoscopic visual depth image through optoelectronic devices such as a camera on the user's MR headset terminal (it can also be considered that a binocular stereoscopic visual screen is generated, but the color value of each pixel in the binocular stereoscopic visual screen image is 0, that is, a completely black image). The user sends a command to the combined scene parameter setting module 40 through the user interface 10 to set parameters such as the imaging interval of each sub-scene, the display interval of each sub-scene in the combined scene, and the display sub-interval of the real scene. The system can also send a command to the combined scene parameter setting module 40 to set the combined scene parameters. The set parameters will be sent to the instances of each virtual sub-scene and the interval identification module 50. Each virtual sub-scene instance receives user posture and interactive operation information from the user interface 10 in real time, responds to the interactive operation, and each virtual sub-scene generates a user binocular stereoscopic visual experience screen according to the imaging interval, the user's pupil distance in its coordinate system and the real-time posture. Each virtual sub-scene sends the binocular stereoscopic visual user experience screen to the image synthesis module 20. The image synthesis module 20 synthesizes the real scene stereoscopic visual imaging screen read from the real scene imaging module 30 in real time with the user experience screen sent by each virtual sub-scene. When synthesizing the image, it is necessary to obtain the display sub-interval to which each pixel of the real scene experience screen belongs, and further perform image synthesis according to the occlusion relationship between the display intervals of each virtual sub-scene or between the display interval of the virtual sub-scene and the display sub-interval of the real scene. The image synthesis module 20 sends the synthesized image to the user interface 10 for display to the user.
[0073] In the embodiment of the present invention, in order to simplify the real scene s0 and the combined scene Common coordinate system. Figure 1 As shown, the combined scene experience generation method of this embodiment includes the following steps:
[0074] Step S00: Setting the generation parameters of the combined scene experience screen, including setting the imaging interval of each virtual sub-scene, the pupil distance of the user p in each virtual sub-scene, the rotation and translation transformation relationship between the combined scene coordinate system and the virtual sub-scene coordinate system, the display interval of each virtual sub-scene in the combined scene, and setting the display sub-interval of the real scene in the combined scene.
[0075] Step S10, generating the experience screen of each sub-scene of the combined scene, wherein all arbitrary virtual sub-scenes s i Obtain the position of user p in its coordinate system in real time, and use this real-time position and s i The imaging interval of user p in s i The pupil distance in the scene space generates the user p to s i The experience screen;
[0076] Step S20: The binocular stereoscopic vision experience images of each sub-scene are occluded and calculated to be synthesized into a combined scene experience image. For any two pixels τ of the experience image that share the user's sight line but belong to different sub-scenes, i With τ j , if τ i With τ j Both image the scene content and calculate τ i With τ j The occlusion relationship is as follows: i The display interval of the combined scene is related to τ j The occlusion relationship of the display interval of the combined scene is assigned as τ i With τ j Occlusion relationship;
[0077] Wherein, a single sub-scene may have multiple display intervals in the combined scene, and any sub-scene of the combined scene is a virtual scene or a real scene. For a sub-scene s having multiple display intervals in the combined scene, i When calculating the occlusion relationship between any pixel τ and other sub-scene pixels, it is also necessary to determine the display interval to which τ belongs.
[0078] The specific implementation of step S00 is as follows:
[0079] In the combined scene parameter setting module 40, various combined scene parameters are set: each virtual sub-scene s1, s2, ..., s N The imaging intervals are Ω1, Ω2, ..., Ω N; Set the three-dimensional interval Ω′0 in the combined scene where the virtual scene cannot be deployed. For example, the interval occupied by the wall, equipment and other items in the real scene s0 cannot be deployed with the virtual scene. This interval is used to represent the display interval of the real scene s0 when judging the occlusion relationship between the virtual sub-scene and the real scene s0, but it does not actually limit the imaging of the real scene s0; Set the user p in each virtual sub-scene s1, s2, ..., s N The pupil distances are d1, d2, ..., d N ; The user's pupil distance in the combined scene is Set each virtual sub-scene s1, s2, ..., s N In the combined scene The display intervals are Ω′1, Ω′2, ..., Ω′ N ; Set the rotation, translation and scaling relationship of each virtual sub-scene coordinate system to the combined scene coordinate system. For any virtual sub-scene s i (1≤i≤N), scene s i The rotation, translation and scaling relationship between the coordinate system and the combined scene coordinate system is: i , β i , α i , translate along the X, Y, and Z axes respectively The scaling factor of the three axes Z, X, and Y is λ i ;
[0080] The above parameters are not completely independent and need to meet the following constraints: the pupil distance used by the user p's terminal to generate a real scene stereoscopic visual experience screen or depth image is d0 (d0 needs to be as close to the real pupil distance of user p as possible), and the user Pupillary distance The pupil distance for any virtual sub-scene s i ,have Any virtual sub-scenes i , its imaging interval Ω i For any point b0 in the i Coordinate system to combined scene After the coordinate system is transformed by rotation, translation and scaling, the point Belong to Ω′ i ; Display interval Ω′1, Ω′2, ..., Ω′ N They do not overlap with each other, Ω′1, Ω′2, ..., Ω′ N None of them overlap with Ω′0.
[0081] Furthermore, for Ω′0, several display subintervals are set Completely cover Ω′0, and all display sub-intervals do not intersect with the display intervals of other sub-scenes of the combined scene.
[0082] Furthermore, it is ensured that each set display interval of each virtual sub-scene has a separation plane with each display sub-interval of the real scene s0, that is, constant one-way occlusion.
[0083] Furthermore, it is ensured that there are separation planes between the set display intervals of each virtual sub-scene, and that they are all constant one-way occlusions.
[0084] The specific implementation of step S10 is as follows:
[0085] Any time t j , user p in the combined scenario The pose in the coordinate system is [W j Q j ], where W j is the position, Q j is the attitude angle, for each virtual sub-scene s1, s2, ..., s N , receiving the real-time pose of user p from the user interface [W j Q j ], and according to the rotation, translation and scaling relationship between each coordinate system and the combined scene coordinate system, the user's position in each virtual scene coordinate system is calculated. For s1, s2, ..., s N Any virtual scene in i , according to s i Coordinate system and combined scene The rotation, translation and scaling relationship of the coordinate system is calculated to obtain the user p in s i The pose in the coordinate system is Calculating the position and posture of the user p in the virtual sub-scene coordinate system by the rotation, translation and scaling relationship of the coordinate system is well known to those skilled in the art and will not be described in detail here. N Any virtual scene in i , according to the user in s i The pupil distance d i , pose The imaging interval Ω i The scene content in the image is imaged to generate an immersive binocular stereoscopic visual experience screen, where s i The left and right eye images of the stereoscopic vision picture are respectively and It indicates that, and It is necessary to identify which pixels correspond to empty scene contents, for example, the values of pixels with empty scene contents in the experience screen image are set to a specific value τ null(e.g. blue), or it can be identified by a specific array, each element in this array corresponds to a pixel in the image, and the element can only take the value of 0 or 1 to identify whether the corresponding pixel is imaging the scene content is empty. In the embodiment of the present invention, the experience screen image is selected and The value of the pixel with empty scene content is set to a specific value τ null .
[0086] The left and right eye images of the stereoscopic visual experience of the real scene s0 are and Corresponding depth image and Let the depth image generated by the depth camera be and And the maximum effective depth value of the depth camera is T d ,make Construct 2 blocks of size n H ×n W The matrix G L With G R , where n H Experience the image in real scene stereoscopic vision and The number of pixel rows, n W Experience the image in real scene stereoscopic vision and The number of pixel columns. Any pixel in If the depth value of the imaged scene content does not exceed T d When G L (uv)=0, otherwise let G L (uv)=1. Similarly, in G R Record Whether the scene content of the corresponding pixel image exceeds T d .like and The depth value of the scene content imaged by pixels exceeds T d , then the depth value of this part of pixels needs to be calculated by binocular stereo vision, and the depth value of this part of pixels is assigned to and Calculating pixel depth values through binocular stereo vision is a common technical means used by technicians in this field. For example, Fibio Tosi et al. proposed a high-performance method for calculating stereo vision depth information in the paper “NeRF-Supervised deep stereo” [1], which will not be elaborated here.
[0087] The specific implementation of step S20 is as follows:
[0088] The specific method of the image synthesis module 20 to synthesize an image is as follows: at any time t j , the left and right eye images of the stereoscopic visual experience picture received from the real scene imaging module 30 are and Corresponding depth image and For any scene s in the virtual sub-scene k , the stereoscopic vision images received by the left and right eyes are and
[0089] The specific implementation of this step also includes the following steps:
[0090] S201, identifying the real scene display sub-interval to which each pixel in the real scene stereoscopic visual experience picture belongs in the combined scene;
[0091] S202, obtaining the occlusion relationship between all display sub-intervals of the real scene in the combined scene and the display intervals of each virtual sub-scene under the current user posture;
[0092] S203: synthesize the real scene experience picture and the virtual sub-scene experience picture according to the above occlusion relationship.
[0093] Step S201 is specifically implemented as follows:
[0094] In the current pose of user p [W j Q j ], for the real scene s0 experience screen left and right eye depth image and Any pixel According to the transformation relationship between the image coordinates and the display surface coordinate system, the pixel The corresponding image point on the display surface The two-dimensional coordinates of the image point u in the user's field of view can be obtained by the transformation relationship between the two-dimensional coordinate system of the display surface and the three-dimensional coordinate system of the user's field of view. The transformation relationship of the coordinate system can get the image point In the combined scene The three-dimensional coordinate value in the coordinate system. Take the optical center of the human eye as the starting point (if is the pixel of the left eye image, then the optical center here is the left eye optical center), and the image point Connect and extend the line to get ray L, calculate the real scene through which ray L passes in the combined scene The display sub-interval of the ray L is denoted by the set Indicates that M0 is the number of sub-intervals of the real scene that L passes through.
[0095] Construct 2 blocks of size n H ×n W The matrix E 0,0 、E 0,1 , where n H Experience the image in real scene stereoscopic vision and The number of pixel rows, n W Experience the image in real scene stereoscopic vision and The number of pixel columns is initially set to E 0,0 、E 0,1 All elements in are -1.
[0096] When M0 is not 0, the pixel For example, the left eye image has a pixel value of Pixel In the depth image The medium depth value is If G L (uv) takes a value of 0. According to the depth value accuracy of the depth camera, the pixel The range of the true depth value of the corresponding object point is the interval If G L (uv) is taken as 1, and the pixel is calculated according to equations (5.11) and (5.16) in Section 5.2 Accuracy Analysis of Binocular Stereo Vision in Machine Vision edited by Zhang Guangjun (ISBN 7-03-014717-0, published in 2005) Depth value precision, get pixel The range of the actual depth value of the corresponding object point is the interval The depth value intervals of the intersection points of ray L and all displayed subintervals in Λ are calculated respectively, and the depth value intervals of the intersection points of ray L and all displayed subintervals in Λ are calculated to be This calculation method is well known to those skilled in the art and will not be described in detail again. The depth value interval of the only displayed subinterval in Λ If there is an intersection, then set E 0,0 (uv) = k0. If M0 ≥ 2, traverse the depth value intervals of all intersections of the rays L in Λ and the displayed subintervals in Λ in turn. If there is and If there is an intersection, then set E 0,0 (uv) = k iSimilarly, according to the pixel depth value range, the pixel depth in the right eye image of the real scene stereoscopic visual experience screen is identified. The display sub-interval to which it belongs is recorded in E 0,1 middle.
[0097] Step S202 is specifically implemented as follows:
[0098] For real scenes, any subinterval Ω′ is displayed in the combined scene 0,r , get an Ω′ 0,r With any virtual sub-scene display interval Ω′ i The separation plane of the user is calculated, and the relationship between the user's current position and this separation plane is calculated. From this relationship, Ω′ can be obtained under the current user posture. 0,r and Ω′ i According to this method, the occlusion relationship between the real scene in all display sub-intervals of the combined scene and the display intervals of each virtual sub-scene can be obtained. The above calculation process is described in detail in paragraph 0157 of the patent application "Ubiquitous Training Campus Construction Method, System and Storage Medium Based on XR Technology" (Application No.: CN202210908795.0), which will not be repeated here.
[0099] Construct a matrix E of size (m0+1)×(N+1), where m0+1 is the number of sub-intervals displayed in the combined scene, and N is the combined scene. The number of virtual sub-scenes contained in E. All elements in E are initially set to 0, and the occlusion relationship between all display sub-intervals of the real scene in the combined scene and the display intervals of each virtual sub-scene is traversed. If it is calculated that under the current user posture, the real scene is in any display sub-interval of the combined scene Ω′ 0,r With any virtual sub-scene display interval Ω′ i The occlusion relationship is Ω′ 0,r Occlusion Ω′ i , then set E(r,i)=1.
[0100] Construct a matrix E′ of size (N+1)×(N+1), where N is the combined scene The number of virtual sub-scenes contained in E′, all elements in E′ are initialized to 0. According to the patent application "Ubiquitous Training Campus Construction Method, System and Storage Medium Based on XR Technology" (Application No.: CN202210908795.0) paragraph 0157, all virtual sub-scenes in the combined scene are obtained. The occlusion relationship between the intervals is displayed. Traverse to obtain all virtual sub-scenes in the combined scene The occlusion relationship between the intervals is shown in i With s j In the display interval Ω′ of the combined scenei and Ω′ j , if Ω′ is i One-way occlusion Ω′ j , then set E′(i,j)=1, otherwise set E′(j,i)=1.
[0101] Step S203 is specifically implemented as follows:
[0102] The left and right eye images of the synthesized combined scene binocular stereoscopic vision experience picture are Define two two-dimensional auxiliary matrices Q L , Q R .Q L The number of rows and columns and pictures The number of rows and columns is the same, Record The value of each pixel in Q is the pixel value of which virtual sub-scene or which display sub-interval of the real scene. R The number of rows and columns and pictures The number of rows and columns is the same, Record The value of each pixel in is the pixel value of which virtual sub-scene or which display sub-interval of the real scene.
[0103] The implementation of step S203 further specifically includes the following steps:
[0104] S2031, synthesizing all virtual sub-scene experience pictures to obtain a virtual scene synthesized experience picture;
[0105] S2032, the real scene experience picture and the virtual scene synthetic experience picture are synthesized to obtain a combined scene experience picture.
[0106] Step S2031 is specifically implemented as follows:
[0107] Let Q L All elements in are initially 0, and the left eye image of the virtual scene synthesis experience screen is obtained by synthesizing all virtual sub-scene experience screens. Initial Setup if That is There is a non-sky area for imaging objects in the scene, then in Experience the left eye image of the virtual sub-scene s1, traverse all other virtual sub-scenes in the combined scene, and for any virtual sub-scene s i , the left eye image of the experience screen is Traverse all its pixels, if when If is 0, when Need to retrieve E′ to determine the scene In the combined scene The inner three-dimensional display interval and s i The occlusion relationship between the three-dimensional display intervals. Query element if The value is 1, no processing, if If is 0, After all the left eye images of the virtual sub-scenes are traversed and all the pixels of the left eye images are traversed, the virtual scene left eye synthetic experience screen image is synthesized according to the occlusion relationship. Similarly, the virtual scene right eye synthetic experience screen image is obtained according to the above method.
[0108] Step S2032 is specifically implemented as follows:
[0109] Traversing the stereoscopic visual experience of the real scene s0 left eye image Each pixel, for any pixel If Q L The value of (uv) is not 0, and E 0,0 (uv) is not -1, and E(E 0,0 (uv)Q L (uv)) has a value of 0, that is, Combining scenes for real scenes The scene imaging in the display sub-interval corresponds to the display sub-interval Virtual scene If the display interval is blocked, no processing will be done; otherwise After completing the traversal of all pixels, we get The left eye image of the combined scene experience picture after synthesis. Similarly, the right eye image of the combined scene experience picture can be synthesized.
[0110] Second embodiment:
[0111] In the first example, the pixel depth value interval cannot ensure correct identification of the display subinterval of the real scene in the combined scene, especially when the pixel depth value interval intersects with the display subintervals of multiple real scenes in the combined scene. This embodiment improves the first embodiment. The improvements are as follows.
[0112] In step S00, it also includes marking each display sub-interval of the real scene in the combined scene. Object category. Let any display subinterval Ω′ 0,i The set of object categories is Γ i express.
[0113] The specific implementation of step S201 also includes the following:
[0114] For images and Semantic segmentation is performed to obtain the category to which each pixel imaging content belongs. Image semantic segmentation is a well-known technology for those skilled in the art. For example, Hu, Yubin et al. have proposed an image semantic segmentation method based on a neural network with excellent results [2], which will not be described in detail here. Any pixel in The imaging content belongs to the category C L (uv) indicates that Any pixel in The imaging content belongs to the category C R (uv) indicates.
[0115] by Any pixel in For example, in the current position of user p [W j Q j ], starting from the human eye (if is the pixel of the left eye image, then the optical center here is the left eye optical center), and Connect the image points on the display surface and extend the line to get the ray L, and calculate the real scene through which the ray L passes in the combined scene The display subinterval of . Let the real scene subinterval through which ray L passes be represented by the set Indicated by. Where M0 is the number of real scene subintervals that L passes through. Traverse all display subintervals in Λ, if there is a display subinterval in Λ satisfy Then set E 0,0 (uv) = i. For any display subinterval The object category collection contained. This method traverses All pixels in the left eye image can get the real scene stereoscopic visual experience picture. Similarly, it is possible to identify the pixel in the right eye image of the real scene stereoscopic visual experience screen in the combined scene The display sub-interval to which it belongs is recorded in E 0,1 middle.
[0116] Through the above improvements, the beneficial effect obtained is: when the display sub-interval of the real scene in the combined scene cannot be correctly identified based on the pixel depth value interval, the method of this embodiment can identify the display sub-interval based on the object category information of the imaging content of each pixel.
[0117] Third embodiment
[0118] The position of the person in the real scene in the combined scene is constantly changing, and will enter the display interval of the virtual sub-scene in the combined scene. The occlusion relationship between the person in the real scene and the content in the virtual sub-scene cannot be obtained through the occlusion relationship between the real scene display sub-interval and the virtual sub-scene. This embodiment improves the second embodiment, and step S201 also includes and After semantic segmentation is performed to obtain the category to which each pixel imaging content belongs, pixels classified as people are removed, and then and Perform image restoration. Image restoration is a well-known technical method for technicians in the field. For example, Xu Rui proposed a video image restoration method called "Deep-flow guided video impainting". [4] , which will not be elaborated here. Thus, the personnel in the real scene are eliminated, and the occlusion calculation between the personnel and the virtual sub-scene is avoided. Other moving objects in the real scene can also be eliminated in the same way.
[0119] Benefits: It avoids the erroneous occlusion of people and virtual sub-scenes in the real scene experience.
[0120] Fourth embodiment
[0121] This embodiment calculates the occlusion relationship between the person in the real scene and the content in the virtual sub-scene at the cost of generating and transmitting the depth image of the virtual sub-scene in the combined scene. i A depth image corresponding to the experience screen is also generated. In step S201, the image and After semantic segmentation is performed to obtain the category to which each pixel imaging content belongs, the pixels whose imaging content is a person are also marked. In step S2032, the image and For the pixels marked as personnel, the depth values of the corresponding depth images in the real scene experience screen are compared with the corresponding pixel depth values of the virtual scene synthetic experience screen images, and the occlusion relationship between the imaging pixels of the real scene personnel and the corresponding pixels of the virtual scene synthetic experience screen is obtained, thereby obtaining the pixel values of the synthesized combined scene experience screen image.
[0122] Benefits: The occlusion calculation is performed on the people in the real scene and the virtual scene content. However, since the accuracy of the depth value of the real scene experience cannot be ensured, the occlusion calculation result may have a large deviation, and the virtual sub-scene needs to generate and transmit depth images, which increases the network bandwidth usage.
[0123] Fifth embodiment
[0124] The fourth embodiment is improved as follows: in step S10, only the virtual sub-scene that is set to generate a depth image needs to generate a depth image while generating the experience screen; in step S201, based on the depth value of the pixel whose imaging content is the person, it is judged whether the imaged object point of the pixel whose imaging content is the person in the real scene experience screen is within the display range of the virtual sub-scene in the combined scene, that is, whether the real scene person enters the virtual sub-scene. For the virtual sub-scene with the real scene person entering, it is set that it needs to generate a depth image, otherwise it is set that it does not need to generate a depth image.
[0125] Advantages: Compared with the fourth embodiment, the number of virtual sub-scenes required to generate transmission depth images can be reduced.
[0126] Sixth embodiment
[0127] In the second and third embodiments, if the same object category exists in multiple display sub-intervals of the real scene, the display sub-interval to which the pixel imaging content belongs may not be identified according to the category of the pixel imaging content. This embodiment improves the first and second embodiments. The specific improvements are as follows.
[0128] In step S00, it also includes marking each display sub-interval of the real scene in the combined scene. Let any display subinterval Ω′ 0,i Contains the collection of object instances using Z i express.
[0129] The specific implementation of step S201 also includes the following:
[0130] For images and Perform instance segmentation to obtain the instance to which each pixel imaging content belongs. Image instance segmentation is a well-known technical means for those skilled in the art. For example, Li, Ruihuang et al. proposed an image instance segmentation method with excellent effect [3]. The image instance segmentation method will not be described in detail here. Any pixel in The imaging content belongs to the example of O L (uv) indicates that Any pixel in The imaging content belongs to the category O R (uv) indicates.
[0131] by Any pixel in For example, in the current position of user p [W j Q j], starting from the human eye (if is the pixel of the left eye image, then the optical center here is the left eye optical center), and Connect the image points on the display surface and extend the line to get the ray L, and calculate the real scene through which the ray L passes in the combined scene The display subinterval of . Let the real scene subinterval through which ray L passes be represented by the set Indicated by. Where M0 is the number of real scene subintervals that L passes through. Traverse all display subintervals in Λ, if there is a display subinterval in Λ Satisfy O R (uv)∈Z i , then set E 0,0 (uv) = i. Where Z i For any display subinterval The collection of instances contained. This method traverses All pixels in the left eye image can get the real scene stereoscopic visual experience picture. Similarly, it is possible to identify the pixel in the right eye image of the real scene stereoscopic visual experience screen in the combined scene The display sub-interval to which it belongs is recorded in E 0,1 Since the set real scene is in the display subinterval of the combined scene, it can be ensured that any instance exists in only one subinterval (large objects can also be divided into multiple instances to ensure that a single instance exists in only a single subinterval), it is possible to always identify the display subinterval to which each pixel in the real scene stereoscopic visual experience screen belongs in the combined scene.
[0132] Seventh embodiment
[0133] When the position information of a person can be collected by a wearable device with a positioning function, relatively accurate depth information of the person can be obtained based on the position of the person in the real scene. The sixth embodiment is improved as follows.
[0134] Step S201 also includes: by instance segmentation, the person ID corresponding to each pixel of the type of person can be marked, and the positioning function of the wearable device or other positioning methods can be used to obtain the person ID of any person. k The position in the combined scene, from which the current position and the user p's current pose [W j Q j ], you can get any person p k A depth value d relative to user p k When the depth value exceeds the effective depth information acquisition range of the XR terminal depth camera, the depth value d k Assign to image and The image content is personnel dk Pixels; Step S2032 also includes: image and The imaging content is the pixels of the person, and the depth value thereof is compared with the depth value of the corresponding pixel of the virtual scene experience screen to obtain the occlusion relationship between the imaging pixels of the person in the real scene and the corresponding pixels of the virtual scene experience screen.
[0135] Eighth embodiment:
[0136] In the first to seventh embodiments, the experience screen images of all virtual sub-scenes need to be transmitted to the terminal for image synthesis. The present embodiment improves this. The experience screen images of all virtual sub-scenes are first synthesized in the cloud, and the synthesized images are then transmitted to the user terminal for synthesis with the user's real scene experience screen to obtain the final virtual-real fusion combined scene experience screen.
[0137] Specific as Fig. 9 As shown. The system composition image synthesis module of the first embodiment is disassembled into the virtual scene image synthesis module 70 and the virtual and real scene image synthesis module 20 in the system of this embodiment. Among them, the virtual scene image synthesis module 70 is deployed in the cloud, and the virtual and real scene image synthesis module 20 is deployed in the user terminal. The virtual scene image synthesis module 70 runs step S2031, and the virtual and real scene image synthesis module 20 runs step S2032.
[0138] The beneficial effect is: reducing the amount of image data transmitted from the cloud to the user terminal.
[0139] Ninth embodiment:
[0140] When the combined scene is entirely virtual sub-scenes and does not include real scenes, especially when there is a virtual sub-scene that is non-convex and non-unidirectionally occluded with other virtual sub-scenes, multiple display sub-intervals can also be constructed for the virtual sub-scene in the combined scene, and the first embodiment is improved as follows: in step S00, a display sub-interval of the virtual sub-scene in the combined scene is also set, and the display sub-interval of the virtual sub-scene in the combined scene is unidirectionally occluded with the display interval or display sub-interval of other virtual sub-scenes in the combined scene; in step S10, for the virtual sub-scene with the set display sub-interval, it is also necessary to mark the display sub-interval to which the imaging content of each pixel in the generated experience screen image belongs, and in step S203, the occlusion relationship between pixels is obtained according to the occlusion relationship between the display interval and the display sub-interval of the virtual scene sub-scene, or between the display interval and the display interval, or between the display sub-interval and the display sub-interval, and further perform image synthesis.
[0141] Tenth embodiment:
[0142] The experience screen generation method of the first to ninth embodiments only uses the experience screen synthesis of some sub-scenes in the combined scene. The synthesis of the semi-finished experience screen and the experience screen of other sub-scenes in the combined scene can adopt the combined scene experience screen generation method given by application No. CN202210908795.0, 2023110288353, 2023110286220, 2023110288353, 2023110286220, etc., or some sub-scene experience screens are first synthesized using the above-mentioned other combined scene experience screen generation methods, and the synthesized semi-finished product is then synthesized with the remaining sub-scene experience screens using the experience screen generation method of the first to ninth embodiments.
[0143] The beneficial effects of the XR combined scene experience screen generation method of the present invention are as follows: the method obtains the occlusion relationship of the pixels from the occlusion relationship between the display sub-interval and other sub-scene display intervals or display sub-intervals by identifying the display sub-interval to which the pixels of the real scene experience screen belong in the combined scene, thereby avoiding obtaining the occlusion relationship by direct comparison of depth information with poor precision, thereby better realizing the correct occlusion of the scene content in the combined scene experience screen.
[0144] To achieve the above objectives, the present invention also proposes an XR combined scene experience screen generation system, the system comprising a memory, a processor, and an XR combined scene experience screen generation method program stored in the memory, and the XR combined scene experience screen generation method program executes the steps of the method described above when run by the processor.
[0145] To achieve the above objectives, the present invention also proposes an XR combined scene experience screen generation terminal, which executes step S203 or step S2031 or step S2032 of the XR combined scene experience screen generation method as described above.
[0146] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the steps of the XR combined scene experience screen generation method as described above are executed.
[0147] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
[0148] References
[0149] [1]Tosi,Fabio,et al."NeRF-Supervised Deep Stereo."Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.2023.
[0150] [2]Hu,Yubin,etal."Efficient Semantic Segmentation by AlteringResolutions for Compressed Videos."Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023.
[0151] [3]Li,Ruihuang,etal."SIM:Semantic-aware Instance Mask Generation forBox-Supervised Instance Segmentation."Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2023.
[0152] [4]Xu,Rui,Li,et al.Deep flow-guided video inpainting.In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition(pp.3723-3732).
Claims
1. A method for generating an extended reality (XR) combined scene experience screen, characterized in that: The method comprises the following steps: Step S20: The binocular stereoscopic experience images of each sub-scene are occluded and calculated to form a combined scene experience image. For any two pixels of the experience image that share the user's line of sight but belong to different sub-scenes, and ,like and Both image the non-sky area of the scene content and calculate and The occlusion relationship is as follows: The display interval of the combined scene is The occlusion relationship of the display interval of the combined scene is assigned as and Occlusion relationship; Among them, a single sub-scene can have multiple display sub-intervals in the combined scene, and any sub-scene of the combined scene is a virtual scene or a real scene. For a combined scene sub-scene with multiple display sub-intervals in the combined scene, , which has any pixel imaging the non-sky area of the scene content When there is an occlusion relationship between the image pixels in the non-sky area of the scene content and other combined scene sub-scenes, it is also necessary to judge The display sub-interval to which it belongs is determined by The occlusion relationship between the display sub-interval of the virtual scene and the display interval or display sub-interval of other combined scene sub-scenes is obtained, and the occlusion relationship between pixels is obtained. The combined scene includes a real scene, and the real scene has multiple display sub-intervals in the combined scene. Each virtual sub-scene in the combined scene is unidirectionally occluded with each display sub-interval of the real scene in the display interval of the combined scene. For any pixel belonging to the real scene experience screen ,calculate The real scene where the user's line of sight passes through is in the display sub-display interval set of the combined scene, and the judgment is The display sub-interval to which it belongs; The judgment The display sub-interval is one of the following: The depth value error range is the same as any display sub-interval in the set When the depth value ranges intersect on the line of sight, Belongs to the display sub-interval , or when The category to which the imaging content belongs belongs to any display sub-interval in the set If the object type is Belongs to the display sub-interval , or when The instance to which the imaging content belongs belongs to any display sub-interval in the set If the object instance is contained Belongs to the display sub-interval .
2. The generation method according to claim 1, characterized in that: The step S20 comprises: S201, identifying the display sub-interval to which each pixel in the real scene stereoscopic visual experience picture belongs in the combined scene; S202, obtaining the occlusion relationship between the display sub-interval of the real scene in the combined scene and the display interval of each virtual sub-scene under the current user posture; S203: synthesize the real scene experience picture and the virtual sub-scene experience picture according to the above occlusion relationship.
3. The generation method according to claim 2, characterized in that: Before step S20, there is step S10, which generates experience pictures of each sub-scene of the combined scene, wherein all any virtual sub-scenes Real-time user acquisition The pose in its coordinate system, and based on this real-time pose and Imaging interval, user exist Interpupillary distance in scene space, generating user right Before step S10, there is step S00: setting the generation parameters of the combined scene experience screen, including setting the imaging interval of each virtual sub-scene, the user Based on the pupil distance of each virtual sub-scene, the rotation and translation transformation relationship between the combined scene coordinate system and the virtual sub-scene coordinate system, and the display interval of each virtual sub-scene in the combined scene, multiple display sub-intervals of the real scene in the combined scene are set.
4. The generation method according to claim 3, characterized in that: The step S00 also includes marking the object categories contained in each display sub-interval of the real scene in the combined scene, and the step S201 also includes semantic segmentation of the real scene experience screen image to obtain the category to which each pixel imaging content belongs.
5. The generation method according to claim 3, characterized in that: The step S00 also includes marking the object instances contained in each display sub-interval of the real scene in the combined scene, and the step S201 also includes performing instance segmentation on the real scene experience screen image to obtain the instance to which each pixel imaging content belongs.
6. The generation method according to any one of claims 2 to 5, characterized in that: The step S00 also includes ensuring that each virtual sub-scene in the display interval of the combined scene is unidirectionally blocked with each display sub-interval of the real scene in the combined scene. In step S20, the blocking relationship between the real scene personnel and the virtual sub-scene content is processed by removing the real scene personnel from the real scene experience screen and repairing the image or obtaining the blocking relationship based on the depth information comparison. Step S203 includes: S2031, synthesizing all virtual sub-scene experience pictures to obtain a virtual scene synthesized experience picture; S2032, the real scene experience picture and the virtual scene synthetic experience picture are synthesized to obtain a combined scene experience picture.
7. An XR combined scene experience screen generation system, the system comprising a memory, a processor, and an XR combined scene experience screen generation method program stored in the memory, wherein the XR combined scene experience screen generation method program executes the steps of any one of the methods described in claims 1 to 6 when executed by the processor.
8. An XR combined scene experience screen generation terminal, wherein the terminal executes step S203 or step S2031 or step S2032 of the method according to any one of claims 2 to 6.
9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when called by a processor, executes the steps of the method according to any one of claims 1 to 6.
