A method, system and medium for automatically combining XR combined scenes
By automatically adjusting the presentation of sub-scenes within a combined scene, the problem of excessive manual adjustment by users in existing technologies is solved, achieving efficient presentation of combined scenes.
Patent Information
- Application Number
- CN202410411734.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-04-08
AI Technical Summary
In existing technologies, the combined scene construction method fails to effectively and automatically adjust the presentation of sub-scenes in the combined scene, which requires users to make heavy manual adjustments to ensure that all important information is presented, resulting in an excessive burden.
By automatically adjusting the presentation of sub-scenes in the composite scene, including adjusting their content, pose angle, position, and scaling size, and utilizing system-generated path optimization goals and constraints, the system ensures that sub-scene information is presented sequentially in the composite scene.
It enables automatic adjustment of sub-scene presentation in combined scenes, reducing the tedious work of manual adjustment by users and improving the efficiency of combined scenes.
Smart Images

Figure CN118200510B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of XR technology, and in particular to an automatic combination method, system and medium for XR combination scenes. Background Art
[0002] Virtual reality (VR), augmented reality (AR), mixed reality (MR) and other technologies are interconnected and integrated, collectively referred to as extended reality (XR). XR technology and the metaverse derived from it will have a profound impact on our learning, social interaction, and work. A combined scene can be constructed by combining two or more single-person experience scenes and / or multi-person collaborative experience scenes. The combined scene is a sub-scene of the combined scene, and the sub-scene can be one or more of a virtual scene, a real scene, and a virtual-real mixed scene. When a user experiences a combined scene, the user can simultaneously view the stereoscopic three-dimensional information of all sub-scenes in the combined scene, and can further interact with all sub-scenes in space. The combined scene can break the spatial isolation between scenes and construct a variety of experience activity types. For example, in a classroom, multiple single-person teaching activity scenes or multi-person collaborative teaching activity scenes can be combined to generate a teaching combination scene containing multiple sub-scenes. By changing the types and number of included scenes, a wide variety of combined scenes can be constructed.
[0003] There are four main technical methods for building a combined scene. The first method is to load all sub-scenes in a single XR application and directly render the user's combined scene experience screen together, as described in paragraph 0094 of the specification of document 1. [1] The second method is to set an imaging interval for the sub-scene, and the scene content within the sub-scene imaging interval is independently rendered to generate the user's experience screen, and then the user's combined scene experience screen is synthesized through occlusion calculation with the experience screen of other contents of the combined scene. [2-8] The third method is to render the sub-scene independently to generate the experience screen, but the imaging interval is not limited to the sub-scene. The sub-scene is cropped according to a certain imaging interval or display interval constraint, and then combined with the binocular stereo vision screen of other contents of the combined scene to form the combined scene experience screen. [9] The fourth method is to generate and transmit the scene 3D information (such as voxel map) in real time in the sub-scene, load the received sub-scene 3D information into the combined scene, and render the combined scene to generate the user experience screen
[10] .
[0004] For any sub-scene, the combined scene only allocates a portion of the three-dimensional space to present this sub-scene. If all the scene contents of the sub-scene are presented simultaneously in the limited three-dimensional space allocated by the combined scene, the sub-scene often needs to be reduced in size. However, reducing the sub-scene is likely to make some of the sub-scene information indistinguishable to the user. Therefore, it is often impossible to present all the information in the scene at the same time in a combined scene. In order to make the information in the scene that is valuable to the user or the information that the user is interested in better perceived by the user, it is necessary to constantly adjust the scene content of the sub-scene presented in the combined scene. If people are asked to make manual adjustments constantly, it will bring a heavy burden. In the existing combined scene construction method [1-10] No automatic adjustment method is given. Summary of the invention
[0005] The main purpose of the present invention is to provide an automatic combination method, system, and medium for XR combined scenes, which automatically adjust the presentation of sub-scenes in the combined scene so that the presentation of sub-scenes in the combined scene can better present all information of interest to users or information valuable to users in the sub-scenes through temporal accumulation.
[0006] In order to achieve the above object, the present invention proposes an automatic combination method of XR combination scenes, the method comprising:
[0007] The system generates a combined scene experience for a first user. In the combined scene experience of the first user, according to the scene content and / or scene information that the first sub-scene needs to present in the combined scene, the system automatically adjusts the presentation of the first sub-scene in the combined scene, so that the scene content and / or scene information of the first sub-scene is presented in the combined scene. The automatic adjustment of the presentation of the first sub-scene in the combined scene specifically includes adjusting one or more of the content, posture angle, position, and scaling size of the first sub-scene presented in the combined scene. The first sub-scene is one of a virtual scene, a virtual-reality fusion scene, and a real scene.
[0008] Among them, the first sub-scene is presented in the first display interval of the combined scene, the first display interval is defined in the combined scene coordinate system or the first user field of view coordinate system, and the content of the first sub-scene presented in the combined scene is adjusted, specifically, the three-dimensional imaging interval of the first sub-scene presented in the combined scene is adjusted, and the position, attitude angle, and scaling size of the first sub-scene presented in the combined scene are adjusted specifically by adjusting the rotation amount, translation amount, and scaling amount in the rotation, translation, and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, and the three-dimensional imaging interval and the rotation, translation, and scaling relationship between the coordinate systems satisfy the constraints of the first display interval; the automatic adjustment of the presentation of the first sub-scene in the combined scene also includes automatically adjusting the playback of the sound of the first sub-scene in the combined scene.
[0009] Preferably, the constraint of the first display interval is that the three-dimensional interval of the combined scene obtained after the three-dimensional imaging interval is transformed by the rotation, translation and scaling relationship of the coordinate system is included in the first display interval or is equal to the first display interval, and the automatic adjustment of the playback of the first sub-scene sound in the combined scene specifically includes automatically adjusting which sound sources of the first sub-scene produce sounds played in the combined scene, and / or the position or posture of each sound source of the first sub-scene played in the combined scene.
[0010] Preferably, the system generates a combined scene experience for the first user in one of the following ways: a picture generated by the first sub-scene under the imaging interval constraint is synthesized with pictures of other scene contents in the combined scene to obtain a combined scene experience picture; a picture generated by the first sub-scene without the imaging interval constraint is cropped under the imaging interval constraint and synthesized with pictures of other scene contents in the combined scene to obtain a combined scene experience picture; the first sub-scene generates scene three-dimensional information and loads it into the combined scene and then renders it to obtain a combined scene experience picture; the first sub-scene and other contents of the combined scene are directly rendered together to generate a combined scene picture.
[0011] Among them, when the first sub-scene generates an image under the constraint of the imaging interval or not under the constraint of the imaging interval, if the first display interval is defined in the first user field of view coordinate system, the image generated by the first sub-scene is not affected by the change of the first user's posture. If the first display interval is defined in the combined scene coordinate system, it is necessary to transform the first user's posture and pupil distance to the first sub-scene coordinate system according to the transformation relationship between the coordinate systems, and generate the first sub-scene image according to the posture and pupil distance obtained after the transformation. In the process of automatically adjusting the first sub-scene imaging interval and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, the system forms a path for automatically adjusting the first sub-scene in the combined scene coordinate system. The path for scene presentation is automatically generated under the set optimization goals and constraints for presenting the scene content and / or scene information of the first sub-scene to the combined scene, and / or the strategy for presenting the scene content and / or scene information of the first sub-scene to the combined scene. Any point on the path has a corresponding imaging interval, a rotation, translation, and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, and / or the time required for the human eye to perceive the information. Based on this path, the system automatically adjusts the imaging interval of the first sub-scene and the rotation, translation, and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, thereby automatically adjusting the presentation of the first sub-scene in the combined scene.
[0012] More specifically, the scene content and / or scene information is one or more of static, real-time dynamic, and non-real-time dynamic, and the optimization goal is one or more of the following: the path length is shortest, the most information of the first sub-scene accumulated in time sequence is presented in the combined scene, the most information of the first sub-scene can be perceived by the first user, the non-real-time dynamic information of the first sub-scene is the least in reverse order, and the time required for the first user to perceive all information of the first sub-scene in the combined scene is shortest; the constraint condition is one or more of the following constraint conditions: all static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene are presented in the combined scene after accumulation in time sequence, all static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene can be perceived by the first user without moving the position The user perceives in the combined scene that the rate of change of the rotation relationship in the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user's field of view coordinate system does not exceed a specific threshold, the rate of change of the translation relationship in the rotation, translation and scaling relationship does not exceed a specific threshold, the rate of change of the scaling relationship in the rotation, translation and scaling relationship does not exceed a specific threshold, the time for the imaging interval to pass through or stay on the scene content and / or scene information meets the time requirement for the first user or all users to fully perceive the scene content and / or scene information, and the dynamic information time is not in reverse order; the strategy is that when the first display interval of the combined scene cannot fully present all real-time dynamic information and / or scene content of the first sub-scene, it is necessary to make a trade-off, and the trade-off strategy is one or more of first-come, first-served, rotation, proximity, priority or information volume competition.
[0013] Preferably, the influencing factors of the measurement value of the path length include one or more of the translation change amount, rotation change amount, and scaling transformation amount in the rotation, translation, and scaling change relationship from the first sub-scene coordinate system to the first user coordinate system or the combined scene coordinate system. The specific method for generating the path for automatically adjusting the presentation of the first sub-scene in the combined scene is: under the constraint that the first sub-scene can cover all scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene after accumulation, or under the constraint that all scene content and / or scene information that needs to be presented in the combined scene can be fully perceived by the first user without the first user moving, a group of imaging areas are selected. As the key imaging interval, and further obtaining the rotation, translation and scaling relationship between the first sub-scene coordinate system corresponding to each key imaging interval and the first user field of view coordinate system or the combined scene coordinate system under the constraint of the first display interval, and according to the human eye perception characteristics of the scene content and / or scene information contained in each key imaging interval that needs to be presented in the combined scene, calculate the length of time that needs to stay or pass through this key imaging interval during the imaging interval adjustment process, any sequence of the key imaging intervals can constitute a path for the system to automatically adjust the presentation of the first sub-scene in the combined scene, and according to the set optimization goals and constraints and / or the presentation of the first sub-scene scene content and / or scene information to the combined scene. Or a strategy for presenting the scene content and / or scene information of the first sub-scene to the combined scene, from which a path consisting of a sequence of key imaging intervals is obtained, and this key imaging interval sequence, the rotation, translation and scaling relationship between the first sub-scene coordinate system corresponding to each key imaging interval and the first user field of view coordinate system or the combined scene coordinate system, and the length of stay in this key imaging interval during the imaging interval adjustment process are directly constructed to form a path for adjusting the presentation of the first sub-scene in the combined scene in a jumpy manner, or the imaging intervals of any key imaging intervals that are adjacent in sequence in the key imaging interval sequence and the corresponding rotation, translation and scaling relationship between the first sub-scene coordinate system and the first user field of view coordinate system or the combined scene coordinate system are interpolated. value, and further construct a path for continuously changing and adjusting the first sub-scene to be presented in the combined scene according to the duration of stay or the duration of time passed in each key imaging interval during the imaging interval adjustment process; or construct a path for adjusting the first sub-scene to be presented in the combined scene with partial jumping changes and partial continuous changes by combining the method of constructing the jumping change path and the continuously changing path; the first sub-scene sound source played in the combined scene must belong to the imaging interval of the first sub-scene, and the position or posture of the first sub-scene sound source played in the combined scene in the combined scene is the position or posture of the first sub-scene sound source in the first sub-scene coordinate system converted according to the transformation relationship between the first sub-scene coordinate system and the combined scene coordinate system.
[0014] In addition, the scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene is specifically embodied as a set of objects and / or object parts and their states, and the dynamic scene content and / or scene information is one or more of the following scene changes: voice, object disappearance, object generation, object rotation, object movement, object scaling, object deformation, object texture change, and chemical change; before the system automatically adjusts the presentation of the first sub-scene in the combined scene, it is also necessary to determine the scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene, and the determination method is one or more of manual calibration, statistical calibration, and artificial intelligence calibration; the system automatically adjusts the presentation of the first sub-scene in the combined scene, which can be specifically automatic adjustment according to the posture change of the first user or the posture change of any object in the first sub-scene; in order to avoid occlusion between objects, some objects in the first sub-scene or the combined scene can be set to a hidden state or a transparent state.
[0015] To achieve the above objectives, the present invention also proposes an XR combination scene automatic combination system, the system comprising a memory, a processor, and an XR combination scene automatic combination method program stored in the memory, and the XR combination scene automatic combination method program executes the above method when run by the processor.
[0016] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the automatic combination method of XR combined scenes as described above is executed.
[0017] The beneficial effect of the automatic combination method, system and medium of XR combined scenes of the present invention is: under certain optimization goals and constraints and / or optimization strategies, the present invention automatically adjusts the presentation of the first sub-scene in the combined scene, thereby eliminating the complicated work of manually adjusting the sub-scene imaging range in the combined scene experience, thereby significantly improving the efficiency of the combined scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is an example of combined scene function;
[0019] Figure 2 It is a schematic diagram of the principle of an automatic combination method of XR combination scenes of the present invention;
[0020] Figure 3 It is an example of the first sub-scene space segmentation of the imaging interval;
[0021] Figure 4 This is an example of setting a mapping relationship between a first sub-scene imaging interval and a first display interval of a combined scene;
[0022] Figure 5It is to set a path optimization example 1 of an automatic combination method of an XR combination scene of the present invention;
[0023] Figure 6 It is to set a path optimization example 2 of an automatic combination method of an XR combination scene of the present invention;
[0024] Figure 7 It is to set a path optimization example 3 of an automatic combination method of an XR combination scene of the present invention;
[0025] Figure 8 is an example of scene dynamic information;
[0026] Fig. 9 This is an example of path optimization in real-time dynamic scenes;
[0027] Fig.10 This is a flow chart of the first invention embodiment of the method for automatically combining XR combined scenes of the present invention.
[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION
[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0030] Combining two or more single-player experience scenes and / or multi-player collaborative experience scenes can construct a combined scene. The combined scenes are simultaneously presented in the combined scene and become sub-scenes of the combined scene. When the sub-scene is presented in the combined scene, the sound of the sub-scene can also be transmitted to the combined scene. Figure 1 For example, in (a), user A is experiencing immersive interaction in scene 1, and in (b), user B is experiencing immersive interaction in scene 2. In (d), part of the three-dimensional area of scene 1 is presented in the experience space of user B, and a combined scene including scene 1 and scene 2 is constructed for user B. When user B experiences scene 2 normally, he can watch the live scene of scene 1 in three dimensions and interact with scene 1 in space. The combined scene can be experienced by one user or by multiple users in collaboration.
[0031] First sub-scene 0 is a sub-scene of the combined scene S, s 0There is scene content and / or scene information that needs to be presented in the combined scene. Scene content refers to the objects, object states, relationships between objects, and changes in object states and relationships between objects contained in the scene, while scene information refers to the information reflected by the scene content. Since scene content and scene information are related, for simplicity, when describing the scene content and / or scene information of the combined scene below, "scene content" is omitted and the scene information of the combined scene is directly used. The scene information needs to be presented in the combined scene because it is valuable to the user or the user is interested in it. The first sub-scene s 0 Only the first display interval of the combined scene is assigned to present s 0 The scene content, so it will appear that you cannot put s 0 All scene information is presented in the combined scene at the same time. 0 In the presentation of the combined scene, 0 There are two issues worth considering. First, if a static scene or all non-dynamic information of a scene is fully presented in a combined scene through time-series accumulation, how to adjust s 0 The scene content presented in the combined scene minimizes the time and / or provides the best user experience. Secondly, if it is a dynamic scene, especially a real-time dynamic scene, dynamic information will be lost in a moment. When it is impossible to present all the information in the scene in the combined scene, how to choose and discard information? The scene information is embodied in the information carrier, which can be any object including people, animals, plants, production equipment, products, etc., or a group composed of the above multiple objects or any component or part of the above single object.
[0032] In the present invention, the system forms a path for automatically adjusting the presentation of the first sub-scene in the combined scene during the process of automatically adjusting the imaging interval of the first sub-scene and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system. The path is automatically generated under the set optimization goals and constraints for presenting the first sub-scene scene content and / or scene information to the combined scene, and / or the strategy for presenting the first sub-scene scene content and / or scene information to the combined scene. Any point on the path has a corresponding imaging interval, rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, and time information. Based on this path, the system automatically adjusts the imaging interval of the first sub-scene and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, thereby automatically adjusting the presentation of the first sub-scene in the combined scene. Figure 2For example, a first display interval is set in the combined scene to present the first sub-scene, and the system automatically adjusts the imaging intervals to sequentially present the scene contents in the first imaging interval, the second imaging interval, and the third imaging interval of the first sub-scene in the first display interval of the combined scene. The first sub-scene is presented in the imaging interval of the combined scene, moves from the first imaging interval to the second imaging interval, and then moves to the third imaging interval, thereby forming a path for automatically adjusting the presentation of the first sub-scene in the combined scene.
[0033] The optimization goal is to have one or more of the following: the path length is shortest, the most information of the first sub-scene accumulated in time sequence is presented in the combined scene, the most information of the first sub-scene can be perceived by the first user, the non-real-time dynamic information of the first sub-scene is the least in reverse order, and the time required for the first user to perceive all information of the first sub-scene in the combined scene is shortest; the constraint condition is one or more of the following constraint conditions: all static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene are presented in the combined scene after being accumulated in time sequence, all static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene can be perceived by the first user in the combined scene without the first user moving his position, and the coordinates of the first sub-scene are the same as those of the first sub-scene. The rate of change of the rotation relationship in the rotation, translation and scaling relationship between the system and the combined scene coordinate system or the first user's field of view coordinate system does not exceed a specific threshold, the rate of change of the translation relationship in the rotation, translation and scaling relationship does not exceed a specific threshold, the rate of change of the scaling relationship in the rotation, translation and scaling relationship does not exceed a specific threshold, the time for the imaging interval to pass through or stay on the scene content and / or scene information meets the time requirement for the first user or all users to fully perceive the scene content and / or scene information, and the dynamic information time is not in reverse order; the strategy is that when the first display interval of the combined scene cannot fully present all the real-time dynamic information and / or scene content of the first sub-scene, it is necessary to make a trade-off, and the trade-off strategy is one or more of first-come, first-served, rotation, proximity, priority or information volume competition.
[0034] Important technical terms involved in the present invention are explained below.
[0035] Scenarios and Scenario Examples
[0036] A scene defines the objects, object states, object operation logic, and interaction logic between objects contained in a three-dimensional space; a scene instance is a program process or thread that the system calls on computer processors, memory, graphics cards and other computing resources to run in real time according to the scene definition. This program process or thread calculates the state of each object in the scene in real time, renders the screen, and responds to user interaction. When a single scene has multiple user experiences at the same time, if the computing resources available to a single scene instance cannot generate experience screens for all users in real time, it is necessary to generate multiple scene instances for this scene and allocate them to each user. These scene instances establish communication connections to synchronize the state of objects in the scene, and each scene instance generates an experience screen for the corresponding user in real time, so that each user can experience this scene together.
[0037] Pupillary distance and stereoscopic vision
[0038] When the human eye looks at an object, the left eye and the right eye form an image of the object respectively. The image positions of the same object point in the left eye and the right eye are different. This difference is parallax. Parallax allows the human eye to correctly perceive the three-dimensional information of the object point. When the left and right eye stereoscopic pictures are shown to people, the stereoscopic visual information perceived by the person is affected by the ratio of his or her own pupil distance to the pupil distance of the picture imaging. The image depth information perceived by the human eye needs to be corrected according to this pupil distance ratio.
[0039] User experience space and coordinate system
[0040] The user experience space is a three-dimensional space for the user's immersive experience. The user experience space coordinate system can be pre-set or automatically constructed by the XR headset. In the embodiment of the present invention, the user experience space coordinate system is set as follows: this coordinate system defines the positive direction of the y-axis in the vertical upward direction of the horizontal plane, and defines the x-axis and z-axis on the horizontal plane.
[0041] Combined scene space and coordinate system
[0042] The coordinate system of the combined space composed of multiple sub-scenes can share the coordinate system with the user experience space, or share a coordinate system with one of the sub-scenes of the user immersive experience. The coordinate system of the combined scene can also be newly constructed according to needs.
[0043] Field of view and coordinate system
[0044] The field of view is the spatial range visible to the user. The user's field of view changes with the movement of the user's head and eyeballs. The embodiment of the present invention defines the user's field of view coordinate system with reference to the user's head, so that the movement of the user's head and eyeballs does not affect the position relationship of the user's field of view coordinate system relative to the user's head. In the embodiment of the present invention, the field of view coordinate system is set in the same way as the binocular camera coordinate system: on the line connecting the binocular center of the human eye, the midpoint between the binocular center of the binocular eye is used as the origin of the coordinate system, the direction of the binocular line connecting the human eye from the left eye to the right eye is defined as the positive direction of the x-axis, and the direction of the face is defined as the positive direction of the z-axis, and the positive direction of the y-axis is defined to be perpendicular to the x-axis and the z-axis and satisfy the left-hand coordinate system.
[0045] Similarity of three-dimensional intervals
[0046] Ω and For two three-dimensional intervals in the same coordinate system or different coordinate systems, if there is a rotation, translation and scaling relationship H, but Similar to Ω.
[0047] Invention Example 1.
[0048] In the combined scene construction method described in references [2-6] of this embodiment, the first sub-scene is presented in the first display interval of the combined scene, the first display interval is a three-dimensional interval defined in the combined scene coordinate system, and the sub-scene content visible to the first user changes with the posture of the first user. The user real-time experience screen generated by the first sub-scene is synthesized with the experience screen of other contents of the combined scene to obtain the combined scene experience screen.
[0049] Among them, for the imaging interval of any point on the changing path of the imaging interval of the first sub-scene, after determining the mapping relationship between the points in the imaging interval and the points in the first display area in the combined scene, the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system at the arbitrary point in the imaging interval of the first sub-scene can be calculated. In an embodiment of the present invention, the imaging interval of the first sub-scene and the first display area in the combined scene must have similarity. Therefore, the corresponding rotation and translation relationship between the first sub-scene coordinate system and the combined scene coordinate system can be calculated based on the mapping relationship between the boundary points of the imaging interval and the boundary points of the display interval. Further, the virtual camera posture and pupil distance used by the first sub-scene to generate the combined scene experience screen can be obtained.
[0050] Sub-scenes 0 The information that needs to be presented in the combined scene is collected Indicates that any information l in L i The carrier object o i Indicates that object o i In s 0 The three-dimensional space occupied by iIndicates that the carrier of all information in L is in s 0 The three-dimensional space occupied by express.
[0051] The embodiment of the present invention is in a combined scenario In the coordinate system, the first sub-scene s is pre-set 0 The display interval of the combined scene is the first display interval System automatic adjustment 0 The imaging interval of the first sub-scene s 0 Presented in combined scene The content in s 0 The imaging interval of L is completely covered by the accumulation in time series, so that all the information in L is presented in the combined scene. In this embodiment, when the imaging interval is automatically adjusted, only the position and size of the imaging interval are changed, and the posture is not adjusted. In addition, in this embodiment, the first sub-scene s 0 It is a static scene or only presents non-dynamic information of a dynamic scene.
[0052] The method of this embodiment comprises the following steps:
[0053] Step S00, marking the information of the first sub-scene that needs to be presented in the combined scene;
[0054] Step S10, generating a first sub-scene key imaging interval;
[0055] Step S20, calculating the rotation, translation and scaling relationship between the first sub-scene coordinate system corresponding to each key imaging interval and the combined scene coordinate system;
[0056] Step S30, searching for the optimal path;
[0057] Step S40, the sub-scene generates a first sub-scene experience screen according to the order of each key imaging interval on the path, the dwell time, and the calculated imaging pupil distance and posture of the first user in the first sub-scene. The first sub-scene experience screen is synthesized with the experience screens of other scene contents of the combined scene to obtain the combined scene experience screen.
[0058] The specific implementation of each step of this embodiment is described in detail below.
[0059] Step S00: marking the information that the first sub-scene needs to be presented in the combined scene
[0060] Mark the first sub-scene s 0 The information to be presented in the combined scene includes the three-dimensional space of the first sub-scene occupied by the marking information carrier and / or the carrier. 0 The information that needs to be presented in the combined scene is collected Indicates that any information l in Li The carrier object o i Indicates that object o i In s 0 The three-dimensional space occupied by i Represents. Object o i Yes i The information carrier is the object o i The existence, action, and other states of the i .
[0061] The scale of each information can be further marked. i The scale of δ i This means that when the sub-scene s 0 Reduced to a size no smaller than the original size Times, information l i It can still be fully perceived by the user, otherwise it cannot be perceived. For example, for a machine tool, the information scale of the overall outline of the machine tool is obviously larger than the information scale of the controls on the control panel. Therefore, when the size of the machine tool is reduced, the overall outline of the machine tool may still be fully perceived by the user, but the control information on the control panel cannot be distinguished and thus cannot be perceived by the user. The same information carrier can contain information of multiple different scales.
[0062] The time required for the user to perceive each piece of information can also be further marked. i The time required is Δt i Indicates that Δt i is the shortest or average time required for users to perceive this information, and information l i is positively correlated with the amount of information.
[0063] The calibration methods include manual calibration, statistical calibration and artificial intelligence calibration.
[0064] 1) Manual calibration
[0065] The staff responsible for information calibration will subjectively judge the first sub-scenario s based on their knowledge, experience, and scene characteristics. 0 Check the carriers of each information to be presented in the combined scene. After checking the information carrier, the information carriers in the s 0 The 3D scene space occupied by the object. For example, in XR engines such as Unity3D, each object in the scene has a corresponding surface 3D mesh data. The surface and the area surrounded by the surface are the 3D scene space occupied by the object. If there is no direct surface data of the information carrier, such as the space occupied by an object in a real scene, the 3D scene space occupied by the object can be obtained by automatic object recognition and 3D reconstruction. You can also use the box selection tool to manually select a certain area as the space occupied by the object.0 Three-dimensional scene space.
[0066] Then the scale of each information and the time required for user perception are further calibrated.
[0067] 2) Statistical calibration
[0068] Scenes 0 Provide a group of users for experience, and record the objects or parts of objects that all users focus on during the experience. Specifically, when the user is in a gaze state, record the scene object or part of the scene object that the user's sight is pointing to. Pupil size is related to a person's attention level. When people focus on a task, the pupil will dilate. By measuring the change in pupil diameter, the change in attention can be evaluated. High-performance XR headsets represented by Apple Vision Pro have high-precision perception capabilities of human eye status information, and the perceived human eye status information can be used to calculate the user's pupil diameter. Set a threshold for the pupil diameter of human eye gaze. When the user's pupil aperture is larger than the threshold, it is considered that the user is looking.
[0069] Therefore, for any user, when the user is in a fixation state according to the pupil diameter, the scene object or scene object part touched by the user's sight is recorded, as well as the time the user's sight touches the object or object part, and the distance between the user and the object or object part is recorded. The larger the distance, the larger the information scale. 0 When experiencing an immersive experience, the user’s gaze data is used to obtain the scene s 0 The information carrier and information scale that need to be presented in the combined scene and the time required for the human eye to perceive the information. For example: the system sets the information that needs to be presented in the combined scene. The user's average gaze time threshold is The base distance for viewing information is For scenes 0 Any object or part of an object i , the average user gaze duration is Δt i seconds, and the average distance between the user and the object or object part when looking at the object is d i meters, when Then object o i The information carried by the carrier is i Indicates that the occupied s 0 Space is o i The time required for the user to perceive the information in the interval surrounded by the outline is Δt i , the information scale is Among them, the specific method for obtaining the scene object or the part of the scene object touched by the user's line of sight is: for virtual or virtual-real scenes, XR engines such as unity3d have collision calculation functions, which can directly give the scene virtual object or the part of the scene virtual object touched by the user's line of sight. For objects in the real scene, first obtain the real scene image corresponding to the user's binoculars, determine the pixels corresponding to the user's line of sight, and perform segmentation and recognition calculations on the image. Then, the scene object or the part of the scene object to which the pixels corresponding to the user's line of sight belong can be obtained, and thus the scene object or the part of the scene object touched by the user's line of sight is obtained.
[0070] 3) Artificial Intelligence Calibration
[0071] With the development of big models and AIGC technology, the performance of artificial intelligence has improved rapidly. Artificial intelligence can be based on the application purpose of combined scenarios and sub-scenario s 0 The content characteristics and other information of the scene can be further communicated with the user to obtain accurate user experience needs, and on this basis, the sub-scene s can be automatically and accurately determined. 0 The information that needs to be presented in the combined scene, the information carrier, the scale of each information, and the time required for the user to perceive each information.
[0072] Step S10: Generate the first sub-scene key imaging interval
[0073] In the first sub-scene 0 In the example, a group of imaging intervals is selected. After the information that can be perceived by the user in the combined scene is accumulated, it completely covers s 0 The information that needs to be presented in the combined scene. Between these imaging intervals, the duplication of information that can be perceived by the user in the combined scene should be avoided as much as possible, so that the s 0 The complete coverage of the combined scene information needs to be presented. In the present invention, the selected group of imaging intervals is called the key imaging interval.
[0074] The strategies for selecting the key imaging interval can vary greatly. In the embodiments of the present invention, two strategies for selecting the key imaging interval are described in detail.
[0075] Selection strategy 1
[0076] Statistics of the first sub-scene s 0 All information scales that need to be presented in the information set L of the combined scene. For each information scale, the first sub-scene s 0 The three-dimensional space of the image is evenly divided into a series of imaging intervals of the same size. All imaging intervals are not only similar to the combined scene display interval, but also have a size ratio equal to the corresponding information scale. If the display interval is not square, in order to make the accumulation of all the segmented imaging intervals cover s0 In the three-dimensional space, adjacent imaging intervals need to partially overlap. Specifically, a series of imaging intervals are obtained by translating and / or rotating the same imaging interval template to achieve s 0 For any information scale Its imaging interval template is s 0 Internal and combined scenes Display range Similar, the scale ratio is Thus, the imaging interval template is available Indicates that It means Scale on the x, y, and z axes times. 0 In a series of imaging intervals generated by spatial segmentation, traverse all the image intervals that need to be presented in the combined scene. The scale is Sub-scenes 0 The information carrier, calculate the space occupied by each information carrier 0 The imaging interval of the three-dimensional space to which the information carrier belongs is calculated by sampling a number of discrete points from the space occupied by the information carrier. The coordinate values of these discrete points can be used to calculate the imaging interval to which the information carrier belongs. Any carrier of information i , and its imaging interval is s 0 The three-dimensional space occupied by i The imaging interval to which it belongs becomes the key imaging interval. In order to reduce the number of key imaging intervals, Select any two or more adjacent key imaging intervals and find whether there is one that passes The imaging interval obtained by translation and rotation can cover the scales contained in the multiple adjacent key imaging intervals. The specific method of searching is: The translation and rotation relationship interpolation of the image is performed by setting a set of different interpolation parameters to obtain multiple new imaging intervals. If a new imaging interval covers the scale contained in the adjacent key imaging interval The need to present in the combined scene If the information is , the adjacent imaging intervals in this group are no longer set as key imaging intervals, and the new imaging interval is added as a key imaging interval. For all information scales, s are evenly divided in the above manner. 0 After the three-dimensional space is obtained, the key imaging intervals are selected and all the selected key imaging intervals are grouped together. On this basis, the key imaging intervals can be further reduced. Information carrier Key imaging interval It is the imaging interval template Rotate and translate to get like Completely includes The three-dimensional space occupied by No need to consider the information carrier when selecting the key imaging interval
[0077] by Figure 3 For example, the combined scene is used to present the sub-scene s 0 The display interval is a size of The shape of the three-dimensional interval is a rectangular parallelepiped, so for the information scale Splits 0 The imaging interval template is of size A three-dimensional interval with a rectangular shape, such as Figure 3 After segmentation, a series of regularly arranged rectangular imaging intervals are obtained. Figure 3 The virtual objects are all information carriers to be presented in the combined scene, and all the rectangular imaging intervals containing the virtual objects are key imaging intervals.
[0078] Selection strategy 2
[0079] Different from strategy 1 which evenly divides the sub-scene space to generate imaging intervals, this strategy generates key imaging intervals according to the shape of the information carrier.
[0080] For s 0 Anything that needs to be presented in the combined scene Information in i , whose carrier is object o i , object o i In s 0 The three-dimensional space occupied by i , the information scale is δ m If there exists a process that satisfies δ′≤δ m Imaging interval template The imaging interval obtained after rotation and translation completely covers υ i , then select an imaging interval as the key imaging interval; if it does not exist, select the imaging interval template After rotation and translation, multiple (but as few as possible) imaging intervals are accumulated to completely cover v i Among them, the constraint condition δ′≤δ m To ensure s 0 After zooming out and presenting in the combined scene, the information l i Still visible.
[0081] Determine whether there is δ′≤δ m Imaging interval template The imaging interval obtained after rotation and translation completely covers υ i The specific method is: first find υ i The boundary points (for example, in Unity3D, the surface mesh is used to represent the shape of the object, then all the grid points on the surface mesh are the boundary points), find the average value of the coordinate values of the boundary points, and get υ i Center point coordinates, imaging interval template To translate The center point and υ i Center point alignment, if at this time Contains i For all boundary points, there exists δ′≤δ m Imaging interval template The imaging interval obtained after rotation and translation completely covers υ i If not, set a set of pose values and a set of position offsets to construct the search space. For any offset in this set of offsets, The center point is translated according to this offset, and then adjusted The pose of this set of pose values is any value. According to this method, a set of new imaging intervals can be obtained. If there is a new imaging interval containing υ i For all boundary points, there exists δ′≤δ m Imaging interval template The imaging interval obtained after rotation and translation completely covers υ i .
[0082] When there exists a process satisfying δ′≤δ m Imaging interval template The imaging interval obtained after rotation and translation completely covers v i When , the method of selecting an imaging interval as the key imaging interval is: select an imaging interval with the smallest δ′ Completely cover i , so that the information is presented in the combined scene with the maximum resolution, specifically by setting a set of ratios δ m Small scale value, for any scale value δ″, Determine whether the imaging interval template has complete coverage of the imaging interval after rotation and translation. i (Refer to the previous text for the judgment method), select a minimum scale δ′ that meets the requirements, and select any template of the imaging area obtained by the search After rotation and translation, the complete coverage is i The imaging interval is the key imaging interval.
[0083] If there is no δ′≤δ m Imaging interval template The imaging interval obtained after rotation and translation completely covers υ i , select the imaging interval template The multiple (but as few as possible) imaging intervals obtained after rotation and translation completely cover v i The specific method is: determine whether there is a template The imaging interval obtained after rotation and translation completely covers v i , if it exists, then divide the obtained imaging interval evenly into 2 3 Shape, size and The same imaging interval, that is, 8 imaging intervals are obtained by segmentation, and the ones that do not contain v are eliminated. i The remaining imaging intervals are key imaging intervals. Otherwise, it is determined whether there is a template The imaging interval obtained after rotation and translation completely covers v i , if it exists, then divide the obtained imaging interval evenly into 3 3 Shape, size and The same imaging interval, that is, 27 imaging intervals are obtained by segmentation, and those that do not contain v are eliminated. i The remaining imaging intervals are key imaging intervals, and so on.
[0084] Similar to Strategy 1, measures can also be taken to delete key imaging intervals. For details, see Strategy 1.
[0085] Note that when the combined scene is used to present s 0 When the display interval is different, the selected key imaging interval will generally be different. Let the key imaging interval selected in this step be the set For any key imaging interval Ω i ′, you can use Indicates that H i ′ represents the rotation, translation and scaling relationship between the combined scene coordinate system and the first sub-scene coordinate system.
[0086] Step S20: Calculate the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system corresponding to each key imaging interval
[0087] For any critical imaging interval Ω in Λ i ′, set Ω i 'Midpoint and the first display interval of the combined scene The mapping relationship of the inner points can be used to obtain the rotation and translation transformation relationship between the coordinate systems of the first sub-scene and the combined scene. Among them, the mapping relationship or the rotation and translation transformation relationship between the coordinate systems of the first sub-scene and the combined scene needs to satisfy the first display interval constraint. The constraint of the first display interval is specifically that the three-dimensional interval of the combined scene obtained after the three-dimensional imaging interval is transformed by the coordinate system rotation, translation and scaling relationship is included in the first display interval or equal to the first display interval. Figure 4 For example, the first sub-scene s 0 In the combined scene Display range With s 0 The first imaging interval Ω 0 Similarly, in this example, it is preferred to let Ω 0 After translation, rotation and scaling Align, so that the first imaging interval boundary vertices A, B, C, D, E, F, G and the first display interval boundary vertices A', B', C', D', E', F', G' are respectively located at similar interval boundary positions, and points at similar positions are set to have a one-to-one corresponding mapping relationship. The first sub-scene s can be solved by the coordinate value mapping relationship of these points. 0 The rotation, translation and scaling relationship H between the coordinate system and the combined scene coordinate system 0 The specific calculation method for obtaining the coordinate system transformation relationship from the mapping relationship between points is a common technical means and will not be repeated here.
[0088] Step S30, search for the optimal path
[0089] Any ordering of all key imaging intervals in the set Λ All of them constitute an imaging interval change path. i are all key imaging intervals in Λ, and the optimal path is searched therein. There are many ways to set the optimization target. In the embodiment of the present invention, the optimal path is set as the path with the shortest length. The distance between any two points can be defined in many ways in mathematics, such as Euclidean distance and Chevsheff distance, so there are also many ways to define the path length. In the embodiment of the present invention, the path length is set as follows: the length of the path Γ Among them, in r i In the rotation, translation and scaling relationship between the corresponding combined scene coordinate system and the first sub-scene coordinate system, the translation amount is [x i y i z i ], the rotation amount is Scaling size is σ i ′,r i-1 The corresponding translation is [x i-1 y i-1 z i-1], the rotation amount is Scaling size is σ i ' 1 , r i With r i-1 The distance between
[0090] , where λ 1 With λ 2 is a pre-set weight. This optimization problem:
[0091] Optimization goal: Constraints: 1)r i ∈Λ, where i=1,2,3,…,N 1
[0092] 2) The union of all key imaging intervals in Γ It must cover all the first sub-scene information that needs to be presented in the combined scene.
[0093] There are N in the set Λ 1 +1 key imaging interval, the number of paths it constitutes is (N 1 +1)! , exhaustively enumerate all paths and calculate the length of each path, and the path with the shortest length is the optimal path to be found. You can also adopt a certain search strategy to use any imaging interval Ω in Λ 0,0 As the starting point, select the distance Ω in Λ 0,0 The most recent imaging interval Ω 1,0 , construct the sequence Γ′=<Ω 0,0 Ω 1,0 >, find the imaging interval Ω in Λ except the imaging interval in Γ′ and the front imaging interval Ω in Γ′ 0,0 The imaging interval with the shortest distance Ω′ 0,0 , and also find the imaging interval Ω in Λ except the imaging interval in Γ′ and the end of Γ′ 1,0 The minimum imaging interval Ω 1 ' ,0 , if Ω 0,0 and Ω′ 0,0 Distance less than Ω 1,0 With Ω 1 ' ,0 The distance between 0,0 Add to the front of the sequence Γ′, otherwise add Ω 1 ' ,0is the end of Γ′, and so on. For the imaging intervals in Λ that are not added to Γ′, the imaging intervals closest to the first or last imaging interval in Γ′ are selected, and then the imaging interval with the smallest distance value is selected and added to the sequence. If the newly added imaging interval has the smallest distance to the first imaging interval in the original sequence, the newly added imaging interval is ranked at the front of the sequence Γ′, otherwise it is ranked at the back. After adding all the imaging intervals in Λ to Γ′, the final Γ′ is not necessarily the optimal solution, but a suboptimal solution. Compared with the exhaustive calculation method, the amount of calculation is greatly reduced.
[0094] by Figure 5 For example, for the sake of simplicity, sub-scene s 0 The three-dimensional space is divided into cubes of size ε×ε×ε, and the length, width, and height of each cube space obtained by the division are further specified to correspond to the sub-scene s 0 The x, y, and z axes of the rectangular coordinate system are parallel, wherein each cubic space obtained by segmentation is considered as a voxel in the embodiment of the present invention. For the sake of convenience, s is specified as 0 All the information that needs to be presented in the combined scene has a voxel z coordinate of 0. Please note that the above setting is only for the convenience of description and is not a combined part of the method of the present invention. Figure 5 In this example, the information carriers with a scale of 1 shown in (a) are voxels a, b, c, d, and the information carriers with a scale of 2 shown in (b) are voxels e, f, g, h, i, j. Figure 5 In the example, the combined scene presents sub-scenes s 0 The display interval size is a cube of 2ε×2ε×2ε, so the imaging interval template M corresponding to the information carrier with information scale 1 1 is: a cube of size 2ε×2ε×2ε; the imaging interval template M corresponding to the information carrier with information scale 2 2 is: a cube of size 4ε×4ε×4ε.
[0095] like Figure 5 As shown in (c), the sub-scene key imaging interval selection strategy 1 of step S10 is adopted, and the imaging interval template M is used. 1 The sub-scenes s are evenly divided at intervals of 2ε on the x-axis, y-axis, and z-axis. 0 After the space is divided, the voxels a, b, c, and d with information are respectively in the imaging intervals A, B, C, and D obtained after uniform division. The imaging intervals A, B, C, and D are set as key imaging intervals, such as Figure 5 As shown in (d), the scene information contained in the imaging intervals C and D can be completely covered by the imaging interval C′, so the key imaging intervals C and D are cancelled, and the imaging interval C′ is supplemented as the key imaging interval; the imaging interval template M is used 2The sub-scenes s are evenly divided at intervals of 4ε on the x-axis, y-axis, and z-axis. 0 After space, voxels e and f with information of scale 2 are in the imaging interval E, and voxels g, h, i, and j are in the imaging interval F. Therefore, the set of candidate key imaging intervals is Λ = {A, B, C′, E, F}.
[0096] Changes 0 The display interval within the combined scene will change the key imaging interval. 0 The first display interval is modified to a cube of size ε×ε×ε, and the imaging interval template M corresponding to the scene information of scale 1 1 ′ is: a cube of size ε×ε×ε, and the imaging interval template M corresponding to the scene information of scale 2 2 ' is a cube of size 2ε×2ε×2ε. Figure 5 The scene represented by (a) and (b) 0 For example, information such as Figure 6 As shown in Figure (a), the imaging interval template M 1 ′ Divide the sub-scenes s evenly at intervals of ε on the x-axis, y-axis, and z-axis 0 After the space is transformed, the voxels a, b, c, and d with information are in the imaging intervals A, B, C, and D respectively. Figure 6 As shown in Figure (b), the imaging interval template M 2 ′Evenly divide the sub-scene s at intervals of 2ε on the x-axis, y-axis, and z-axis 0 After the space is formed, the voxels e and f with information are in the imaging intervals E and F respectively, the voxels g and h are in the imaging interval G, and the voxels i and j are in the imaging interval H, such as Figure 6 As shown in Figure (c), the information contained in imaging intervals E and F can be completely contained in imaging interval E', so imaging intervals E and F are deleted and key imaging interval E' is added. Therefore, the set of key imaging intervals is Λ = {A, B, C, D, E', G, H}.
[0097] exist Figure 5 , Figure 6 In this example, there is no angular rotation between the key imaging intervals, so the distance between the imaging intervals is the Euclidean distance. Figure 5 The optimal (shortest) path found is Γ=<E,A,B,C′,F> , Figure 6 The optimal (shortest) path found is Γ=<E′,A,B,C,D,H,G> .
[0098] Step S40: The sub-scenes generate the first sub-scene experience screen and the experience screens of other scene contents of the combined scene according to the order of each key imaging interval on the path, the dwell time, and the calculated imaging pupil distance and posture, and synthesize the combined scene experience screen.
[0099] At any time t 0 , combined scene The first sub-scene s is presented in 0 For the path obtained after optimizing in step S30 Any key imaging interval r i The residence time is Δt i Indicates that Δt i Equal to the human eye's complete perception r i The time required to present the information in the combined scene, so in the time interval First sub-scene 0 The imaging interval stays at point r i In step S10, the time required for human eyes to perceive the corresponding scale information contained in each information carrier is marked. For any key imaging interval r i , which may contain multiple information carriers, Δt i You can choose to equal r i The sum of the time required for the human eye to perceive the corresponding scale information of all information carriers. Since humans have strong parallel information processing capabilities, Δt i You can also choose to equal r i The maximum value of the time required for the human eye to perceive the corresponding scale information of all information carriers in the image. Furthermore, since the same information carrier may be in multiple key imaging intervals, for any key imaging interval where the information carrier is located, the time required to perceive the information of the incomplete information carrier in this imaging interval can be the time duration for perceiving all the information of the information carrier, or can be set to an average of this time duration.
[0100] Let at any time t k , the first user p in the combined scene The lower posture is [W k Θ k ]、Pupillary distance is By k The above time interval can be used to determine s 0 At this time, the imaging interval corresponding to the path Γ is represented by r j Indicates that the corresponding first sub-scene s 0 Coordinate system Ψ 0 With combined scenes Coordinate system The rotation, translation and scaling relationship between them is H j , by Hj It can be solved that at this moment, the first user p is in s 0 Coordinate system Ψ 0 The pose under is [W 0,k Θ 0,k ] and pupil distance d 0,k .in, λ j H j The scaling factor in .
[0101] The first sub-scene is in the binocular camera pose [W 0,k Θ 0,k ] and pupil distance d 0,k Under the imaging parameters of 0 Imaging interval r j Imaging, generating a binocular stereoscopic visual experience picture of the first user p, the generated binocular stereoscopic visual experience picture and the binocular stereoscopic visual experience picture of other contents of the combined scene are synthesized to obtain a complete binocular stereoscopic visual experience picture of the combined scene. 0 For specific methods of synthesizing the experience screen with other content experience screens of the combined scene, please refer to references [2-6].
[0102] Note: In the embodiment of the present invention, the display interval of the combined scene used to present the first sub-scene is preferably fixed, but the display interval of the combined scene used to present the first sub-scene can also be dynamically adjusted, for example: real-time statistics of idle three-dimensional intervals without scene content in the combined scene are obtained, and a completely idle three-dimensional interval or a three-dimensional interval with as many idle intervals as possible that meets specific size and shape requirements is determined in real time, and the interval is adjusted in real time as the display interval of the first sub-scene.
[0103] In the embodiment of the present invention, the combined scene presents the first sub-scene and also presents the sound information of the first sub-scene. All the sound information of the first sub-scene can be transmitted to the combined scene and played. For the first sub-scene, only the sound emitted by the object in the imaging interval needs to be transmitted to the combined scene and played. Thus, when the first sub-scene automatically adjusts the imaging interval, the sound source transmitted and played by the first sub-scene to the combined scene will change accordingly. The imaging interval is a three-dimensional interval used to constrain the first sub-scene to present the scene content in the combined scene. In addition, any sound of the first sub-scene, the object emitting this sound is used to i Indicates that the object o i For this sound source, according to the rotation transformation relationship between the first sub-scene and the combined scene coordinate system, we get the object o i The position or posture presented in the combined scene. If this sound is played in the combined scene, the position or posture of the sound source is set to the object o iThe position or posture presented in the combined scene can generate stereo sound according to the position or posture. When the first user points to the person or robot in the first sub-scene through sight in the combined scene, and / or the first user is associated with the person or robot in the first sub-scene through voice or other actions, the combined scene determines that the first user is communicating with the person or robot in the first sub-scene by voice, then the combined scene sends the first user's voice or the associated sound in the combined scene to the first sub-scene for playback, or directly sends the first user's voice or the associated sound in the combined scene to the person or robot in the first sub-scene.
[0104] Invention Example 2
[0105] In the first embodiment of the invention, in the process of automatically adjusting the presentation of the first sub-scene in the combined scene, the switching of the key imaging interval is realized by jumping. In the embodiment of the present invention, the imaging interval realizes the switching of the key imaging interval by continuously changing. In the embodiment of the present invention, the imaging intervals are limited to maintain similarity during the change process, and the continuous change of the imaging interval is realized by continuously changing the position, attitude angle and scaling size of the imaging interval. Any key imaging interval r i , the time required for complete perception of information is Δt i It means that when the imaging interval is in the key imaging interval r i The residence time reaches Δt i After 0 The imaging interval is continuously changed from r i Move to r i+1 , where it stays in the key imaging interval r i When the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system is i y i z i ], the attitude angle is expressed by Indicates that the scaling size is σ i ' means; stay in the key imaging interval r i+1 When, in the rotation-translation-scaling relationship, the translation amount is [x i+1 y i+1 z i+1 ], the rotation amount is Indicates that the scaling size is σ i + 1 .s 0 The imaging interval is continuously changed from r i Move to r i+1 In the process, the translation amount of the rotation-translation-scaling relationship is calculated by i y i z i] and [x i+1 y i+1 z i+1 ] is interpolated, and the rotation amount is obtained by and The interpolation is obtained, and the scaling is done by i ′ and σ′ i+1 The interpolation is obtained. During the interpolation, the translation speed, rotation speed, and scaling speed can be further set to adjust the switching path and duration of the key imaging interval. Interpolation is a common technical means and will not be described in detail here.
[0106] Also, for the path Any key imaging interval r i , the time required for complete perception of information is Δt i It means that when the imaging interval changes continuously to r i The imaging interval is controlled to pass through r i The duration of i The information contained is fully perceived by the user.
[0107] Invention Example 3
[0108] In the first and second embodiments of the invention, although all the information in the sub-scene is presented in the combined scene by changing the imaging interval. However, the scene information is often reflected in the surface features of the object, and the surface of the object has an orientation. Since the orientation of the surface of the information carrier is not considered when selecting the key imaging interval and setting the mapping relationship between the points in the imaging interval and the points in the combined scene display interval in the first and second embodiments of the invention, it may cause the situation that when the scene information is presented in the combined scene display interval, the surface of the information carrier containing the information faces away from the user, and the user needs to move to a position where the scene information can be seen. In the process of automatic sub-scene presentation in the combined scene, the embodiment of the present invention needs to ensure that the sub-scene information points to the first user in the presentation direction of the combined scene, so that the first user can see the sub-scene information without moving the position. Note that this embodiment only ensures that the first user does not need to move, but other users who are in different positions from the first user may still need to move to see the sub-scene information.
[0109] Improvements are made to Invention Example 1 and Invention Example 2.
[0110] In step S00, the human eye-perceivable direction of the information is also marked;
[0111] In step S10, when selecting the key imaging interval, it is necessary to make the information contained in the key imaging interval present after the combined scene display interval, so that the first user can perceive it as much as possible. The first sub-scene s0 Key imaging interval Ω i ' is presented in the first display interval of the combined scene, o j is Ω i 'Information carrier contained, o j In the combined scene In the coordinate system, the information can be perceived by the human eye in the direction v j , o j The direction of the ray to the first user is v′ j , you need to make v j and v′ j As much as possible. j The ray to the first user is generally o j The ray from the center point or information attention point (the location point on the information carrier that the user pays most attention to) to the midpoint of the first user's binocular connection line. In addition, v j and v′ j As similar as possible, which can be to make v j and v′ j Equal, but directly make v j and v′ j Equality may lead to the combination of scenes, s 0 The ground plane of v is not parallel to the ground plane of other contents in the composite scene; therefore, j and v′ j As much as possible, you can also take the same 0 Under the constraint that the ground plane is parallel to the ground plane of other scene contents in the combined scene, v j and v′ j The angle between them is the smallest, and the smallest angle is the direction vector v j and v′ j When they intersect, their common surface is perpendicular to the ground plane.
[0112] Make the first sub-scene s 0 Key imaging interval Ω i ′ is presented in the first display interval of the combined scene, v j and v′ j The specific implementation of equality is: The direction vector from the center point to the midpoint of the first user's binocular connection line As v′ j , find the combined scene Coordinate system to subscenes 0 A rotation relation R′ of the coordinate system makes Switch to subscenes 0 After the coordinate system, with the direction vector v j Same or parallel, combined scene first display area After R′ rotation, we get s 0 Imaging interval Information carrier j The corresponding information scale is σ j ′ indicates that As the imaging area template, the method for selecting the key imaging interval of the first invention is adopted, wherein in the selection process, all imaging intervals are compared with There can be no rotation transformation between them, so we can get the value used to present o j The key imaging intervals are traversed through all information carriers in this way, and the key imaging intervals obtained from different information carriers are grouped together. In addition, the same method as in the first embodiment of the invention can be used to replace some key imaging intervals to reduce the number of key imaging intervals. In many cases, the same information carrier may have multiple surfaces with different aspects that contain information. This information carrier can be further divided into sub-information carriers, or the direction of a key surface can be used to represent the direction of the carrier information. Among them, the combined scene is obtained Coordinate system to subscenes 0 A rotation relation R′ of the coordinate system makes Switch to subscenes 0 Coordinate system and direction vector v j Same or parallel, specifically: in s 0 Coordinate system o j Center point or o j Information attention points and combination scenarios Coordinate system The center point establishes a mapping relationship, with s 0 Coordinate system o j Center point or o j The information attention point is the starting point and the direction is v j Generate a ray, on which the distance from o j Center point or o j A point a′ is selected at the unit distance of the information attention point, and the combined scene Coordinate system The center point is the starting point and the direction is v′ j Generate a ray on which A point b' is selected at a unit distance from the center point, and a mapping relationship between a' and b' is established. From this mapping relationship between the two sets of points, multiple s can be obtained. 0 Coordinate system and The rotation relationship of the coordinate system is obtained by a conventional calculation method and will not be described again. One of the rotation relationships is selected as the rotation relationship R′.
[0113] Under the constraint of being parallel to the ground plane, make the first sub-scene s 0 Key imaging interval r iAfter being presented in the first display interval of the combined scene, v j and v′ j The specific implementation of the common ground plane vertical plane in the case of intersection is: The direction vector from the center point to the midpoint of the first user's binocular connection line As v′ j , find the combined scene Coordinate system to subscenes 0 A rotation relation R′ of the coordinate system makes Switch to subscenes 0 After the coordinate system, with the direction vector v j Coplanar, this surface is perpendicular to the ground plane, the first display area of the combined scene After R' rotation, we get s 0 Imaging interval Information carrier j The corresponding information scale is σ j ′ indicates that As the imaging area template, the method for selecting the key imaging interval of the first invention is adopted, wherein in the selection process, all imaging intervals are There can be no rotation transformation between them, so we can get the value used to present o j , traverse all information carriers in this way, and collect the key imaging intervals obtained from different information carriers together. In addition, the same method as in the first embodiment of the invention can be used to replace some key imaging intervals to reduce the number of imaging intervals. In many cases, the same information carrier may have multiple different surfaces that contain information. This information carrier can be further divided into sub-information carriers, or the direction of a key surface can be used to represent the direction of the carrier information. Coordinate system to subscenes 0 A rotation relation R′ of the coordinate system makes Switch to subscenes 0 After the coordinate system, with the direction vector v j Coplanar, this plane is perpendicular to the ground plane. This solution process is a conventional geometric equation calculation and will not be described in detail here.
[0114] In this step, the selected key imaging interval is set These key imaging intervals not only cover all the information in the scene that needs to be presented in the combined scene, but also satisfy the requirement that the information is as perceptible as possible to the first user.
[0115] Step S20 is the same as that in the first embodiment of the invention.
[0116] Step S30: Optimization
[0117] Any sequence of all key imaging intervals in the set Λ They all constitute an imaging interval change path, and the path with the shortest length is found as the optimal path. The length of path Γ The specific characterization is the same as that of Invention Example 1. The optimization problem is as follows.
[0118] Optimization goal: Constraints: 1)r i ∈Λ, where i=1,2,3,…,N 1 ;
[0119] 2) The union of all key imaging intervals in Γ
[0120] by Figure 7 For example, in Figure 7 In (a1), the voxels a, b, c, d, and e of the first subscene contain information of scale 1. The size of each voxel is ε×ε×ε. The size of the first display interval of the combined scene is a cube of 2ε×2ε×2ε. The imaging interval template corresponding to scale 1 is: a cube of 2ε×2ε×2ε. If the human eye can perceive the direction of the information without considering the key imaging interval is as follows Figure 7 As shown in (a2), there are imaging intervals A, B, and C, and the shortest path is A→B→C. Figure 7 In (b1), the voxels a, b, c, d, and e in the first sub-scene are marked with the directions of information that can be perceived by the human eye. For example, the direction of voxel a is due east, and the user is to the east of voxel a and can best perceive the information contained in voxel a by looking from east to west. The direction of voxel b information that can be perceived by the human eye is west, and the user is to the west of voxel b and can best perceive the information contained in voxel b by looking from west to east. The direction of the imaging interval mapped to the display interval toward the user must be aligned with the perceptible direction of the information. Therefore, in Figure b2, the direction of imaging interval A toward the user is aligned to the due east, the direction of imaging interval B toward the user is aligned to the due west, and the direction of imaging interval C toward the user is aligned to the due east. The path length increases the factor of the change in attitude angle. In the path A→B→C, A→B changes the attitude angle once, and B→C changes the attitude angle again, while the path A→C→B only changes the attitude angle once in C→B, and the change amount is the same as the change amount of B→C in the path A→B→C. Therefore, when the weight of the attitude angle change amount in the distance metric is large enough, the optimal path will be A→C→B. In addition, the perceptible directions of the human eye are different between different parts of the information contained in the same imaging interval. For example, in Figure 7In (c1), the information of voxels c and d is perceived in different directions by the human eye. Although imaging interval C can include voxels c and d at the same time, when imaging interval C is presented in the combined scene, the rotation relationship between the first sub-scene coordinate system and the combined scene coordinate system can be changed so that information of different directions that can be perceived by the human eye can be perceived by the user. Alternatively, multiple imaging intervals with overlapping positions but different posture angles can be used to ensure that information of different directions that can be perceived by the human eye can be perceived by the user. Therefore, Figure 7 In (c2), the imaging interval C and the imaging interval C′ overlap, but the posture angles are different, corresponding to different information about the directions that the human eye can perceive. The optimal path of the imaging interval is A→C→C′→B. Figure 7 In the example given in (d1), the information of voxel c and voxel d can be perceived by the human eye in the same direction. Figure 7 Similar to (a2), when imaging interval C is used to present the information of voxel c and voxel d in the combined scene at the same time, it may happen that voxel d occludes voxel c, and the user cannot perceive the information of voxel c, and the occlusion cannot be eliminated by adjusting the imaging interval attitude angle. In this case, either the scene content in voxel d is hidden or made transparent after the information of voxel d is presented in imaging interval C, or imaging interval C is decomposed into multiple imaging intervals. Figure 7 In (d2), the imaging interval C is decomposed into imaging intervals E and F, and the optimal path of the imaging intervals is A→E→F→B.
[0121] S40 is the same as in Example 1 or 2 of the invention.
[0122] Invention Example 4
[0123] In the embodiment of the present invention, the sub-scene s 0 It is a real-time dynamic scene, and its scene information needs to be presented in real time in the combined scene Inside.
[0124] The method for automatically constructing a combined scene according to an embodiment of the present invention comprises the following steps:
[0125] Step S00, obtaining scene dynamic information, and marking in real time the information of the first sub-scene that needs to be presented in the combined scene;
[0126] Step S10, selecting the key imaging interval of the first sub-scene in real time;
[0127] Step S20, calculating and obtaining the rotation, translation and scaling relationship between the first sub-scene coordinate system corresponding to the key imaging interval and the combined scene coordinate system;
[0128] Step S30, determining the key imaging interval of the first sub-scene in real time;
[0129] Step S40, calculate the user's posture and pupil distance in the first sub-scene in real time, so as to generate the first sub-scene experience screen and the experience screen of other scene contents of the combined scene according to the imaging interval, imaging pupil distance and posture of the first sub-scene, and synthesize the combined scene experience screen.
[0130] The specific implementation of each step is as follows.
[0131] Step S00, obtaining scene dynamic information, and annotating in real time the information of the first sub-scene that needs to be presented in the combined scene.
[0132] If the first sub-scene s 0 It is a pure virtual scene built by 3D engines such as unity3d. The engine records s 0 The state of each virtual object in the system. When the state of any object changes, the state change of this object generates dynamic information, and this object is the carrier of this dynamic information. 0 If it is a real scene or a virtual-real fusion scene, the real objects in the scene are detected and identified in real time. If the state of the real object changes, the state change of this object generates dynamic information, and this object is the carrier of this dynamic information.
[0133] For rigid objects, the three-dimensional space occupied and the direction perceptible to the human eye are calibrated in the coordinate system of the object. When the position and posture of the object changes, the position and posture change and the coordinate system of the object and the first sub-scene s 0 The rotation and translation relationship between coordinate systems can be calculated in real time to obtain the object in the sub-scene s 0 The three-dimensional space occupied and the direction of information that can be perceived by the human eye. For non-rigid objects, if they are virtual objects, the three-dimensional engine records the deformation of the surface mesh in real time. From this surface mesh information, the three-dimensional space occupied by the object and the direction of information that can be perceived by the human eye can be calculated. If the object is a real object, the three-dimensional space occupied by the object and the direction of information that can be perceived by the human eye can be obtained through three-dimensional reconstruction.
[0134] Information and carriers that need to be presented in the combined scene are selected from the dynamic information. The selection method can be statistical selection or artificial intelligence selection, or it can be selected according to artificial preset rules. Among them, statistical selection is to count the dynamic information that the user is concerned about. When the dynamic information that the user is concerned about is generated, this dynamic information is selected to be presented to the user in the combined scene, or artificial intelligence, especially general artificial intelligence, independently determines the real-time dynamic information that needs to be presented to the user based on the content of the scene and the characteristics of the user. The artificial preset rules are to pre-set which dynamic information needs to be presented, or which dynamic information needs to be presented under which conditions. After determining the information that needs to be presented, a set of key imaging intervals can be generated in accordance with the method of embodiment 1 of the invention.
[0135] Steps S10 and S20 are the same as those in Embodiment 1 of the present invention.
[0136] Step S30: Determine the first sub-scene imaging interval in real time
[0137] Since sub-scene s 0 The information in the video is changing in real time. The real-time dynamic information is lost in a moment. When the limited display area of S cannot fully display the video, 0 When all real-time dynamic information is available, a trade-off needs to be made. The trade-off rule can be one or more of first-come, first-served, rotation, proximity, priority, or information volume competition. 0 The real-time dynamic information of the first object is being presented in the combined scene. In the following time, if the action of the first object is not finished, the real-time dynamic information of the first object remains presented in the combined scene S. The said rotation means that when the sub-scene s 0 After the real-time dynamic information of the first object is being presented in the combined scene and has been presented for a certain time threshold, the first object is presented first in the combined scene S. 0 Dynamic information of other objects; the near means giving priority to presenting dynamic information close to the imaging interval at the previous moment in real time, and the imaging interval refers to s 0 The priority or information content competition refers to the priority presentation of the dynamic information with the largest priority or the largest amount of information. 0 The dynamic information is presented in real time in the combined scene S, and the non-dynamic information associated with the dynamic information can also be presented in real time in the combined scene, by pre-setting the dynamic information association rules, for example: for a person object, it can be set that if the person object moves, the scene information of a certain range around the person object or a certain range in front of the person object needs to be presented in the combined scene. Similarly, the priority or information volume of the object can be pre-calibrated.
[0138] by Figure 8 For example, Figures (a), (b), (c), and (d) are the scene states of the first sub-scene at four consecutive moments. Among them, from the first to the third moment, there is a car object moving in the sub-scene, generating dynamic information. At the fourth moment, the car object stops moving and no dynamic information is generated. From the second to the fourth moment, there is a person object moving in the sub-scene, generating dynamic information. At the first moment, the candidate imaging interval is only A, at the second moment, the candidate imaging intervals are B and D, at the third moment, the candidate imaging intervals are C and E, and at the fourth moment, the candidate imaging interval is only F. It can be seen that at the second and third moments, it is necessary to make a choice among multiple imaging intervals.
[0139] exist Fig. 9 In this paper, four different selection strategies are given. Figure 8 Imaging interval adjustment path for the given example. Fig. 9 (a1) and Fig. 9 (a2) shows an example of the imaging interval adjustment path under the first-come-first-served selection strategy. In this example, since there is only A among the candidate imaging intervals at the first moment, the system selects s at the first moment. 0 The imaging interval is A, and the candidate imaging intervals at the second moment are B and D. Among them, the scene content in the imaging interval B is similar to the scene content in the previous moment. 0 Imaging interval A is the same, so the system selects imaging interval B as the imaging interval at the second moment. Similarly, imaging interval C is selected at the third moment. At the fourth moment, there is only imaging interval F, so imaging interval F is selected. Therefore, the path selected under this strategy is A→B→C→F. Fig. 9 (b1) and Fig. 9 (b2) gives an example of the imaging interval adjustment path under the rotation strategy. When the sub-scene s 0 After the dynamic information of any object is continuously presented in the combined scene for more than 2 moments, if there is dynamic information of other objects in the scene, the presentation of the dynamic information of other objects is given priority. Therefore, interval A is selected at the first moment. At the second moment, the scene object in interval B is the same as that in interval A, and the dynamic information of this object is presented in the combined scene for no more than 2 moments, so interval B is selected at the second moment. At the third moment, the scene object in interval C is the same as that in interval B, but this scene object has been presented in the combined scene for 2 moments continuously, so interval E is selected at the third moment, and interval F is selected at the fourth moment. Therefore, the path selected under this strategy is A→B→E→F. Fig. 9 (c1) and Fig. 9 (c2) shows an example of imaging interval adjustment under the priority or information competition strategy. The system sets the priority of dynamic information of personnel objects to be higher than the priority of dynamic information of vehicles. Therefore, at the second and third moments, when dynamic information of personnel appears in the sub-scene, the system automatically selects imaging intervals D and E respectively. Therefore, the path selected under this strategy is A→D→E→F. Fig. 9 (d1) and Fig. 9 (d2) gives an example of the proximity strategy. After imaging interval A is selected at the first moment, the imaging interval closest to A at the second moment is B, and the imaging interval closest to B at the third moment is C. Therefore, the path selected under this strategy is A→B→C→F.
[0140] Therefore, when the number of key imaging intervals generated in real time in step S10 is 0 (for example, when there is no dynamic information), the selected imaging interval can be any imaging interval or an imaging interval at the previous moment; if the number of key imaging intervals generated is 1, then this key imaging interval is determined as the imaging interval that needs to be presented in real time in the combined scene; when the number of key imaging intervals generated is greater than 1, then according to the above-mentioned selection strategy, one of the key imaging intervals is selected to be determined as the imaging interval that needs to be presented in real time in the combined scene.
[0141] Step S40: Calculate the user's posture and pupil distance in the first sub-scene in real time, so as to generate the first sub-scene experience screen and the experience screen of other scene contents of the combined scene according to the imaging interval, imaging pupil distance and posture of the first sub-scene, and synthesize the combined scene experience screen
[0142] Let at any time t k , the first user p in the combined scene The lower posture is [W k Θ k ]、Pupillary distance is The imaging interval determined in real time in step S30 is Ω k , Ω calculated in step S20 k Corresponding to the first sub-scene s 0 The rotation, translation and scaling relationship H between the coordinate system and the combined scene coordinate system k , by H k It can be solved that at time t k , the first user p in s 0 Coordinate system Ψ 0 The pose under is [W 0,k Θ 0,k ] and pupil distance d 0,k .in, λ k H k The scaling factor in .
[0143] The first sub-scene is in the binocular camera pose [W 0,k Θ 0,k ] and pupil distance d 0,k Under the imaging parameters of 0 Imaging interval Ω k Imaging, generating a binocular stereoscopic visual experience picture of the first user p, the generated binocular stereoscopic visual experience picture and the binocular stereoscopic visual experience picture of other contents of the combined scene are synthesized to obtain a complete binocular stereoscopic visual experience picture of the combined scene. 0 For specific methods of synthesizing the experience screen with other content experience screens of the combined scene, please refer to references [2-6].
[0144] Invention Example 5
[0145] In the fourth embodiment of the invention, there is no transition when the sub-scene imaging interval is adjusted in real time, and the change is made directly. In the embodiment of the present invention, the sub-scene imaging interval is adjusted in real time by continuous change. Based on the fourth embodiment of the invention, an improvement is made. In step S30, the imaging interval selected according to the strategy is only used as the destination of the imaging interval adjustment. Let at any time t k The imaging interval selected in step S30 is Ω k , at the previous moment t k-1 , the imaging interval is Ω′ k-1 , with Ω′ k-1 is the starting point, k The end point is the time t at a certain imaging interval position, rotation angle, and a certain change rate of the zoom scale. k The actual imaging interval. For example: at time t k-1 , taking the imaging interval as an object that can be rotated, translated and scaled, the first sub-scene imaging interval Ω′ k-1 The position is [x k-1 y k-1 z k-1 ], the attitude angle is Scaling size is σ′ k-1 , at time t k , destination imaging interval Ω k Location is The attitude angle is Scale to The x-axis moving speed set by the system is Δx, the absolute value of the x-axis moving amount at each time interval. k ,when Then the first sub-scene imaging interval Ω set by the system k The coordinate value of the position of ′ on the x-axis is Otherwise Similarly, according to the set attitude angle change speed, scale change speed and the moving speed of other coordinate axes, the first sub-scene imaging interval Ω set by the system can be obtained. k The position of ′ on the y-axis and z-axis, as well as the values of the attitude angle and the scaling scale. Set the first sub-scene imaging interval Ω k ′The mapping relationship between the inner point and the inner point of the first display interval of the combined scene is calculated, and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system is calculated, thereby calculating the pupil distance and posture of the user in the first sub-scene, and the first sub-scene generates a first sub-scene experience screen according to the first sub-scene imaging interval and the imaging pupil distance and posture. The experience screen and the experience screen of other scene contents of the combined scene are synthesized to obtain the combined scene experience screen.
[0146] Invention Example 6
[0147] In the embodiment of the present invention, the sub-scene s 0 It is a dynamic non-real-time scene, that is, the dynamic information in the scene is recorded. Referring to the embodiments 1-5 of the invention, the key imaging interval set of the static information of the scene can be obtained. The duration of dynamic scene change is t 0 arrive At any time t k , the key imaging interval of dynamic information is
[0148] gather Any sequence of all key imaging intervals in They all constitute an imaging interval change path, but in order to avoid time confusion, constraints need to be added: for any two key imaging intervals r in Γ i With r j , if r i With r j Both contain scene dynamic information, where i<j, r i With r j The dynamic information generated at the time t i With t j , then there must be t i <t j . Find the shortest path as the optimal path. The length of path Γ The specific characteristics are the same as those in Invention Example 1. This optimization problem:
[0149] Optimization goal: Constraints:
[0150] 1) Where i = 1, 2, 3, ..., N 3
[0151] 2) Dynamic information key imaging interval order preservation: Any two key imaging intervals r in Γ i With r j , if r i With r j Both contain scene dynamic information, where i<j, r i With r j The dynamic information generated at the time t i With t j , then there must be t i <t j ;
[0152] 3) The union of all key imaging intervals in Γ It must cover the static information of all sub-scenes that need to present the combined scene, and as much dynamic information as possible, and / or satisfy the perception of the information to the first user. The dynamic information as much as possible can use the method of selecting key imaging intervals of real-time dynamic scenes in Example 4 of the invention to eliminate the unselected key imaging intervals containing dynamic information so that they do not appear in the generated path.
[0153] Exhaust all paths and calculate the length of each path. The path with the shortest length is the optimal path to be found. You can also adopt a certain search strategy to use any imaging interval Ω in Λ 0,0 As the starting point, select Λ and Λ 0 Medium distance Ω 0,0 The most recent imaging interval Ω 1,0 , construct the sequence Γ=<Ω 0,0 Ω 1,0 >, if Ω 1,0 ∈Λ, find Λ 0 and the imaging interval in Λ except for the imaging interval in Γ′ and the front imaging interval Ω in Γ′ 0,0 The imaging interval with the shortest distance Ω′ 0,0 , and also look for Λ 0 and the imaging interval Ω at the end of Γ except the imaging interval in Γ in Λ 1,0 The minimum imaging interval Ω 1 ' ,0 , if Ω 0,0 and Ω′ 0,0 Distance less than Ω 1,0 and Ω′ 1,0 The distance between 0,0 Add to the front of the sequence Γ, otherwise add Ω′ 1,0 is the end of Γ, if Ω 1,0 ∈Λ 0 , find Λ 1 and the first imaging interval Ω in Γ′ except the imaging interval in Γ′ in Λ 0,0 The imaging interval with the shortest distance Ω′ 0,0 , and also look for Λ 1 and the imaging interval Ω at the end of Γ′ except the imaging interval in Λ 1,0 The minimum imaging interval Ω 1 ' ,0 , if Ω 0,0 and Ω′ 0,0 Distance less than Ω 1,0 and Ω′ 1,0 The distance between 0,0 Add to the front of the sequence Γ′, otherwise add Ω′ 1,0 is the end of Γ′, and so on. iThere are dynamic information key imaging intervals at all times and one of the key imaging intervals has been added to the sequence Γ′, t i The next critical imaging interval with dynamic information after time is t j and there are no critical imaging intervals added to the sequence Γ′, imaging intervals not added to Γ in Λ, and j In the union of the key imaging intervals in Γ, the key imaging intervals closest to the front or the last imaging interval in Γ are selected respectively, and then the imaging interval with the smallest distance value is selected and added to the sequence. If the newly added imaging interval has the smallest distance to the front imaging interval of the original sequence, the newly added imaging interval is ranked at the front of the sequence Γ′, otherwise it is ranked at the back. The final Γ is not necessarily the optimal solution, but a suboptimal solution. Compared with the exhaustive calculation method, the amount of calculation is greatly reduced. In the obtained Γ, there may be dynamic information key imaging intervals that also contain scene static information, so the static information key imaging intervals are traversed. If its information is already included in the dynamic information key imaging interval, this key imaging interval is removed from Γ.
[0154] After the key imaging interval is obtained, the generation of the first sub-scene experience picture and the synthesis of the combined scene picture are the same as in the embodiments 1-5 of the invention.
[0155] Invention Example 7
[0156] Embodiment 6 of the invention avoids time confusion, but cannot avoid the fact that dynamic scene information cannot be fully presented in the combined scene.
[0157] In the embodiment of the present invention, the order-preserving constraint of the dynamic information key imaging intervals is removed, but a time reversal penalty component is added to the distance metric between the dynamic information key imaging intervals.
[0158] r i With r j are two dynamic information key imaging intervals on the imaging interval change path, where r i In the rotation, translation and scaling relationship between the corresponding combined scene coordinate system and the first sub-scene coordinate system, the translation amount is [x i y i z i ], the rotation amount is Scaling size is σ i ′,r j The corresponding translation is [x j y j z j ], the rotation amount is Scaling size is σ j ′, if t i >t j , from r iMove to r j The distance is Among them, λ, λ 2 , 3 is a pre-set weight. This optimization problem is to find Meet the optimization goal: Constraints:
[0159] 1) Where i = 1, 2, 3, ..., N 4
[0160] 2) The union of all key imaging intervals in Γ It must cover the static and dynamic information of all sub-scenes that need to be presented in the combined scene and / or meet the directional requirements that the first user of the information can perceive.
[0161] Exhaust all paths and calculate the length of each path. The path with the shortest length is the optimal path to be found. You can also adopt a certain search strategy, let Take any imaging interval Ω in Λ 0,0 As a starting point, select Medium distance Ω 0,0 The most recent imaging interval Ω 1,0 , construct the sequence Γ=<Ω 0,0 Ω 1,0 >, also looking for Except for the imaging interval in Γ and the imaging interval at the end of Γ, Ω 1,0 The imaging interval with the shortest distance Ω′ 1,0 , if Ω 0,0 and Ω′ 0,0 Distance less than Ω 1,0 and Ω′ 1,0 The distance between 0,0 Add to the front of the sequence Γ, otherwise add Ω 1 ' ,0 is the end of Γ, and so on. For the imaging intervals that are not added to Γ, the key imaging intervals that are closest to the front or the last imaging interval in Γ are selected respectively, and then the imaging interval with the smallest distance value is selected and added to the sequence. If the newly added imaging interval has the smallest distance to the front imaging interval of the original sequence, the newly added imaging interval is ranked at the front of the sequence Γ, otherwise it is ranked at the back. After adding all the imaging intervals in Γ to Γ, the final Γ is not necessarily the optimal solution. In the obtained Γ, there may be dynamic information key imaging intervals that also contain scene static information. Therefore, the static information key imaging intervals are traversed. If its information is already included in the dynamic information key imaging interval, this key imaging interval is removed from Γ.
[0162] After the key imaging interval is obtained, the generation of the first sub-scene experience picture and the synthesis of the combined scene picture are the same as in the embodiments 1-5 of the invention.
[0163] Invention Examples 8-14
[0164] In the invention embodiments 1-7, the first sub-scene experience screen and the experience screens of other scene contents of the combined scene are generated, and the combined scene experience screen is synthesized. The invention embodiments 1-7 are modified to obtain the invention embodiments 8-14 respectively, and the modification content is: the first sub-scene generates the scene three-dimensional information and loads it into the combined scene and then renders to obtain the combined scene experience screen, specifically, the first sub-scene generates the three-dimensional information of the scene content in the imaging interval in real time (the scene three-dimensional information can be represented by voxels, point clouds, etc.), and the generated first sub-scene s 0 The three-dimensional information is transmitted to the combined scene in real time, and the combined scene S is loaded in the first display interval. 0 3D information, real-time rendering loaded with s 0 The combined scene experience screen of the three-dimensional information generates a combined scene interactive experience for the user. Taking the improvement of the real-time example 1 of the invention as an example, step 40 is changed to: the first sub-scene generates the scene three-dimensional information in the imaging interval of the first sub-scene according to the order and residence time of each key imaging interval on the path, and the scene three-dimensional information is loaded into the combined scene The position and size of the inner S are transformed according to the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system calculated in step S20, and then the combined scene Render the user experience screen so that users can get the content 0 For the specific implementation of loading sub-scene 3D information in real time in a combined scene, please refer to the process of loading teaching content scenes in the classroom scene in paragraphs 0088 to 0112 of the specification of the invention patent application
[10] .
[0165] Invention Examples 15-21
[0166] In the embodiments 1-7 of the invention, the first sub-scene is presented in the first display interval of the combined scene, and the first display interval is a three-dimensional interval defined in the coordinate system of the combined scene. The embodiments 1-7 of the invention are modified to obtain the embodiments 15-21 of the invention respectively, and the modification content is: the first sub-scene is presented in the first display interval of the combined scene, and the first display interval is defined as a three-dimensional interval in the coordinate system of the first user field of view. Taking the modification of the embodiment 1 of the invention as an example, steps S00, S10, and S30 remain unchanged. Step S20 is changed to: calculate the rotation, translation, and scaling relationship between the first sub-scene coordinate system corresponding to each key imaging interval on the path and the first user field of view coordinate system. Among them, the mapping relationship between the points in the imaging interval of the first sub-scene and the points in the first display interval in the first user field of view is set, and the rotation, translation, and scaling relationship between the first sub-scene coordinate system and the first user field of view coordinate system can be obtained from this mapping relationship, and the first display interval is used to present the first sub-scene. In step S40, let at any time t k , the pose of the first user p in its field of view coordinates is [W k Θ k ]、Pupillary distance is The above-mentioned posture and pupil distance are preferably fixed. Referring to the first embodiment of the invention, for any time t k , by t k The time interval to which it belongs can be determined 0 At this time, the imaging interval change path Γ corresponds to the imaging interval, and this imaging interval is represented by r j Indicates that the corresponding first sub-scene s 0 Coordinate system Ψ 0 The rotation, translation and scaling relationship between the coordinate system of the first user field of view is H j , by H j It can be solved that at time t k , the first user p in s 0 Coordinate system ψ 0 The pose under is [W 0,k Θ 0,k ] and pupil distance d 0,k .in, λ j H j The first sub-scene is in the stereo camera pose [W 0,k Θ 0,k ] and pupil distance d 0,k Under the imaging parameters of 0 Imaging interval Ω j Imaging is performed to generate a binocular stereoscopic vision picture of the first sub-scene, and the generated binocular stereoscopic vision picture is synthesized with the binocular stereoscopic vision experience picture of other contents of the combined scene to obtain a complete binocular stereoscopic vision experience picture of the combined scene. 0For specific methods of generating the experience screen and synthesizing it with other content experience screens of the first user, please refer to references [7-8]. 0 The binocular stereo vision image corresponds to the live broadcast image in the literature [7-8].
[0167] In the embodiment of the present invention, when the combined scene presents the sound information of the first sub-scene, for any sound of the first sub-scene, the object that emits the sound is represented by o i Indicates that, according to the rotation transformation relationship between the first sub-scene and the first user coordinate system, the object o is obtained. i The pose presented in the first user field of view coordinate system can be obtained from the rotation and translation relationship between the first user field of view coordinate system and the combined scene coordinate system. i The corresponding position in the combined scene coordinate system. If this sound is played in the combined scene, set the position or position of the sound source to the object o i The position or posture presented in the combined scene, based on which stereo sound can be generated.
[0168] Inventive Examples 22-28
[0169] In the embodiments 15-21 of the invention, the first sub-scene generates a picture under the imaging interval constraint, and the generated first sub-scene picture is synthesized with the pictures of other contents in the combined scene S to obtain the combined scene experience picture. The embodiments 15-21 of the invention are modified to obtain the embodiments 22-28 of the invention respectively, and the modified contents are as follows: the picture generated by the first sub-scene without the imaging interval constraint is cropped under the imaging interval constraint and then synthesized with the pictures of other scene contents in the combined scene S to obtain the combined scene experience picture, wherein cropping under the imaging interval constraint means that the pixels in the experience picture image that image the scene contents in the imaging interval are retained, otherwise they are discarded. For the cropping of sub-scene pictures and the synthesis of combined scene experience pictures, refer to the document [9], which will not be elaborated here.
[0170] Inventive Examples 29-35
[0171] In the invention embodiments 1-7, the first sub-scene and other contents in the combined scene are separately rendered into experience screens, and then the experience screens are processed and synthesized to obtain the combined scene experience screen. The invention embodiments 1-7 are modified to obtain the invention embodiments 29-35 respectively, and the modification content is: the first sub-scene and other contents of the combined scene are directly rendered to generate the combined scene screen, specifically, the first sub-scene and other contents of the combined scene are loaded into the same scene instance, and the combined scene instance containing all the sub-scenes is directly rendered to obtain the combined scene experience screen. For specific implementation, see paragraph 0094 of the specification of reference [1].
[0172] Note that in the above embodiments, the optimization goal may be one or more of the following: the path length is short, the maximum information of the first sub-scene is presented in the combined scene after the accumulation in time sequence, the maximum information of the first sub-scene can be perceived by the first user, the non-real-time dynamic information of the first sub-scene is the least in reverse order, and the time required for the first user to perceive all the information of the first sub-scene in the combined scene is the shortest; the constraint condition may be one or more of the following constraint conditions: all static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene are presented in the combined scene after the accumulation in time sequence, all static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene can be perceived by the first user in the combined scene without the first user moving, The rate of change of the rotation relationship in the rotation, translation and scaling relationship between the sub-scene coordinate system and the combined scene coordinate system or the first user's field of view coordinate system does not exceed a specific threshold, the rate of change of the translation relationship in the rotation, translation and scaling relationship does not exceed a specific threshold, the rate of change of the scaling relationship in the rotation, translation and scaling relationship does not exceed a specific threshold, the time for the imaging interval to pass through or stay in the scene content and / or scene information meets the time requirement for the first user or all users to fully perceive the scene content and / or scene information, and the dynamic information time is not in reverse order; the strategy can be that when the first display interval of the combined scene cannot fully present all the real-time dynamic information and / or scene content of the first sub-scene, it is necessary to make a trade-off, and the trade-off strategy is one or more of first-come, first-served, rotation, proximity, priority or information competition. Under certain optimization goals and constraints or strategies, the calculation process of obtaining the optimization goal is a common technical means, which will not be repeated here.
[0173] The beneficial effect of the method for automatically combining extended reality (XR) combined scenes of the present invention is as follows: the presentation of sub-scenes in the combined scene is automatically adjusted so that the presentation of sub-scenes in the combined scene can better present all information of interest to the user or information valuable to the user in the sub-scenes through temporal accumulation.
[0174] To achieve the above objectives, the present invention also proposes an XR combination scene automatic combination system, the system comprising a memory, a processor, and an XR combination scene automatic combination method program stored in the memory, and the XR combination scene automatic combination method program executes the above method when run by the processor.
[0175] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the automatic combination method of XR combined scenes as described above is executed.
[0176] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Equivalent structures or equivalent process changes made by the scheme using the contents of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
[0177] References
[0178] 1. Cai Tiefeng. Teaching methods, systems and media based on immersive XR smart classroom. CN202310228496.7, 2023-03-10
[0179] 2. Cai Tiefeng. Construction method, system and storage medium of ubiquitous training campus based on XR technology. CN202210908795.0[P], 2022-07-29
[0180] 3. Cai Tiefeng. A new dimension space construction method, system and platform based on XR technology. CN202210022608.9[P], 2022-01-10
[0181] 4. Cai Tiefeng. A method, system, terminal and medium for generating XR combined scene experience images. CN2023112052542[P], 2023-09-19
[0182] 5. Cai Tiefeng. A method, system and storage medium for generating XR combined scene experience. CN2023110288353[P], 2023-08-16
[0183] 6. Cai Tiefeng. A method, system and medium for generating combined scene experience based on XR technology. CN2023110286220[P], 2023-08-16
[0184] 7. Cai Tiefeng. A method, system, anchor terminal, client terminal and medium for constructing XR stereoscopic interactive live broadcast. CN2023110283716[P], 2023-08-16
[0185] 8. Cai Tiefeng. Immersive interactive live broadcast construction method, system and medium based on XR technology. CN202210906282.6[P], 2022-07-29
[0186] 9. Cai Tiefeng. XR combined scene construction method, system and medium based on image cropping. CN2024100048837[P], 2024-01-03
[0187] 10. Cai Tiefeng. An immersive classroom construction method, system and storage medium based on XR technology. CN202210427227.9[P], 2022-04-22
Claims
1. A method for automatically combining scenes using extended reality technology, characterized in that: The system generates a combined scene experience for a first user. In the combined scene experience, according to the scene content and / or scene information that the first subscene needs to present in the combined scene, the system automatically adjusts the presentation of the first subscene in the combined scene so that the scene content and / or scene information of the first subscene is presented in the combined scene. The automatic adjustment of the presentation of the first subscene in the combined scene specifically includes adjusting one or more of the content, posture angle, position, and scaling size of the first subscene presented in the combined scene. The first subscene is one of a virtual scene, a virtual-reality fusion scene, and a real scene. The adjustment of the content of the first subscene presented in the combined scene specifically includes adjusting the three-dimensional imaging interval of the first subscene presented in the combined scene. The adjustment of the position, posture angle, and scaling size of the first subscene presented in the combined scene specifically includes adjusting the translation amount, rotation amount, and scaling amount in the rotation, translation, and scaling relationship between the first subscene coordinate system and the combined scene coordinate system or the first user field of view coordinate system. In the process of automatically adjusting the three-dimensional imaging interval of the first sub-scene and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, the system forms a path for automatically adjusting the presentation of the first sub-scene in the combined scene, specifically including: automatically generating the path under the set optimization goals and constraints for presenting the first sub-scene scene content and / or scene information to the combined scene, and / or the strategy for presenting the first sub-scene scene content and / or scene information to the combined scene, any point on the path has a corresponding three-dimensional imaging interval, a rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system, and / or the time required for the human eye to perceive the information, and the system automatically adjusts the three-dimensional imaging interval of the first sub-scene and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system according to this path, thereby automatically adjusting the presentation of the first sub-scene in the combined scene.
2. The automatic combination method of combined scenes according to claim 1, characterized in that: The first sub-scene is presented in the first display interval of the combined scene, the first display interval is defined in the combined scene coordinate system or the first user field of view coordinate system, the three-dimensional imaging interval and the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user field of view coordinate system satisfy the constraints of the first display interval; the automatic adjustment of the presentation of the first sub-scene in the combined scene also includes automatically adjusting the playback of the first sub-scene sound in the combined scene.
3. The automatic combination method of combined scenes according to claim 2, characterized in that: The constraint of the first display interval is specifically that the three-dimensional interval of the combined scene obtained after the three-dimensional imaging interval is transformed by the rotation, translation and scaling relationship is included in the first display interval or is equal to the first display interval, and the automatic adjustment of the playback of the first sub-scene sound in the combined scene specifically includes automatically adjusting which sound sources of the first sub-scene produce sounds played in the combined scene, and / or the position or posture of each sound source of the first sub-scene played in the combined scene.
4. The automatic combination method of combined scenes according to claim 3, characterized in that: The way in which the system generates a combined scene experience for the first user is one of the following ways: the picture generated by the first sub-scene under the constraint of the three-dimensional imaging interval is synthesized with the pictures of other scene contents in the combined scene to obtain a combined scene experience picture; the picture generated by the first sub-scene without the constraint of the three-dimensional imaging interval is cropped under the constraint of the three-dimensional imaging interval and synthesized with the pictures of other scene contents in the combined scene to obtain a combined scene experience picture; the first sub-scene generates scene three-dimensional information and loads it into the combined scene and then renders it to obtain a combined scene experience picture; the first sub-scene and other contents of the combined scene are directly rendered together to generate a combined scene picture.
5. The automatic combination method of combined scenes according to claim 4, characterized in that: When the first sub-scene generates an image under the constraint of the three-dimensional imaging interval or not under the constraint of the three-dimensional imaging interval, if the first display interval is defined in the first user field of view coordinate system, the image generated by the first sub-scene is not affected by the change of the first user's posture; if the first display interval is defined in the combined scene coordinate system, the first user's posture and pupil distance must be transformed to the first sub-scene coordinate system according to the transformation relationship between the coordinate systems, and the first sub-scene image is generated according to the posture and pupil distance obtained after the transformation.
6. The automatic combination method of combined scenes according to claim 5, characterized in that: The scene content and / or scene information is one or more of static, real-time dynamic, and non-real-time dynamic, and the optimization goal is one or more of the following: the path length is shortest, the most information of the first sub-scene is presented in the combined scene after the accumulation in time sequence, the most information of the first sub-scene can be perceived by the first user, the non-real-time dynamic information of the first sub-scene is the least in reverse order, and the time required for the first user to perceive all the information of the first sub-scene in the combined scene is shortest; the constraint condition is one or more of the following constraint conditions: all the static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene are presented in the combined scene after the accumulation in time sequence, all the static scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene can be perceived by the first user in the combined scene without moving the position The combined scene is perceived, the rotation relationship change rate in the rotation, translation and scaling relationship between the first sub-scene coordinate system and the combined scene coordinate system or the first user's field of view coordinate system does not exceed a specific threshold, the translation relationship change rate in the rotation, translation and scaling relationship does not exceed a specific threshold, the scaling relationship change rate in the rotation, translation and scaling relationship does not exceed a specific threshold, the time for the three-dimensional imaging interval to pass through or stay on the scene content and / or scene information meets the time requirement for the first user or all users to fully perceive the scene content and / or scene information, and the dynamic information time is not in reverse order; the strategy is that when the first display interval of the combined scene cannot fully present all the real-time dynamic information and / or scene content of the first sub-scene, it is necessary to make a trade-off, and the trade-off strategy is one or more of first-come, first-served, rotation, proximity, priority or information volume competition.
7. The automatic combination method of combined scenes according to claim 6, characterized in that: The influencing factors of the measurement value of the path length include one or more of the translation change, rotation change, and scaling transformation amount in the rotation, translation, and scaling change relationship between the first sub-scene coordinate system and the first user coordinate system or the combined scene coordinate system. The specific method for generating the path for automatically adjusting the presentation of the first sub-scene in the combined scene is: under the constraint that the first sub-scene can cover all scene content and / or scene information of the first sub-scene that needs to be presented in the combined scene after accumulation, or under the constraint that all scene content and / or scene information that needs to be presented in the combined scene can be fully perceived by the first user without the first user moving, a group of three-dimensional imaging intervals are selected as key points. key imaging interval, and further obtain the rotation, translation and scaling relationship between the first sub-scene coordinate system corresponding to each key imaging interval and the first user field of view coordinate system or the combined scene coordinate system under the constraint of the first display interval, and calculate the time required to stay or pass through this key imaging interval during the three-dimensional imaging interval adjustment process according to the scene content and / or scene information human eye perception characteristics contained in each key imaging interval that need to be presented in the combined scene. Any sequence of the key imaging intervals can constitute a path for the system to automatically adjust the presentation of the first sub-scene in the combined scene, and present the first sub-scene scene content and / or scene information to the optimization goal and constraint conditions and / or the first sub-field according to the set first sub-scene scene content and / or scene information to the combined scene. A strategy for presenting scene content and / or scene information to the combined scene is used to obtain a path consisting of a sequence of key imaging intervals, and a path for adjusting the presentation of the first sub-scene in the combined scene in a jumpy manner is directly constructed by using the sequence of key imaging intervals, the rotation, translation and scaling relationship between the first sub-scene coordinate system corresponding to each key imaging interval and the first user field of view coordinate system or the combined scene coordinate system, and the length of time the key imaging interval stays in the key imaging interval during the three-dimensional imaging interval adjustment process, or the three-dimensional imaging intervals of any key imaging intervals that are adjacent in sequence in the key imaging interval sequence and the rotation, translation and scaling relationship between the corresponding first sub-scene coordinate system and the first user field of view coordinate system or the combined scene coordinate system are interpolated, and then Further, according to the length of stay or the length of time passed in each key imaging interval during the three-dimensional imaging interval adjustment process, a path for continuously changing and adjusting the first sub-scene to be presented in the combined scene is constructed, or a path for adjusting the first sub-scene to be presented in the combined scene with partial jumping changes and partial continuous changes is constructed by combining the methods of constructing the jumping change path and the continuously changing path; the first sub-scene sound source played in the combined scene must belong to the three-dimensional imaging interval of the first sub-scene, and the position or posture of the first sub-scene sound source played in the combined scene in the combined scene is the position or posture of the first sub-scene sound source in the first sub-scene coordinate system converted according to the transformation relationship between the first sub-scene coordinate system and the combined scene coordinate system.
8. The automatic combination method of combined scenes according to claim 7, characterized in that: The scene content and / or scene information that the first sub-scene needs to present in the combined scene is specifically embodied as a set of objects and / or object parts and their states, and the dynamic scene content and / or scene information is one or more of the following scene changes: voice, object disappearance, object generation, object rotation, object movement, object scaling, object deformation, object texture change, and chemical change; before the system automatically adjusts the presentation of the first sub-scene in the combined scene, it is also necessary to determine the scene content and / or scene information that the first sub-scene needs to present in the combined scene, and the determination method is one or more of manual calibration, statistical calibration, and artificial intelligence calibration; the system automatically adjusts the presentation of the first sub-scene in the combined scene specifically by automatically adjusting according to the posture change of the first user or the posture change of any object in the first sub-scene; in order to avoid occlusion between objects, some objects in the first sub-scene or the combined scene are set to a hidden state or a transparent state.
9. An automatic combination system for combined scenes using extended reality technology, characterized in that: The system includes a memory, a processor, and an extended reality technology combined scene automatic combination method program stored in the memory, and the extended reality technology combined scene automatic combination method program is executed by the processor to perform the method as claimed in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is called by the processor, the method for automatically combining scenes using extended reality technology as described in any one of claims 1 to 8 is executed.
