Multi-terminal fusion system and method for three-dimensional data space

Abstract

The invention discloses a multi-terminal fusion system and method for a three-dimensional data space, and the system comprises an application layer which provides the front-end effect display of the system for a user through detecting a video image rendered by a server; the interface layer is used for providing various service modules for supporting business functions and data processing; the service layer is responsible for data storage, management, query and analysis, at least comprises space rendering service, scheduling service, user authentication, session service, instruction control, security service and data storage, and is used for providing data support for the system; various types of data generated by the system are used for supporting operation of the system, and the data storage layer at least comprises a service database, a conference database, an instruction information base and a message event. The application breaks through the limitation of single-channel data transmission and planar presentation, and realizes integration and synchronous display of multimedia data such as voice, video, image and the like in the same space.

Description

Technical Field

[0001] The present invention relates to the field of audio and video technology, and in particular to a multi-terminal fusion system and method for three-dimensional data space. Background Art

[0002] Existing video conferencing systems mainly use a two-dimensional grid layout, where participants are divided into different small windows, lacking the sense of being in the same space. This "screen segmentation" weakens the sense of presence and boundaries between participants. At the same time, limited by the single-channel data transmission mode, different media content (such as voice, video, documents, etc.) is difficult to achieve seamless integration and synchronous presentation, affecting the effective communication and understanding of information.

[0003] In addition, the traditional flat presentation method fails to fully utilize people's spatial perception ability. The lack of immersive experience makes it difficult for users to feel as if they are in the scene. Coupled with the sense of separation between different terminal devices, it is impossible to achieve seamless connection and real-time interaction across platforms, which further limits the effectiveness of virtual meetings.

[0004] In summary, there is an urgent need for an innovative solution that breaks through two-dimensional limitations, supports multi-channel data fusion, and provides an immersive experience. Summary of the invention

[0005] In response to the above technical problems, the present invention provides a multi-terminal fusion system and method for a multi-channel, multi-dimensional three-dimensional data space, which breaks through the limitations of single-channel data transmission and planar presentation, and realizes the integration and synchronous display of multimedia data such as voice, video, and images in the same space.

[0006] The first embodiment of the present invention provides a multi-terminal fusion system of a three-dimensional data space, and the system includes: an application layer, which provides users with a front-end effect display of the system by detecting the video screen rendered by the server; an interface layer, which provides various service modules that support business functions and data processing; a service layer, which is responsible for the storage, management, query and analysis of data, and at least includes space rendering services, scheduling services, user authentication, session services, command control, security services, data storage, and data storage, which are used to provide data support for the system; a data storage layer, various types of data generated by the system are used to support the operation of the system, and the data storage layer includes at least a business database, a conference database, a command information library, and a message event.

[0007] Optionally, the system further includes: an infrastructure layer, which provides underlying technology and hardware support for the system. The underlying infrastructure includes servers, networks and cluster management, which provide hardware and network support and dynamic capacity expansion for the system.

[0008] Optionally, the application layer includes at least a mobile phone, a head display, a computer, a web page and other SDKs.

[0009] Optionally, the interface layer includes at least audio stream receiving / pushing, command triggering, state control / opening / closing, and data interaction interface.

[0010] Optionally, the head display includes a first head display, a second head display, a computer and a mobile terminal, and the mobile terminal includes a mobile phone.

[0011] Optionally, the first head display and the second head display are respectively connected to the RTC service for transmitting subscribed audio and video data streams to achieve real-time audio and video communication; the first head display and the second head display are respectively connected to the spatial rendering service for synchronizing business data and operation instructions. After processing the operation instructions or business data from the first head display and the second head display, the spatial rendering service pushes the results to the data pipeline subscribed by the first head display or the second head display.

[0012] Optionally, the mobile terminal and the computer share a channel to send subscription information to the RTC service, and send operation instructions, status messages and events to the spatial rendering service.

[0013] The second embodiment of the present invention also provides a multi-terminal fusion method for three-dimensional data space, which is applied to the multi-terminal fusion system for three-dimensional data space described in any of the above items, and the method includes: after the spatial rendering service receives the start instruction, it creates a corresponding virtual camera according to the micro-terminal identifier of the micro-terminal that initiates the instruction, and establishes a one-to-one logical connection with it; the spatial rendering service opens an RTC subscription channel for the virtual camera, which is used to continuously push the camera image to the corresponding micro-terminal.

[0014] Optionally, the method also includes: when the position or orientation of the local camera changes, the micro-terminal sends a synchronization instruction to the spatial rendering service; after receiving the synchronization instruction, the spatial rendering service immediately updates the spatial parameters of the virtual camera corresponding to the local camera, and adjusts the observation position and angle to align with the perspective adjusted by the micro-terminal; the adjusted picture is transmitted back to the micro-terminal in real time through the RTC channel to achieve picture synchronization from the cloud to the micro-terminal.

[0015] Optionally, the method further includes: after receiving the startup instruction, the spatial rendering service creates a first camera position by default and subscribes to a fixed RTC channel, and the first camera position supports changing the acquisition position through instructions to obtain panoramic images at different angles.

[0016] In the technical solution provided by the embodiment of the present invention, the coupling degree between the micro-terminal and the system is relatively low, and the micro-terminal only needs to focus on its own data collection and instruction sending, as well as the data in the subscribed pipeline, without having to worry about the specific processing logic of the data; the spatial rendering service is responsible for the unified processing of data, which can realize the centralized management and scheduling of business logic, and provide system maintainability and business maintainability; the storage and distribution of static resources can be handed over to a dedicated service to complete, which reduces the burden on the micro-terminal and the compatibility of different micro-terminals relying on hardware, and improves the startup speed and operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural schematic diagram of a multi-terminal fusion system of a three-dimensional data space of the present invention.

[0018] Figure 2 It is a flow chart of a multi-terminal fusion system of a three-dimensional data space of the present invention.

[0019] Figure 3 The present invention is a flowchart of a multi-terminal fusion method of a three-dimensional data space. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Please refer to Figure 1As shown, the present invention provides a multi-terminal fusion system for three-dimensional data space, the system includes an application layer, an interface layer, a service layer and a data storage layer, the application layer provides the user with a front-end effect display of the system by detecting the video screen rendered by the server; the interface layer is used to provide various service modules that support business functions and data processing; the service layer is responsible for the storage, management, query and analysis of data, including at least space rendering service, scheduling service, user authentication, session service, command control, security service, data storage, data storage, which is used to provide data support for the system; various types of data generated by the system are used to support the operation of the system, and the data storage layer includes at least a business database, a conference database, a command information library, and a message event.

[0024] The application layer includes mobile phones, head-mounted displays, tablet computers, handheld computers, desktop computers, laptop computers, ultra-mobile personal computers (UMPC), netbooks, cellular phones, personal digital assistants (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, augmented reality (AR), virtual reality (VR), mixed reality (MR) and other extended reality (XR) devices, vehicle-mounted devices or smart city devices. The embodiments of the present application do not impose special restrictions on the specific types of electronic devices. The application layer provides users with the front-end effect display of the system by detecting the video screen rendered by the server.

[0025] The interface layer is used to provide various service modules that support business functions and data processing, such as audio stream reception / push, command triggering, state control / open / close, data interaction interface, etc. This layer plays a connecting role, connecting the view layer and the data layer.

[0026] The service layer is responsible for data storage, management, query and analysis, including space rendering services, scheduling services, user authentication, session services, command control, security services, data storage, data storage, etc. This layer provides data support for the system.

[0027] The data storage layer: various types of data generated by the system, including business database, conference database, instruction information database, message events, etc. These basic data support the operation of the entire system.

[0028] The system also includes infrastructure, which is used to provide underlying technology and hardware support for the system. The underlying infrastructure includes servers, networks and cluster management, which provides hardware and network support for the entire system, as well as dynamic expansion.

[0029] The micro-client, as a client, can be any head-mounted display device, including the first head-mounted display, the second head-mounted display, a computer and a mobile terminal, and the mobile terminal includes a mobile phone. Please refer to Figure 2 As shown, the first head display is, for example, an A-end head display, and the second head display is, for example, a B-end head display. The head display devices will establish connections with the RTC service and the spatial rendering service respectively during initialization. The main purpose of establishing a connection with the RTC service is to subscribe to audio and video data streams and realize real-time audio and video communication. The purpose of establishing a connection with the spatial rendering service is to synchronize other business data and operation instructions, such as user actions, scene status, etc. The micro-end also needs to agree on some data formats and processing rules with the spatial rendering service in order to correctly parse and display the data stream pushed by the spatial rendering service. After processing the instructions or other business data from the micro-end, the spatial rendering service will push the results to the data pipeline subscribed by the micro-end.

[0030] The advantages of the present invention are:

[0031] The micro-client has a low degree of coupling with the system and only needs to focus on its own data collection and command sending, as well as subscribing to data in the pipeline, without having to worry about the specific data processing logic. The spatial rendering service is responsible for unified data processing, which can achieve centralized management and scheduling of business logic, and provide system maintainability and business maintainability. The storage and distribution of static resources can be handed over to dedicated services, which reduces the burden on the micro-client and the compatibility of different micro-clients relying on hardware, and improves startup speed and operating efficiency.

[0032] Thanks to the decoupling of micro-clients from RTC services and spatial rendering services, the system supports flexible access to various forms of terminals. However, this architecture places higher demands on the processing capabilities and data distribution efficiency of the server. The spatial rendering service needs to have sufficient performance and stability to carry a large number of concurrent requests. The pipelines subscribed by each micro-client must be distinguished and isolated to prevent data cross-contamination.

[0033] The mobile terminal and computer terminal of the present invention share a channel and uniformly receive the audio and video content of the entire space, which is similar to the public stream information of the live broadcast system. Sending the fixed collected video stream images to the public channel not only reduces the complexity of the system, but also eliminates the need to maintain RTC service and space rendering service instances for the mobile terminal and computer terminal respectively, thereby simplifying the system architecture, saving computing and storage resources, and avoiding unnecessary redundant computing and storage.

[0034] The advantages of the present invention are that data consistency is guaranteed. All terminals obtain data from the same resource space, and problems such as picture and sound asynchrony caused by differences in data sources will not occur, which facilitates information extraction and display of content. The optimization of bandwidth and traffic requires the use of multi-resolution and multi-bitrate adaptive technology, and dynamic adjustment of relevant parameters according to different terminals. When rendering on the mobile terminal, relative adjustments should be made according to screen size, interaction and other methods.

[0035] Please refer to Figure 3 As shown, the present invention further provides a multi-terminal fusion method for three-dimensional data space, the method is applied to the multi-terminal fusion system for three-dimensional data space described in any of the above embodiments, the method comprising:

[0036] After receiving the start command, the spatial rendering service will first create a dedicated virtual camera instance (such as Figure 3 S001) and establish a one-to-one logical connection with it. At the same time, the spatial rendering service will open an RTC subscription channel for the virtual camera to continuously push the camera image to the corresponding micro-end. Since different micro-ends are connected to different virtual cameras, their subscriptions through RTC are also independent of each other. This allows multiple concurrently connected micro-ends to see spatial images from different perspectives.

[0037] In one embodiment of the present invention, in order to synchronize the image of the virtual camera with the actual viewing angle of the micro-end, the micro-end will send a synchronization instruction to the spatial rendering service when the local camera changes (such as changing the direction or position). After receiving the instruction, the spatial rendering service immediately updates the spatial parameters of the corresponding virtual camera and adjusts its observation position and angle to align with the latest viewing angle of the micro-end. The adjusted image is transmitted back to the micro-end in real time through the RTC pipeline, thereby completing a cloud-to-micro-end image synchronization process.

[0038] In addition to the virtual cameras (e.g. S001) that correspond one-to-one to the micro-client, a special first camera position is created by default when the spatial rendering service is started, such as Figure 3The S000 camera position is created and subscribed to a fixed RTC channel for it. Unlike other virtual cameras, the S000 camera position provides a perspective that overlooks the entire virtual space, and can simultaneously observe the virtual avatars of all micro-end users and other objects in the space. The S000 camera also supports changing the acquisition position through instructions to obtain panoramic images from different angles. Unlike the on-demand creation and destruction of micro-end cameras, the S000 camera will continue to run during the entire declaration cycle of the space service and will not be shut down because a micro-end exits. For example: when a micro-end actively disconnects from the space service, the space service will destroy the corresponding virtual camera instance and close the corresponding RTC subscription channel to release system resources. The cameras of other micro-ends and the S000 camera are not affected and continue to run until the space service itself is shut down.

[0039] This design mode based on virtual cameras and dynamic RTC pipelines also maximizes the use of cloud resources and simplifies the calculation and transmission logic of the micro-end. The micro-end only needs to send lightweight synchronization instructions according to the changes in the local perspective, without having to worry about the complex 3D rendering process. The spatial service takes over the scheduling and declaration cycle management of all virtual cameras, and can use the S000 camera to achieve global monitoring and provide support for other business logic.

[0040] The present invention adopts innovative data compression, transmission and synchronization mechanisms to overcome the problem of real-time fusion of multi-source heterogeneous data such as voice, video, image and action, ensuring that various types of data can be seamlessly connected and synchronously presented in three-dimensional space, laying a data foundation for immersive experience.

[0041] The present invention utilizes the uniqueness of the micro-terminal head display device to dynamically generate an independent virtual camera and personalized observation perspective for each micro-terminal user. Through real-time synchronization and parameter adjustment of the virtual camera, the micro-terminal screen is always consistent with the user's perspective, which is a significant improvement over the traditional single perspective. The present invention centrally reconstructs and renders multi-channel fusion data in the cloud, and creates a realistic immersive space through spatial audio, holographic projection and other means. The management of virtual cameras is also uniformly scheduled in the cloud, and then distributed to each micro-terminal through a dynamic RTC channel, thereby achieving high-performance and high-consistency spatial presentation.

[0042] It should be understood that the "one embodiment" mentioned in the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the "one embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0043] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-terminal fusion system for three-dimensional data space, characterized in that: The system comprises: The application layer provides users with the front-end effect display of the system by detecting the video screen rendered by the server; The interface layer provides various service modules that support business functions and data processing; The service layer is responsible for the storage, management, query and analysis of data, including at least spatial rendering service, scheduling service, user authentication, session service, command control, security service, data storage, and data storage, which is used to provide data support for the system; Data storage layer: various types of data generated by the system are used to support the operation of the system. The data storage layer at least includes business database, conference database, instruction information library, and message events.

2. The multi-terminal fusion system for three-dimensional data space according to claim 1 is characterized in that: The system further comprises: The infrastructure layer provides the underlying technology and hardware support for the system. The underlying infrastructure includes servers, networks, and cluster management, which provide hardware and network support for the system, as well as dynamic expansion.

3. The multi-terminal fusion system for three-dimensional data space according to claim 1 is characterized in that: The application layer includes at least mobile phones, head displays, computers, web pages and other SDKs.

4. The multi-terminal fusion system for three-dimensional data space according to claim 1 is characterized in that: The interface layer at least includes audio stream receiving / pushing, command triggering, state control / opening / closing, and data interaction interface.

5. The multi-terminal fusion system for three-dimensional data space according to claim 3 is characterized in that: The head display includes a first head display, a second head display, a computer and a mobile terminal, and the mobile terminal includes a mobile phone.

6. The multi-terminal fusion system for three-dimensional data space according to claim 5 is characterized in that: The first head display and the second head display are respectively connected to the RTC service for transmitting subscribed audio and video data streams to realize real-time audio and video communication; the first head display and the second head display are respectively connected to the spatial rendering service for synchronizing business data and operation instructions. After processing the operation instructions or business data from the first head display and the second head display, the spatial rendering service pushes the results to the data pipeline subscribed by the first head display or the second head display.

7. The multi-terminal fusion system for three-dimensional data space according to claim 6 is characterized in that: The mobile terminal and the computer share a channel to send subscription information to the RTC service, and send operation instructions, status messages and events to the spatial rendering service.

8. A multi-terminal fusion method for three-dimensional data space, characterized in that: The method is applied to the multi-terminal fusion system for three-dimensional data space according to any one of claims 1 to 7, and the method comprises: After receiving the start command, the spatial rendering service creates a corresponding virtual camera according to the micro-end identifier of the micro-end that initiated the command, and establishes a one-to-one logical connection with it; The spatial rendering service opens an RTC subscription channel for the virtual camera to continuously push the camera image to the corresponding micro-end.

9. The multi-terminal fusion method for three-dimensional data space according to claim 8, characterized in that: The method further comprises: When the position or orientation of the local camera changes, the micro client sends a synchronization command to the spatial rendering service; After receiving the synchronization command, the spatial rendering service immediately updates the spatial parameters of the virtual camera corresponding to the local camera, and adjusts the observation position and angle to align with the adjusted viewing angle of the micro-end; The adjusted image is transmitted back to the micro-end in real time through the RTC channel, realizing image synchronization from the cloud to the micro-end.

10. The multi-terminal fusion method for three-dimensional data space according to claim 8, characterized in that: The method further comprises: After receiving the startup command, the spatial rendering service creates the first camera position by default and subscribes to a fixed RTC channel. The first camera position supports changing the acquisition position through commands to obtain panoramic images from different angles.

Images