Multi-screen interaction system based on virtual scene

Abstract

The invention belongs to the technical field of multi-screen interaction systems, and particularly relates to a multi-screen interaction system based on a virtual scene, which comprises a virtual scene generation module, a virtual scene generation module generation model and a virtual scene, a virtual scene model and a model template called from an established resource library, and a multi-screen connection and identification module. The multi-screen connection and identification module is used for connecting a plurality of display devices and adaptively adjusting the display effects of the plurality of display devices; the interaction control module is used for receiving and synchronizing interaction operation instructions of the display devices and the control end; the data synchronization and management module is used for realizing real-time synchronization of virtual scene data among screens and management of system data; and a display system. According to the method and the device, the generation of the virtual scene can be realized, and the interaction experience is improved in a multi-screen interaction process.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-screen interactive systems, and in particular relates to a multi-screen interactive system based on a virtual scene. Background Art

[0002] Multi-screen interaction refers to the use of IGRS protocols, Miracast protocols, etc., through WIFI network connection, on different multimedia terminals (such as different smart terminal devices based on different operating systems such as HarmonyOS, IOS, Android, such as mobile phones, PADs, TVs, etc.), to perform a series of operations such as transmission, analysis, display, and control of multimedia (audio, video, picture) content. The display content can be shared simultaneously on different platform devices to enrich the user's multimedia life. The multi-screen interactive system based on virtual scenes is a solution that combines virtual reality (VR), augmented reality (AR) or mixed reality (MR) technology to allow users to interact through multiple screens. This system can be used in various application fields, such as education, entertainment, commercial display, remote collaboration, etc.

[0003] Problems with existing technologies: However, multi-screen interactive scenes often fail to realize the actual generation and processing of various elements in the scene in terms of scene model generation, and the generation speed is slow. Especially after the virtual scene is generated, the interactive experience will cause the process to be stuck, and the stuck will affect the experience of scene interaction. Therefore, most of the methods are to reduce the structural complexity and image quality of the model in the scene. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-screen interactive system based on virtual scenes, which can realize the generation of virtual scenes and enhance the interactive experience through the multi-screen interactive process.

[0005] The technical solution adopted by the present invention is as follows: A multi-screen interactive system based on virtual scenes, comprising: A virtual scene generation module, which generates models and virtual scenes, and retrieves virtual scene models and model templates from an established resource library; A multi-screen connection and identification module is used to connect multiple display devices and perform adaptability adjustment on the display effects of multiple display devices; An interactive control module, which is used to receive and synchronize interactive operation instructions between each display device and the control terminal; Data synchronization and management module: The data synchronization and management module is used to achieve real-time synchronization of virtual scene data between various screens and management of system data; Display system, which is a plurality of display devices or a combination of a plurality of display devices, and a coordinated display between a connected control terminal and a plurality of display devices; The virtual scene generation module includes at least one generation module, at least one analysis module, at least one distribution module, at least one central server, multiple sub-center servers, and a control terminal connected to the central server and / or the sub-center server; The interactive control module runs the interactive system. The interactive system realizes data connection with the display device through the multi-screen connection and identification module by inputting one or more control terminals into the data synchronization and management module, and realizes data synchronization between the control terminal and the multi-screen connection and identification module to realize synchronous feedback of data.

[0006] The analysis module uses the model segmentation method to segment the model, including the following steps: The analysis model identifies and segments each model element based on the model segmentation principle to generate multiple elements; Divide the display canvas inside the display system into multiple depth canvases in sequence; Classify multiple features according to feature depths and depth canvases corresponding to the depths; Matching the corresponding depth element with the corresponding depth canvas; The dimensions and physical characteristics of multiple elements are obtained, and the data of multiple elements are matched to realize the segmentation and combination of the model.

[0007] The principles of model splitting include but are not limited to the number of components, the shape of each component, the physical characteristics of each component, the motion state, rendering data or the depth of the component.

[0008] The interactive control module includes at least one control terminal corresponding to the display system, and the control terminal generates at least one free model in the display system through the interactive control module, and the free model moves relative to the plurality of canvases and / or corresponding elements; Movement methods include but are not limited to penetration through multiple layers of canvases and interaction between elements; Data synchronization and feedback between the control end and the free model.

[0009] The control end includes but is not limited to VR devices, handle devices, motion capture devices, eyeball devices, force feedback devices or modeling devices.

[0010] The control end is used for synchronizing the feedback data of the free model. The control end is provided with at least one set of current control module and mechanical control module. The current control module and the mechanical control module are used for outputting the feedback data of the free model synchronized by the control end.

[0011] A multi-screen interactive method based on a virtual scene comprises the following steps: Obtain data information of models and virtual scenes, as well as free model feedback data; The output end is used to perceive the free model feedback data and interact with the model and the virtual scene; Acting on the free model in the display scene of one or more virtual scenes through the output end; The output end interacts with the model located in the display scene through the free model.

[0012] The interactive method includes the following steps: Obtain the model features in the display scene, and the relative coordinates between the free model and the model; Through the relative movement between free models, various forces acting on the models can be realized; Realize changes to various forms and states of the model; Acquire multi-screen connection information, realize the movement of models in multiple screens through free models, and interact between terminal-side free models generated by multiple terminals connected to multiple screens and models.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any of the aforementioned methods is implemented.

[0014] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the aforementioned methods is implemented.

[0015] The technical effects achieved by the present invention are: The present invention distributes the virtual scene under the corresponding canvas to multiple display devices through the distribution module, and further distributes the model to the display devices at corresponding positions according to the canvas corresponding to multiple elements through the distribution module, and then distributes the generated model to the sub-center server, and realizes data processing of the interaction between the virtual scene and the model during the interaction process through the central server and / or the sub-center server.

[0016] The present invention, when the central server is in a fully loaded state, processes data through the sub-central server to reduce the data processing pressure of the central server. For models or virtual scenes that cannot be processed by the sub-central server, they are queued for processing at the central server, or the model is segmented by the analysis module using a model segmentation method to achieve further processing of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural schematic diagram of the multi-screen interactive system of the present invention; Figure 2It is a structural schematic diagram of a virtual scene generation module in the present invention; Figure 3 It is a flow chart of a method for segmenting a model using a model segmentation method by the analysis module in the present invention; Figure 4 It is a schematic diagram of the structural connection of the interactive control module in the present invention; Figure 5 This is a schematic diagram of the process of the multi-screen interaction method in the present invention. Figure 1 ; Figure 6 This is a schematic diagram of the process of the multi-screen interaction method in the present invention. Figure 2 . DETAILED DESCRIPTION

[0018] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] According to an embodiment of the present invention, a method embodiment of a multi-screen interaction method based on a virtual scene is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] like Figure 1-4 As shown, a multi-screen interactive system based on a virtual scene includes: The virtual scene generation module generates models and virtual scenes, and retrieves virtual scene models and model templates from the established resource library. The virtual scene models and model templates are stored inside the central server. By accepting the data request of the terminal and using the matching method in each sub-center server, the virtual scene models and model templates in the corresponding central server are retrieved. The matching method is used as follows Figure 3 The corresponding elements in the deep canvas and the corresponding models can reduce the generation of duplicate models, reduce the computing pressure of the central server, and increase the generation speed of models and virtual scenes; Multi-screen connection and identification module, which is used to connect multiple display devices and adapt the display effects of multiple display devices. Multiple display devices are connected by wired or wireless means, including but not limited to HDMI data cable, DP data cable, USB data cable, WiFi, Star Flash, Bluetooth, etc.; An interactive control module, which is used to receive and synchronize interactive operation instructions between each display device and the control terminal; Data synchronization and management module: The data synchronization and management module is used to realize the real-time synchronization of virtual scene data between various screens and the management of system data. Data synchronization includes synchronization of instruction-level data, feedback-level data, and synchronization between various components in the system. The display system works in coordination with the multi-screen connection and identification module. The display system is a plurality of display devices or a combination of a plurality of display devices, and a coordinated display between the connected control terminal and the plurality of display devices. The corresponding free model generated in the control terminal is displayed in the plurality of display devices for interaction through the display system to realize the corresponding interactive operation; Among them, the virtual scene generation module includes at least one generation module, at least one analysis module, at least one distribution module, at least one central server, multiple sub-center servers, and a control terminal connected to the central server and / or the sub-center server.

[0022] Further, the generation module generates a corresponding model for the received instruction to generate a corresponding model, and firstly retrieves the material in the model template and realizes the basic combination of each model element through simple combination, and further analyzes the position information of each model element to be composed through the analysis module to generate a complete virtual scene and the model set in the virtual scene. Specifically, the analysis module uses the model segmentation method to segment the model, including the following steps: S101, the analysis model identifies and segments each element of the model based on the model segmentation principle to generate multiple elements; S102, dividing the display canvas in the display system into multiple depth canvases in sequence; S103, classifying multiple elements according to element depths and depth canvases corresponding to the depths; S104, matching the corresponding depth element with the corresponding depth canvas; S105, obtaining the sizes and physical characteristics of the multiple elements, and performing data matching on the multiple elements to realize segmentation and combination of the model.

[0023] The model generated by the above segmentation is distributed to multiple display devices through a distribution module, and the virtual scene under the corresponding canvas is distributed to multiple display devices through the distribution module, and the model is further distributed to the display devices at corresponding positions according to the canvas corresponding to multiple elements. Subsequently, the generated model is distributed to the sub-center server, and data processing of the interaction between the virtual scene and the model is realized through the central server and / or the sub-center server during the interaction process.

[0024] Optionally, when the central server is fully loaded, data is processed through the sub-center server to reduce the data processing pressure of the central server. Models or virtual scenes that cannot be processed by the sub-center server need to be queued for processing on the central server, or the model segmentation method is used to segment the model through the above-mentioned analysis module to achieve further processing of the model. The difference in the processing method is that the processing of virtual scenes or complex models is cancelled to avoid the sub-center server from freezing and other situations.

[0025] Optionally, most model problems can be handled by data processing performed by the sub-center server when the central server is busy, and the sub-center server focuses on the logical processing between the model and the free model.

[0026] For example, when the model is used to demonstrate the falling of an object under the action of gravity, or the state demonstration after falling.

[0027] For example, when the model is demonstrating the free module holding, pushing, pulling, and other operations on the model, the state of the model occurs.

[0028] For further information, please refer to Figure 4 The interactive control module runs the interactive system. The interactive system inputs one or more control terminals into the data synchronization and management module, realizes data connection with the display device through the multi-screen connection and identification module, and realizes data synchronization between the control terminal and the multi-screen connection and identification module to realize synchronous feedback of data.

[0029] Optionally, when the feedback is electric current, an electrode sheet is used to generate electric current to stimulate muscles to achieve a simulated state, such as pain in the hand or touch in other parts of the body.

[0030] Optionally, when the feedback is gravity sensing, a mechanical arm or mechanical transmission is used to generate power or gravity sensing to achieve the sensing of pressure and other sensations by the hands or other parts of the body in the simulated state.

[0031] Optionally, when the feedback is lighting, simulated natural light is used to achieve lighting demonstration of each model in the scene, or the reaction or change of objects under various lighting conditions, and the human body's perception of lighting, such as the human eye's perception of light intensity, the temperature generated by the skin's perception of strong light, etc.

[0032] Optionally, the above scenarios are used, but not limited to the above scenarios, and also include technologies such as vision-related, hearing-related, touch-related, motion capture, facial expression tracking, smell and taste perception, and full-body tracking.

[0033] Optionally, multi-screen display is used to realize the display of the above-mentioned scenes on the screen, and the interaction with the model is realized through the combination of vision and other body perceptions, wherein the model interacting with the model is a free model generated for the control end, or a combination of the free model realized through the terminal.

[0034] Among them, the terminal includes a control terminal, which can realize the control and operation interaction of various model elements in the scene. In addition to the above operations, the terminal also uses the manipulation and control of products such as touch screens.

[0035] As an optional embodiment, the model splitting principle includes but is not limited to: The number of components, such as using a large AI model or directly using a convolutional neural network to identify each component in the model, identify various categories of objects, and locate each component to generate positioning data; The form of each element; Physical characteristics of each element, such as gravity information calculated by density based on the acquired object information, morphological center of gravity information, length, width and height information, etc. Motion state, including changes in rotation, lifting and lowering, and changes in motion state that will occur when changing the state of each element according to the characteristics of each element object; Rendering data or feature depth, color rendering of the scene surface, and feature depth obtained based on the number of constituent elements, the form of each element, and the object characteristics of each element. The depth standard is: based on the model elements in the virtual scene where the model is located, the model simulation form is calculated to be close to the underlying canvas of the virtual scene as the basis for selection; Or, three-dimensional transformation in the virtual scene can be achieved by implementing three-dimensional modeling of the virtual scene and each element in the virtual scene.

[0036] As an optional embodiment, the interactive control module includes at least one control end corresponding to the display system. The control end and the terminal or display device can be integrated, or through an external device such as a VR device, a handle device, a motion capture device, an eyeball device, a force feedback device or a modeling device, the control end generates at least one free model in the display system through the interactive control module, and the free model moves relative to multiple canvases and / or corresponding elements.

[0037] Furthermore, the movement modes include but are not limited to penetration between multiple layers of canvases and interaction between elements.

[0038] Furthermore, data synchronization and feedback are achieved between the control end and the free model.

[0039] For example, when used through a handle or a motion capture device, a hand structure or a body limb structure is generated in the virtual scene, and the hand structure or the body limb structure can be used to grab, push, pull and other actions on each element in the model, and the gravity information of each element can be used to realize the use of electric current or mechanical feedback on the hands and limbs.

[0040] As an optional embodiment, the control end is used to synchronize the free model feedback data, and the control end is provided with at least one set of current control module and mechanical control module, and the current control module and the mechanical control module are used to output the feedback data of the control end synchronized free model.

[0041] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0042] According to another aspect of an embodiment of the present invention, an electronic device is provided. The electronic device includes a memory and a processor; the memory is used to store programs; and the processor executes the program to implement any one of the aforementioned methods.

[0043] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any of the aforementioned methods is implemented.

[0044] Refer to the attached Figure 5 , a multi-screen interactive method based on a virtual scene, comprising the following steps: S1, obtaining data information of the model and virtual scene, as well as free model feedback data; S2, through the output end, is used to perceive the free model feedback data and interact with the model and the virtual scene; S3, acting on the free model in the display scene of one or more virtual scenes through the output end; S4. The output end interacts with the model in the display scene through the free model.

[0045] As an optional embodiment, refer to the attached Figure 6 , the interactive method includes the following steps: S5, obtaining the model features in the display scene, and the relative coordinates between the free model and the model; S6. Realize various forces on the model through relative movement between free models; S7, realize the change of various forms and states of the model; S8. Acquire multi-screen connection information, realize the movement of the model in the multi-screen through the free model, and the interaction between the terminal-side free model generated by multiple terminals connected to the multi-screen and the model.

[0046] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the aforementioned methods is implemented.

[0047] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A multi-screen interactive system based on virtual scenes, characterized in that: include: A virtual scene generation module, which generates models and virtual scenes, and retrieves virtual scene models and model templates from an established resource library; A multi-screen connection and identification module, which is used to connect multiple display devices and perform adaptability adjustment on the display effects of the multiple display devices; An interactive control module, the interactive control module is used to receive and synchronize interactive operation instructions between each display device and the control terminal; A data synchronization and management module, which is used to achieve real-time synchronization of virtual scene data between screens and management of system data; A display system, wherein the display system is a plurality of display devices or a combination of a plurality of display devices, and a coordinated display between the connected control terminal and the plurality of display devices; The virtual scene generation module includes at least one generation module, at least one analysis module, at least one distribution module, at least one central server, multiple sub-center servers, and a control terminal connected to the central server and / or the sub-center server; The interactive control module runs an interactive system, which realizes data synchronization feedback by inputting one or more control terminals into the data synchronization and management module, and realizing data connection with the display device through the multi-screen connection and identification module, and realizing data synchronization between the control terminal and the multi-screen connection and identification module.

2. A multi-screen interactive system based on virtual scenes according to claim 1, characterized in that: The analysis module uses a model segmentation method to segment the model, including the following steps: The analysis model identifies and segments each model element based on the model segmentation principle to generate multiple elements; Divide the display canvas inside the display system into multiple depth canvases in sequence; Classify multiple features according to feature depths and depth canvases corresponding to the depths; Matching the corresponding depth element with the corresponding depth canvas; The dimensions and physical characteristics of multiple elements are obtained, and the data of multiple elements are matched to realize the segmentation and combination of the model.

3. The multi-screen interactive system based on virtual scenes according to claim 1, characterized in that: The model splitting principles include but are not limited to the number of constituent elements, the shape of each element, the physical characteristics of each element, the motion state, rendering data or element depth.

4. The multi-screen interactive system based on virtual scenes according to claim 1, characterized in that: The interactive control module includes at least one control terminal corresponding to the display system, and the control terminal generates at least one free model in the display system through the interactive control module, and the free model moves relative to the plurality of canvases and / or corresponding elements; The movement modes include but are not limited to penetration between multiple layers of canvases and interaction between elements; Data synchronization and feedback between the control end and the free model.

5. The multi-screen interactive system based on virtual scenes according to claim 1, characterized in that: The control end includes but is not limited to VR equipment, handle equipment, motion capture equipment, eyeball equipment, force feedback equipment or modeling equipment.

6. The multi-screen interactive system based on virtual scenes according to claim 1, characterized in that: The control end is used for synchronizing the feedback data of the free model. The control end is provided with at least one set of current control module and mechanical control module. The current control module and mechanical control module are used for outputting the feedback data of the control end synchronizing the free model.

7. A multi-screen interactive method based on a virtual scene, using the multi-screen interactive system as claimed in any one of claims 1 to 6, characterized in that: The steps include: Obtain data information of models and virtual scenes, as well as free model feedback data; The output end is used to perceive the free model feedback data and interact with the model and the virtual scene; Acting on the display scene of the free model in one or more virtual scenes through the output end; The output end interacts with the model located in the display scene through the free model.

8. The multi-screen interactive method based on virtual scene according to claim 7, characterized in that: The interactive method comprises the following steps: Obtain the model features in the display scene, and the relative coordinates between the free model and the model; Through the relative movement between free models, various forces acting on the models can be realized; Realize changes to various forms and states of the model; Acquire multi-screen connection information, realize the movement of models in multiple screens through free models, and interact between terminal-side free models generated by multiple terminals connected to multiple screens and models.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 7 or 8 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 7 or 8 is implemented.

Images