Metaverse five-fold screen interactive display system and method

Abstract

The present invention provides a metaverse five-fold screen interactive display system and method, wherein the initial metaverse data stream of a cloud server is collected by a first application host, and the collected initial metaverse data stream is converted into an initial metaverse screen; the initial metaverse screen of the first application host is collected by a second application host, and the collected initial metaverse screen is divided into five sub-screens according to a preset screen division method, and then the five sub-screens are projected one by one into five projection screens, and then the five projection screens are synchronously output one by one to the sky screen, the right screen, the ground screen, the left screen and the front screen for display. The present invention can improve the interactive display effect of the metaverse five-fold screen, and at the same time improve the universality of the metaverse five-fold screen interactive display technology to the existing metaverse data resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a metaverse five-fold screen interactive display system and method. Background Art

[0002] The current metaverse is broadly defined as a virtual world that can interact with the real world using digital technology. It refers to a new type of Internet application and social form that integrates multiple new technologies and provides an immersive experience based on extended reality technology, closely integrating the virtual world with the real world, and allowing each user to produce content and edit the world. In the current metaverse experience, virtual interactive experience often relies on VR and AR helmets / glasses. Its disadvantages are: (1) In terms of content production, since it is impossible to adaptively convert the existing three-dimensional software system / video stream into the scene or video format required in the immersive space, it takes a huge amount of manpower and material resources to develop three-dimensional space modeling, graphics rendering and interactive programs when building the corresponding space scene; (2) When multiple people experience it, everyone needs to wear VR and AR helmets / glasses and enter the common immersive space through the network. This method is restricted by the site environment, the number of VR devices, network bandwidth and the synchronization of VR hosts. In particular, it is impossible to achieve rapid team participation during team visits, training and virtual tourism.

[0003] LED display screens are applied to the Metaverse due to their excellent display performance of ultra-high-definition display, high-order low-gray, and the advantages of being able to easily carry new technologies and enhance the interactive sense of virtual reality. In the process of applying LED display screens to the Metaverse, the immersive five-fold screen display technology of the Metaverse was born. However, the existing three-dimensional content production methods of the five-fold screen require customized three-dimensional videos / scenes, which is a large workload and is not suitable for online three-dimensional video streams with high real-time requirements. In addition, due to the special screen structure of the five-fold screen, the existing three-dimensional video stream cannot be directly used for the playback of the immersive five-fold screen, and the multimedia content, scenes and application interfaces produced for VR glasses cannot be perfectly displayed on the immersive five-fold screen. This has caused many existing Metaverse materials, videos, and application software to be unable to be directly applied on the immersive five-fold screen. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a Metaverse five-fold screen interactive display system and method to alleviate the above-mentioned problems existing in the related art.

[0005] In the first aspect, an embodiment of the present invention provides a metaverse five-fold screen interactive display system, the system comprising a first application host, a second application host, a cloud server, and a five-fold screen consisting of a sky screen, a right side screen, a floor screen, a left side screen, and a front screen; the first application host is connected to the cloud server, and is used to collect the initial metaverse data stream of the cloud server, and convert the collected initial metaverse data stream into an initial metaverse picture; the second application host is respectively connected to the first application host, the sky screen, the right side screen, the floor screen, the left side screen, and the front screen, and is used to collect the initial metaverse picture of the first application host, and divide the collected initial metaverse picture into five sub-pictures according to a preset picture division method, and then project the five sub-pictures one by one into five projection pictures, and then output the five projection pictures one by one to the sky screen, the right side screen, the floor screen, the left side screen, and the front screen for display.

[0006] In the second aspect, an embodiment of the present invention also provides a Metaverse five-fold screen interactive display method, which is applied to the above-mentioned Metaverse five-fold screen interactive display system; the method includes: collecting the initial Metaverse data stream of the cloud server through the first application host, and converting the collected initial Metaverse data stream into an initial Metaverse screen; collecting the initial Metaverse screen of the first application host through the second application host, and dividing the collected initial Metaverse screen into five sub-screens according to a preset screen division method, and then projecting the five sub-screens one by one into five projection screens, and then synchronously outputting the five projection screens one by one to the sky screen, right screen, ground screen, left screen and front screen for display.

[0007] The embodiment of the present invention provides a metaverse five-fold screen interactive display system and method, which collects the initial metaverse data stream of the cloud server through the first application host, and converts the collected initial metaverse data stream into an initial metaverse screen; collects the initial metaverse screen of the first application host through the second application host, and divides the collected initial metaverse screen into five sub-screens according to the preset screen segmentation method, and then projects the five sub-screens one by one into five projection screens, and then outputs the five projection screens one by one to the sky screen, the right screen, the ground screen, the left screen and the front screen for display. Using the above technology, through the collaborative work of the first application host and the second application host, the overall efficiency of the interactive display of the metaverse five-fold screen can be improved, and the interactive display of the metaverse five-fold screen can be avoided, thereby improving the interactive display effect of the metaverse five-fold screen; the second application host performs screen segmentation and transmission projection on the initial metaverse screen in turn, thereby ensuring that the existing metaverse data resources can be directly used for the interactive display of the five-fold screen, thereby improving the universality of the metaverse five-fold screen interactive display technology to the existing metaverse data resources.

[0008] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0009] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 It is a schematic diagram of the unfolded shape of a five-fold screen in the prior art;

[0012] Figure 2 A schematic diagram of a rectangular area when producing three-dimensional content for a five-fold screen in the prior art;

[0013] Figure 3 This is a schematic diagram of the structure of a Metaverse five-fold screen interactive display system in an embodiment of the present invention;

[0014] Figure 4 It is a schematic diagram of a partial three-dimensional structure of a Metaverse five-fold screen interactive display system in an embodiment of the present invention;

[0015] Figure 5 This is a structural example diagram of a Metaverse five-fold screen interactive display system in an embodiment of the present invention;

[0016] Figure 6 This is an example diagram of the initial screen of the metaverse being divided into five sub-screens in an embodiment of the present invention;

[0017] Figure 7 An example diagram of calculating the position and size of the fifth sub-picture relative to the initial metaverse picture in an embodiment of the present invention;

[0018] Figure 8 Another example diagram of calculating the position and size of the fifth sub-picture relative to the initial metaverse picture in an embodiment of the present invention;

[0019] Fig. 9 This is an example diagram of performing transmission transformation on a sub-picture in an embodiment of the present invention;

[0020] Fig.10 This is an example diagram of a Metaverse five-fold screen interactive display system and its processing flow in an embodiment of the present invention;

[0021] Fig.11 It is a flow chart of a method for interactive display of a metaverse five-fold screen in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the current metaverse experience, virtual interactive experience often relies on VR and AR helmets / glasses. The disadvantages are: (1) In terms of content production, since the existing three-dimensional software system / video stream cannot be adaptively converted into the scene or video format required in the immersive space, it takes a huge amount of manpower and material resources to develop three-dimensional space modeling, graphics rendering and interactive programs when building the corresponding space scene; (2) When multiple people experience it, everyone needs to wear VR and AR helmets / glasses and enter the common immersive space through the network. This method will be restricted by the site environment, the number of VR devices, network bandwidth and VR host synchronization. In particular, it is impossible to achieve rapid team participation during team visits, training and virtual tourism.

[0024] As naked-eye 3D large screens, immersive spaces, virtual interactive games, etc. become more and more mature, the application of LED display screens in virtual interactions is becoming more and more extensive. At present, whether it is the application of naked-eye 3D large screens, or the construction of XR virtual spaces and immersive spaces, or even scenes such as smart education and smart meetings, they are inseparable from LED display screens as a carrier of information display and dissemination. LED display screens are applied to the Metaverse due to their own ultra-high-definition display, good display performance of high-order low gray, and the ability to easily carry new technologies and enhance the sense of interaction in virtual reality. In the process of applying LED display screens to the Metaverse, the immersive five-fold screen display technology of the Metaverse was born. The immersive five-fold screen is composed of five adjacent screens spliced ​​at 90 degrees, which can present five seamlessly connected pictures. If the five screens of the five-fold screen are unfolded to a plane, it will present Figure 1 As shown in the shape, the five screens can be divided into a left curtain on the left, a front screen in front, a right curtain on the right, a sky curtain above and a ground curtain below according to the positions of the screens.

[0025] Conventional immersive five-fold screen display systems require customized multimedia content. Currently, there are two common solutions: (1) When producing three-dimensional content for a five-fold screen, Figure 2 The rectangular area shown retains only Figure 1 (2) The three-dimensional content is produced for each of the five screens (sky screen, ground screen, left screen, right screen, and front screen) of the five-fold screen, and five sets of three-dimensional scenes / videos are played synchronously. Both of the above methods require the customization of three-dimensional videos / scenes, which is labor-intensive and not suitable for online three-dimensional video streaming with high real-time requirements.

[0026] In addition, the three-dimensional video streams currently used in PCs, TVs and other media are often in high-definition, 4K, and 8K resolutions, and the screen ratios are often 16:9 or 16:10. Due to the special screen structure of the five-fold screen, the existing three-dimensional video streams cannot be directly used for playback on the immersive five-fold screen; and the resolution of the current mainstream VR glasses is 4K, and the multimedia content, scenes and application interfaces produced for VR glasses cannot be perfectly displayed on the immersive five-fold screen; this has resulted in many existing metaverse materials, videos, and application software being unable to be directly used on the immersive five-fold screen.

[0027] Based on this, the present invention provides a metaverse five-fold screen interactive display system and method, which can alleviate the above-mentioned problems existing in the related technology.

[0028] To facilitate understanding of this embodiment, a metaverse five-fold screen interactive display system disclosed in an embodiment of the present invention is first described in detail, see Figure 3 As shown, the system may include a first application host 10, a second application host 11, a cloud server 13, and a five-fold screen consisting of a skylight 1, a right side curtain 4, a floor curtain 6, a left side curtain 8, and a front screen 12; the first application host 10 is connected to the cloud server 13, and is used to collect the initial metaverse data stream of the cloud server 13, and convert the collected initial metaverse data stream into an initial metaverse picture; the second application host 11 is respectively connected to the first application host 10, the skylight 1, the right side curtain 4, the floor curtain 6, the left side curtain 8, and the front screen 12, and is used to collect the initial metaverse picture of the first application host 10, and divide the collected initial metaverse picture into five sub-pictures according to a preset picture division method, and then project the five sub-pictures one by one into five projection pictures, and then output the five projection pictures one by one to the skylight 1, the right side curtain 4, the floor curtain 6, the left side curtain 8, and the front screen 12 for display.

[0029] The above-mentioned initial metaverse data stream may be in the form of a video stream, an interactive data stream, etc., which is not limited to this.

[0030] The embodiment of the present invention provides a metaverse five-fold screen interactive display system, which collects the initial metaverse data stream of the cloud server through the first application host, and converts the collected initial metaverse data stream into an initial metaverse screen; collects the initial metaverse screen of the first application host through the second application host, and divides the collected initial metaverse screen into five sub-screens according to the preset screen segmentation method, and then projects the five sub-screens one by one into five projection screens, and then outputs the five projection screens one by one to the sky screen, the right screen, the ground screen, the left screen and the front screen for display. Using the above technology, through the collaborative work of the first application host and the second application host, the overall efficiency of the interactive display of the metaverse five-fold screen can be improved, and the interactive display of the metaverse five-fold screen can be avoided, thereby improving the interactive display effect of the metaverse five-fold screen; the second application host performs screen segmentation and transmission projection on the initial metaverse screen in turn, thereby ensuring that the existing metaverse data resources can be directly used for the interactive display of the five-fold screen, thereby improving the universality of the metaverse five-fold screen interactive display technology to the existing metaverse data resources.

[0031] As a possible implementation, see Figure 4 As shown, the five-fold screen can be a semi-enclosed cubic box with an opening facing forward, the sky curtain 1 is the top plate of the cubic box, the right side curtain 4 is the right side plate of the cubic box, the floor curtain 6 is the bottom plate of the cubic box, the left side curtain 8 is the left side plate of the cubic box, the front screen 12 is the rear side plate of the cubic box, and the sky curtain 1, the right side curtain 4, the floor curtain 6, the left side curtain 8 and the front screen 12 are all facing the inner side of the cubic box.

[0032] As a possible implementation, see Figure 4 As shown, the above-mentioned Metaverse five-fold screen interactive display system can also include a cabinet 9, and the first application host 10 and the second application host 11 are both located inside the cabinet 9. The cabinet 9 can protect the first application host 10 and the second application host 11.

[0033] As a possible implementation, see Figure 4 and Figure 5 As shown, the above-mentioned Metaverse five-fold screen interactive display system can also include an array microphone 7 and a speaker 2 connected to the first application host 10; the array microphone 7 and the speaker 2 are both arranged on a side of the skylight 1 located on the inner side of the above-mentioned cubic box, and the distance between the array microphone 7 and the front screen 12 is smaller than the distance between the speaker 2 and the front screen 12; the first application host 10 can be used to control the speaker 2 to output a first audio signal corresponding to the initial audio signal when the array microphone 7 collects the initial audio signal of the surrounding environment of the above-mentioned Metaverse five-fold screen interactive display system, and at the same time control the first change of the initial Metaverse picture.

[0034] For example, see Figure 5 As shown, the first application host 10 is connected to the cloud server 13 via the Internet, and the first application host 10 is connected to the second application host 11 via a DP signal line. The second application host 11 has a built-in video capture card and a graphics card that supports 6-channel outputs; the initial Metaverse data stream (i.e., three-dimensional video / application scene) on the cloud server 13 is collected by the first application host 10 and converted into the corresponding initial Metaverse picture (i.e., scene / three-dimensional video picture, the resolution can be 1080p, 4K, 8k, etc., and the picture ratio can be 16:9) through the graphics card; the scene / video picture on the first application host 10 is collected by the video capture card of the second application host 11, and the 5-channel synchronous output of the graphics card of the second application host 11 is sent to the sky screen 1, the right screen 4, the ground screen 6, the left screen 8 and the front screen 12 of the five-fold screen; the digital audio processing The device 14, the power amplifier 15 and the multiple speakers 2 form an audio system, the power amplifier 15 is respectively connected to the digital audio processor 14 and the multiple speakers 2, and the first application host 10 is respectively connected to the array microphone 7 and the digital audio processor 14; the array microphone 7 collects the initial audio signal (such as the voice of the user, etc.) in the surrounding environment of the Metaverse Five-fold Screen Interactive Display System and transmits the collected initial audio signal to the first application host 10, and the first application host 10 transmits this part of the initial audio signal to the digital audio processor 14 for processing, and the digital audio processor 14 transmits the processed audio signal to the power amplifier 15 for amplification, and transmits the amplified audio signal to each speaker 2 by transmitting it to the power amplifier 15, so as to output the first audio signal corresponding to this part of the initial audio signal through the multiple speakers 2. In addition, after the first array microphone 7 transmits the collected initial audio signal to the first application host 10, the first application host 10 can also generate a corresponding picture adjustment signal and apply this part of the picture adjustment signal to the initial metaverse picture to achieve dynamic adjustment of the initial metaverse picture (i.e., control the first change of the initial metaverse picture).

[0035] With the above design, users can interact through voice, thereby better realizing AI voice interaction and interaction with video images and / or scenes.

[0036] As a possible implementation, see Figure 4 and Figure 5As shown, the above-mentioned Metaverse five-fold screen interactive display system can also include a gesture recognition camera 3 connected to the first application host 10; the gesture recognition camera 3 is arranged on a side of the skylight 1 located on the inner side of the above-mentioned cubic box, and the distance between the gesture recognition camera 3 and the front screen 12 is smaller than the distance between the speaker 2 and the front screen 12; the first application host 10 can be used to control the second change of the initial Metaverse screen when the gesture recognition camera 3 collects the initial gesture signal input by the user corresponding to the above-mentioned Metaverse five-fold screen interactive display system.

[0037] For example, following the previous example Figure 5 , the first application host 10 is connected to the gesture recognition camera 3 via USB; when the user makes a special gesture at the lens of the gesture recognition camera 3, the gesture recognition camera 3 will collect the corresponding initial gesture signal and transmit the collected part of the initial gesture signal to the first application host 10, and the first application host 10 will generate a corresponding picture adjustment signal and apply this part of the picture adjustment signal to the initial metaverse picture to realize dynamic adjustment of the initial metaverse picture (that is, control the initial metaverse picture to undergo a second change).

[0038] With the above design, users can perform interactive operations by making gestures, thereby better realizing AI gesture interaction and interaction with video images and / or scenes.

[0039] As a possible implementation, see Figure 4 and Figure 5 As shown, the above-mentioned Metaverse five-fold screen interactive display system may also include a laser radar 5 connected to the first application host 10; the laser radar 5 is arranged on a side of the ground curtain 6 located on the inner side of the above-mentioned cubic box; the first application host 10 can be used to control the third change of the initial Metaverse screen when the laser radar 5 collects the initial foot motion signal input by the user corresponding to the above-mentioned Metaverse five-fold screen interactive display system.

[0040] For example, following the previous example Figure 5 , the first application host 10 is connected to the laser radar 5 via RJ45; when the user steps on a designated position or performs a designated stepping action on the floor curtain 6, the laser radar 5 will collect the corresponding initial foot action signal and transmit the collected part of the initial foot action signal to the first application host 10, and the first application host 10 will generate a corresponding picture adjustment signal and apply this part of the picture adjustment signal to the initial metaverse picture to realize dynamic adjustment of the initial metaverse picture (that is, control the initial metaverse picture to undergo a third change).

[0041] With the above design, users can trigger corresponding interactive functions by stepping on the floor curtain, thereby better realizing AI foot movement interaction and interaction with video images and / or scenes.

[0042] As a possible implementation, see Figure 3 and Figure 6 As shown, the above-mentioned five sub-screens may include a first sub-screen 16 corresponding to the sky screen 1, a second sub-screen 17 corresponding to the floor screen 6, a third sub-screen 18 corresponding to the left side screen 8, a fourth sub-screen 19 corresponding to the right side screen 4 and a fifth sub-screen 20 corresponding to the front screen 12; the above-mentioned second application host 11 may also be used to: set the screen division method of the initial metaverse screen according to the actual size parameters (including length, width, height, etc.) of the above-mentioned five-fold screen; wherein the actual size parameters may include the width and height of the sky screen 1, the right side screen 4, the floor screen 6, the left side screen 8 and the front screen 12 respectively; the screen division method may include the positions and sizes of the first sub-screen 16, the second sub-screen 17, the third sub-screen 18, the fourth sub-screen 19 and the fifth sub-screen 20 respectively relative to the initial metaverse screen.

[0043] For example, see Figure 6 As shown, the relative position relationship between the first sub-screen 16, the second sub-screen 17, the third sub-screen 18, the fourth sub-screen 19 and the fifth sub-screen 20 may include: the center point position of the fifth sub-screen 20 coincides with the center point position of the initial metaverse screen; the first sub-screen 16 and the second sub-screen 17 are respectively located above and below the fifth sub-screen 20, the top and bottom edges of the first sub-screen 16 coincide with the top edge of the initial metaverse screen and the top edge of the fifth sub-screen 20, and the top and bottom edges of the second sub-screen 17 coincide with the bottom edge of the fifth sub-screen 20 and the bottom edge of the initial metaverse screen; the third sub-screen 18 and the fourth sub-screen 19 are respectively located to the left and right of the fifth sub-screen 20, the left side and right side edges of the third sub-screen 18 coincide with the left side edge of the initial metaverse screen and the left side edge of the fifth sub-screen 20, and the left side and right side edges of the fourth sub-screen 19 coincide with the right side edge of the fifth sub-screen 20 and the right side edge of the initial metaverse screen.

[0044] Based on this, the second application host can also be used to: establish a three-dimensional space model corresponding to the five-fold screen according to the actual size parameters of the five-fold screen; place a virtual camera at a preset distance directly in front of the three-dimensional space model, and control the virtual camera to emit rays to the three-dimensional space model; determine the front plane of the three-dimensional space model as the projection plane of the ray, and determine the position and size of the projection outline of the rear plane of the three-dimensional space model on the projection plane relative to the front plane as the position and size of the fifth sub-screen relative to the initial metaverse screen; based on the relative position relationship, determine the positions and sizes of the first sub-screen, the second sub-screen, the third sub-screen and the fourth sub-screen respectively relative to the initial metaverse screen according to the position and size of the fifth sub-screen relative to the initial metaverse screen.

[0045] For example, see Figures 6 to 8 As shown, the actual size parameters (i.e., length, width, and height) of the five-fold screen can be entered on the second application host 11, and a rectangular three-dimensional space model 21 is established on a horizontal plane (i.e., the ground plane 27) through the second application host 11, and the length, width, and height of the three-dimensional space model 21 correspond to the length, width, and height of the five-fold screen respectively; a virtual camera 22 is placed at a certain distance (i.e., a preset distance, and the specific value can be customized as needed) in front of the three-dimensional space model 21 through the second application host 11, and the virtual camera 22 is controlled to emit rays 23 to each vertex of the three-dimensional space model 21; if the volume of the virtual camera 22 is ignored, the virtual camera 22 can be regarded as the starting point of the ray 23, and the starting point and the four vertices of the back plane of the three-dimensional space model are connected into a a tetrahedron with the starting point as the apex; the front plane 24 of the three-dimensional space model 21 can be determined as a projection surface by the second application host 11, and the projection contour 26 of the rear plane 25 corresponding to the projection surface will be formed after the tetrahedron intersects and is truncated with the projection surface; then the position and size of the projection contour 26 relative to the front plane 24 can be determined as the position and size of the fifth sub-screen 20 relative to the initial metaverse screen by the second application host 11; then the vertices in the upper left corner, upper right corner, lower right corner and lower left corner of the projection contour 26 can be connected to the vertices in the upper left corner, upper right corner, lower right corner and lower left corner of the front plane 24 one by one to form line segments through the second application host 11, thereby determining the positions and sizes of the first sub-screen, the second sub-screen, the third sub-screen and the fourth sub-screen relative to the initial metaverse screen respectively.

[0046] See also Figure 7 and Figure 8 As shown, when calculating the position and size of the projection contour 26 in the projection plane, since the aspect ratio and size of the back plane 25 are known, the position and size of the projection contour 26 in the projection plane can be calculated by the principle of similar triangles, the Pythagorean theorem of triangles and the geometric formula of triangles. Specifically, a spatial rectangular coordinate system XYZ is established with the position of the virtual camera 22 as the coordinate origin O, the X axis is always parallel to the front plane 24, the Y axis passes through the origin O and is perpendicular to the X axis, the Z axis passes through the origin O and is perpendicular to the X axis and the Y axis, the triangle OA'B' is similar to the triangle OAB, the Z axis intersects the line segment AB perpendicularly at point D, the line segment A'B' is the projection of the line segment AB on the front plane 24, and the length value of the line segment B'C can be used to represent the position relationship of the line segment A'B' in the front plane 24, and it is assumed that the line segment BC=a, the line segment CR=b, the line segment OR=c, the line segment OA'=e, and the line segment AB=f. According to the triangle geometric formula, line segment A'B'=f*b / (a+b), line segment B'C=a*c / (a+b). Similarly, the position and size of the projection profile 26 in the projection plane can be calculated.

[0047] By adopting the above operation method, as long as the actual size parameters of the five-fold screen are input on the second application host, the position and size of the fifth sub-screen relative to the initial metaverse screen can be obtained through the second application host, and the overall screen division method of the initial metaverse screen can be obtained by connecting lines.

[0048] As a possible implementation, the second application host can also be used to: set corresponding transmission transformation parameters for each sub-screen based on the actual size parameters of the five-fold screen; for each sub-screen, apply the transmission transformation parameters corresponding to the sub-screen to perform transmission transformation on all pixel points of the sub-screen to obtain the projection screen corresponding to the sub-screen.

[0049] For example, see Fig. 9 As shown, after the second application host divides the initial metaverse screen into the above-mentioned five sub-screens, it can perform a transmission transformation on each sub-screen respectively; taking the first sub-screen 16 as an example, the transmission transformation of the first sub-screen 16 can be performed in the following manner: first, a transmission camera 28 is configured and rays are emitted to the first sub-screen 16 through the transmission camera 28, and then a spatial rectangular coordinate system is constructed with the position of the transmission camera 28 as the origin, the negative direction of the transmission camera 28 as the Z axis, the upward direction of the transmission camera 28 as the Y axis, and the direction orthogonal to the Y axis and the Z axis as the X axis, and the line of sight direction (i.e., direction), field of view angle, upward direction, near plane and far plane of the transmission camera 28 are set respectively. When setting the near plane and far plane of the transmission camera 28, the width and height of the far plane can be set to be equal to the width and height of the sky screen 1, and the near plane is set so that the four vertices of the first sub-screen 16 are respectively located on different cone surfaces of the viewing cone; after setting the sight direction (i.e., orientation), viewing angle, upward direction, near plane and far plane of the transmission camera 28, the position coordinates of the four vertices of the first sub-screen 16 and the four vertices of the far plane in the spatial rectangular coordinate system can be obtained, and the transmission transformation parameters for transmission transformation of all pixel points of the first sub-screen 16 to the far plane are set accordingly; then the transmission transformation parameters are applied to perform transmission transformation on all pixel points of the first sub-screen 16, so as to obtain the corresponding projection screen for display on the sky screen. The transmission transformation method for each of the second sub-screen 17, the third sub-screen 18, the fourth sub-screen 19 and the fifth sub-screen 20 is similar to the transmission transformation method for the first sub-screen 16 in principle, and will not be repeated here.

[0050] When the above-mentioned projection image is displayed through a five-fold screen, since objects outside the virtual camera's visual cone will not be rendered, and objects that partially intersect with the virtual camera's visual cone will be clipped, there will be a situation where distant objects are suddenly drawn when the camera moves forward. A common processing method is to add a layer of special effects (such as fog) in the distance to block the distant objects, so that the distant objects gradually appear when the virtual camera moves forward.

[0051] As a possible implementation, the floor curtain may be an LED display screen packaged by on-board glue.

[0052] In the projection method of the conventional immersive five-fold screen system, when a person enters the five-fold screen, a shadow will be produced, affecting the interactive experience. Although there are LED floor tile screens, due to their structural characteristics, the gap between the lamp beads is large and the pixel density is low, resulting in a low overall resolution.

[0053] In terms of realizing the walk-in interactive experience of the metaverse, the five-fold screen floor curtain in the embodiments of the present invention can adopt an LED screen encapsulated by GOB onboard glue, and a layer of wear-resistant encapsulation glue (the main component is epoxy resin) can be covered on the upper surface of the floor curtain, so that the floor curtain supports people standing and stepping, so that people can directly step into the five-fold screen for the interactive experience of the metaverse, and realize the effect of immersive surround. While not affecting the overall resolution of the floor curtain, it solves the problem of human shadows appearing during projection in conventional immersive five-fold screen systems, thereby enhancing the user's experience of interacting with the metaverse.

[0054] Based on the above, see Fig.10 As shown, the processing flow of the above-mentioned Metaverse five-fold screen interactive display system can be summarized into the following steps:

[0055] Step 1: Set the screen splitting mode in the second application host 11 (only need to set it once for the first application).

[0056] In step 2, the first application host 10 collects the video stream of the cloud server 13 and generates a dynamic picture.

[0057] In step 3, the second application host 11 collects dynamic images of the first application host 10 .

[0058] In step 4, the second application host 11 performs screen segmentation and video processing.

[0059] In step 5, the second application host 11 synchronously outputs the processed image to the five-fold screen for display.

[0060] The above-mentioned Metaverse five-fold screen interactive display system has the following advantages:

[0061] (1) The Metaverse interactive display is highly efficient. The first application host is responsible for the interactive application of the Metaverse, and the second application host is responsible for the acquisition, segmentation, processing and output of the image, forming the characteristics of "dual-machine video stream processing". The first application host and the second application host divide the work and cooperate with each other, which improves the overall performance of the Metaverse five-fold screen interactive display.

[0062] (2) No need to wear VR / AR equipment, and it can be experienced by multiple people at the same time.

[0063] (3) It has a video stream adaptation function. For the existing 3D video streams, application software and games, the screen can be adaptively matched. The first application host uses the cloud video stream normally, and outputs it to each surface of the five-fold screen through the second application host. In this way, the screen matching of the cloud video stream and the metaverse application is achieved, and there is no need to design a UI and application system specifically for the immersive five-fold screen.

[0064] (4) It supports walk-in experience and will not cause shadow obstruction (compared to immersive projection technology), providing a stronger experience.

[0065] The above-mentioned Metaverse five-fold screen interactive display system is suitable for various applications in the Metaverse environment, such as: Metaverse-based fire drills, chemistry / physics experiments, industrial production line demonstrations, military exercises, building construction, aerospace / navigation, multi-dimensional teaching, historical / future scene experience, cultural heritage digital restoration applications, virtual tourism, online museum visits, Metaverse social networking, games, shopping, press conferences, etc.

[0066] Based on the above-mentioned Metaverse five-fold screen interactive display system, an embodiment of the present invention further provides a Metaverse five-fold screen interactive display method, which can be applied to the above-mentioned Metaverse five-fold screen interactive display system; see Fig.11 As shown, the method may include the following steps:

[0067] Step S1102, collecting the initial Metaverse data stream of the cloud server through the first application host, and converting the collected initial Metaverse data stream into an initial Metaverse screen.

[0068] Step S1104, collect the initial metaverse screen of the first application host through the second application host, and divide the collected initial metaverse screen into five sub-screens according to the preset screen division method, and then project the five sub-screens one by one into five projection screens, and then output the five projection screens one by one to the sky screen, right screen, ground screen, left screen and front screen for display.

[0069] The embodiment of the present invention provides a method for interactive display of a five-fold screen of the Metaverse, wherein the first application host collects the initial Metaverse data stream of the cloud server, and converts the collected initial Metaverse data stream into an initial Metaverse screen; the second application host collects the initial Metaverse screen of the first application host, and divides the collected initial Metaverse screen into five sub-screens according to a preset screen segmentation method, and then projects the five sub-screens one by one into five projection screens, and then outputs the five projection screens one by one to the sky screen, the right screen, the ground screen, the left screen and the front screen for display. By adopting the above technology, the overall efficiency of the interactive display of the five-fold screen of the Metaverse can be improved through the collaborative work of the first application host and the second application host, and the interactive display of the five-fold screen of the Metaverse can be avoided, thereby improving the interactive display effect of the five-fold screen of the Metaverse; the second application host sequentially divides and projects the initial Metaverse screen, thereby ensuring that the existing Metaverse data resources can be directly used for the interactive display of the five-fold screen, thereby improving the universality of the Metaverse five-fold screen interactive display technology for the existing Metaverse data resources.

[0070] As a possible implementation, the above-mentioned five sub-screens may include a first sub-screen corresponding to the sky screen, a second sub-screen corresponding to the ground screen, a third sub-screen corresponding to the left screen, a fourth sub-screen corresponding to the right screen and a fifth sub-screen corresponding to the front screen; based on this, the above-mentioned Metaverse five-fold screen interactive display method may also include: setting the screen division method of the initial Metaverse screen according to the actual size parameters of the five-fold screen input by the user through the second application host; wherein the actual size parameters include the width and height of the sky screen, the right screen, the ground screen, the left screen and the front screen respectively; the screen division method includes the positions and sizes of the first sub-screen, the second sub-screen, the third sub-screen, the fourth sub-screen and the fifth sub-screen respectively relative to the initial Metaverse screen.

[0071] As a possible implementation, the relative position relationship between the first sub-screen, the second sub-screen, the third sub-screen, the fourth sub-screen and the fifth sub-screen may include: the center point of the fifth sub-screen coincides with the center point of the initial metaverse screen; the first sub-screen and the second sub-screen are respectively located above and below the fifth sub-screen, the top and bottom edges of the first sub-screen coincide with the top edge of the initial metaverse screen and the top edge of the fifth sub-screen, and the top and bottom edges of the second sub-screen coincide with the bottom edge of the fifth sub-screen and the bottom edge of the initial metaverse screen; the third sub-screen and the fourth sub-screen are respectively located to the left and right of the fifth sub-screen, the left and right edges of the third sub-screen coincide with the left edge of the initial metaverse screen and the left edge of the fifth sub-screen, and the left and right edges of the fourth sub-screen coincide with the right edge of the fifth sub-screen and the right edge of the initial metaverse screen; based on this, the above The interactive display method of the Metaverse five-fold screen may also include: establishing, by the second application host, a three-dimensional space model corresponding to the five-fold screen according to the actual size parameters of the five-fold screen input by the user; placing, by the second application host, a virtual camera at a position directly in front of the three-dimensional space model at a preset distance from the three-dimensional space model, and controlling the virtual camera to emit rays to the three-dimensional space model; determining, by the second application host, the front plane of the three-dimensional space model as the projection plane of the ray, and determining, by the second application host, the position and size of the projection outline of the rear plane of the three-dimensional space model on the projection plane relative to the front plane as the position and size of the fifth sub-screen relative to the initial Metaverse screen; determining, by the second application host based on the above relative position relationship, the positions and sizes of the first sub-screen, the second sub-screen, the third sub-screen and the fourth sub-screen respectively relative to the initial Metaverse screen according to the position and size of the fifth sub-screen relative to the initial Metaverse screen.

[0072] As a possible implementation, the five-fold screen may be a semi-enclosed cubic box having an opening facing forward, the sky curtain being the top plate of the cubic box, the right side curtain being the right side plate of the cubic box, the floor curtain being the bottom plate of the cubic box, the left side curtain being the left side plate of the cubic box, the front screen being the rear side plate of the cubic box, and the sky curtain, the right side curtain, the floor curtain, the left side curtain and the front screen all face the inner side of the cubic box.

[0073] As a possible implementation, the Metaverse five-fold screen interactive display system may further include an array microphone and a speaker connected to the first application host; the array microphone and the speaker are both arranged on a side of the skylight located on the inner side of the cubic box, and the distance between the array microphone and the front screen is smaller than the distance between the speaker and the front screen; based on this, the Metaverse five-fold screen interactive display method may further include: when the array microphone of the first application host collects the initial audio signal of the surrounding environment of the Metaverse five-fold screen interactive display system, the speaker is controlled to output a first audio signal corresponding to the initial audio signal, and at the same time controls the first change of the initial Metaverse picture.

[0074] As a possible implementation, the Metaverse five-fold screen interactive display system may further include a gesture recognition camera connected to the first application host; the gesture recognition camera is arranged on one side of the skylight located on the inner side of the cubic box, and the distance between the gesture recognition camera and the front screen is smaller than the distance between the speaker and the front screen; based on this, the Metaverse five-fold screen interactive display method may further include: when the first application host collects the initial gesture signal input by the user corresponding to the Metaverse five-fold screen interactive display system through the gesture recognition camera, controlling the second change of the initial Metaverse screen.

[0075] As a possible implementation, the Metaverse five-fold screen interactive display system may further include a laser radar connected to the first application host; the laser radar is arranged on one side of the ground curtain located on the inner side of the above-mentioned cubic box; based on this, the Metaverse five-fold screen interactive display method may further include: when the first application host collects the initial foot motion signal of the user input corresponding to the Metaverse five-fold screen interactive display system through the laser radar, the third change of the initial Metaverse screen is controlled.

[0076] As a possible implementation manner, the above-mentioned step of projecting the five sub-screens into five projection screens one by one through the second application host may include: setting corresponding transmission transformation parameters for each sub-screen based on the actual size parameters of the five-fold screen by the second application host; and applying the transmission transformation parameters corresponding to each sub-screen to perform transmission transformation on all pixel points of the sub-screen by the second application host to obtain the projection screen corresponding to the sub-screen.

[0077] The Metaverse five-fold screen interactive display method provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned Metaverse five-fold screen interactive display system embodiment. For the sake of brief description, for matters not mentioned in the method embodiment, reference may be made to the corresponding contents in the aforementioned system embodiment.

[0078] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A metaverse five-fold screen interactive display system, characterized in that: The system includes a first application host, a second application host, a cloud server, and a five-fold screen consisting of a sky screen, a right screen, a floor screen, a left screen, and a front screen; The first application host is connected to the cloud server, and is used to collect the initial Metaverse data stream of the cloud server, and convert the collected initial Metaverse data stream into an initial Metaverse picture; The second application host is connected to the first application host, the sky screen, the right side screen, the floor screen, the left side screen and the front screen respectively, and is used to collect the initial metaverse picture of the first application host, and divide the collected initial metaverse picture into five sub-pictures according to a preset picture division method, and then project the five sub-pictures into five projection pictures in a one-to-one correspondence, and then output the five projection pictures in a one-to-one correspondence and synchronously to the sky screen, the right side screen, the floor screen, the left side screen and the front screen for display; The five sub-screens include a first sub-screen corresponding to the sky screen, a second sub-screen corresponding to the ground screen, a third sub-screen corresponding to the left screen, a fourth sub-screen corresponding to the right screen, and a fifth sub-screen corresponding to the front screen; the second application host is further used to: establish a three-dimensional space model corresponding to the five-fold screen according to actual size parameters of the five-fold screen; place a virtual camera at a preset distance in front of the three-dimensional space model, and control the virtual camera to emit rays to the three-dimensional space model; Determine the front plane of the three-dimensional space model as the projection plane of the ray, and determine the position and size of the projection outline of the back plane of the three-dimensional space model on the projection plane relative to the front plane as the position and size of the fifth sub-screen relative to the initial metaverse screen; based on the relative position relationship among the first sub-screen, the second sub-screen, the third sub-screen, the fourth sub-screen and the fifth sub-screen, determine the positions and sizes of the first sub-screen, the second sub-screen, the third sub-screen and the fourth sub-screen respectively relative to the initial metaverse screen according to the position and size of the fifth sub-screen relative to the initial metaverse screen; The second application host is also used to: set corresponding transmission transformation parameters for each sub-screen based on the actual size parameters of the five-fold screen; for each sub-screen, apply the transmission transformation parameters corresponding to the sub-screen to perform transmission transformation on all pixel points of the sub-screen to obtain the projection screen corresponding to the sub-screen.

2. The system according to claim 1, characterized in that The second application host is further used for: According to the actual size parameters of the five-fold screen, the screen division method of the initial metaverse screen is set; wherein the actual size parameters include the width and height of the sky screen, the right screen, the ground screen, the left screen and the front screen; the screen division method includes the positions and sizes of the first sub-screen, the second sub-screen, the third sub-screen, the fourth sub-screen and the fifth sub-screen respectively relative to the initial metaverse screen.

3. The system according to claim 2, characterized in that The relative position relationship between the first sub-screen, the second sub-screen, the third sub-screen, the fourth sub-screen and the fifth sub-screen includes: the center point position of the fifth sub-screen coincides with the center point position of the initial metaverse screen; the first sub-screen and the second sub-screen are respectively located above and below the fifth sub-screen, the top and bottom edges of the first sub-screen coincide with the top edge of the initial metaverse screen and the top edge of the fifth sub-screen, and the top and bottom edges of the second sub-screen coincide with the bottom edge of the fifth sub-screen and the bottom edge of the initial metaverse screen, respectively; the third sub-screen and the fourth sub-screen are respectively located to the left and right of the fifth sub-screen, the left side and right side edges of the third sub-screen coincide with the left side edge of the initial metaverse screen and the left side edge of the fifth sub-screen, and the left side and right side edges of the fourth sub-screen coincide with the right side edge of the fifth sub-screen and the right side edge of the initial metaverse screen.

4. The system according to claim 1, characterized in that The five-fold screen is a semi-enclosed cubic box with an opening facing forward, the sky curtain is the top plate of the cubic box, the right curtain is the right side plate of the cubic box, the floor curtain is the bottom plate of the cubic box, the left curtain is the left side plate of the cubic box, the front screen is the rear side plate of the cubic box, and the sky curtain, the right side curtain, the floor curtain, the left curtain and the front screen are all facing the inner side of the cubic box.

5. The system according to claim 4, characterized in that The system further comprises an array microphone and a sound box connected to the first application host; the array microphone and the sound box are both arranged on a side of the sky screen located inside the cubic box, and the distance between the array microphone and the front screen is smaller than the distance between the sound box and the front screen; The first application host is used to control the speaker to output a first audio signal corresponding to the initial audio signal when the array microphone collects the initial audio signal of the system's surrounding environment, and at the same time control the first change of the initial metaverse picture.

6. The system according to claim 5, characterized in that The system further comprises a gesture recognition camera connected to the first application host; the gesture recognition camera is arranged on a side of the skylight located inside the cubic box, and the distance between the gesture recognition camera and the front screen is smaller than the distance between the sound box and the front screen; The first application host is used to control the second change of the initial metaverse screen when the gesture recognition camera captures an initial gesture signal input by a user corresponding to the system.

7. The system according to claim 6, characterized in that The system further comprises a laser radar connected to the first application host; the laser radar is arranged on a side of the floor curtain located inside the cubic box; The first application host is used to control the third change of the initial metaverse screen when the laser radar collects the initial foot motion signal input by the user corresponding to the system.

8. The system according to any one of claims 1 to 7, characterized in that: The floor curtain is an LED display screen encapsulated by on-board glue.

9. A metaverse five-fold screen interactive display method, characterized in that: The method is applied to the Metaverse five-fold screen interactive display system according to any one of claims 1 to 8; the method comprises: Collecting the initial Metaverse data stream of the cloud server through the first application host, and converting the collected initial Metaverse data stream into an initial Metaverse picture; The initial metaverse screen of the first application host is collected by the second application host, and the collected initial metaverse screen is divided into five sub-screens according to a preset screen division method, and then the five sub-screens are projected one by one into five projection screens, and then the five projection screens are synchronously output one by one to the sky screen, right screen, ground screen, left screen and front screen for display.

Images